Any idea why Steven Selph's scraper takes so long to scrape at times?

beastmode

I am currently scraping images and video files for the SNES system and it's taking almost 5 hours to scrape 300 rom files. Perhaps I'm doing something wrong?

2017/07/24 01:01:04 INFO: Starting: /home/pi/RetroPie/roms/snes/Super Punch-Out!!
2017/07/24 01:08:34 INFO: Starting: /home/pi/RetroPie/roms/snes/Super R-Type
2017/07/24 01:10:56 INFO: Starting: /home/pi/RetroPie/roms/snes/Super Smash T.V.
2017/07/24 01:16:37 INFO: Starting: /home/pi/RetroPie/roms/snes/Super Star Wars
2017/07/24 01:25:33 INFO: Starting: /home/pi/RetroPie/roms/snes/Super Star Wars - Return of the Jedi.

Any help would be greatly appreciated.

sselph

I'm sorry I didn't see this sooner and you have either given up or dealt with it. I don't really know what the issue could be. The scraping process is

Hash the file (read the file from disk and perform some cpu intense operations on it)
Look the hash up in an in-memory hash table (fast)
If it is in the table, send a request to the server
Receive a response with links to images
Download images

So depending on where it is having issues it could be that the SD card is having issues and it can't read quickly. If you search around I'm sure there are links to benchmark you SD card, or you could just run shasum /home/pi/RetroPie/roms/snes/Super\ Punch-Out* and see how long that takes. The next bottleneck might be your network. You could see about trying to do something like wget http://thegamesdb.net/banners/_gameviewcache/boxart/original/front/1307-1.jpg and see if it takes a while to download. The other issue is this could be thegamesdb and they were being REALLY slow that day and retrying might help.

swim0311

I'm with sselph. Sounds like a sd card issue. I can do around 5000 in 1.5hrs. I have a question for sselph myself. When I download my videos via stephen selph's scraper, I get my videos and boxart fine. My arcade will not work when I have omx player turned to on. When I turn the player off, my arcade videos show up but some are very laggy. Any reason why sselph that is?

sselph

@swim0311 I little off-topic but I suspect the videos on screenscraper.fr are okay and the videos from adb.arcadeitalia.net are in a format that isn't great for the pi. When I was adding adb support the dev told me this:
"Please note that for my videos I use a color space format yuv444p. If codecs can't play videos, you must convert in yuv420p (i.e. "-pix_fmt yuv420p" on avconv/ffmpeg)." so maybe you could bulk covert them using avconv or ffmpeg. Or it could just be they are too large. @ainmosni added an option to transcode with the -convert_videos flag but you'd have to invoke the scraper from the command line and also have the handbrake cli installed or you could also just look at the flags here https://github.com/sselph/scraper/blob/master/rom/rom.go#L399 and bulk convert it with ffmpeg/avconv.

swim0311

cool. thanks for the help. Ill let you know how it went.

hiulit

In case anyone in this thread is interested, I just released v2.1.0 of RetroPie Convert Videos.