Test a new data source for sselph/scraper

sselph

Yes -help should have it. -console_img="b" is the default which is 2D boxart -console_img="3b"would get the 3D boxart. Right now the order is gdb, ss, ovgdb but I need to rework some of this to handle specifying the order.

DrMaxwell

@sselph I think having a preference would be good. Silly question, but does the scraper check against all three sequentially; for example, if the first doesn't provide the metadata, then it goes to the next database until both image and metadata are found?

sselph

@DrMaxwell Thats the idea but it might stop without getting an image and only metadata. I can't remember.

tien_huu_1408

@sselph I like this mix template for scraping. How could you do this? Could you integrate into your script option? Thanks a lot!

tien_huu_1408

@sselph Any update for the script?

paradadf

@sselph it is amazing how fast SS as a source is!!!

I notice two things while using it though...

I can't scrape any arcade games (mame, fba, neogeo)
With one specific gba rom I get always the following error message and the scraper stops:

2016/10/03 13:36:15 INFO: Checking for new hash.csv.
2016/10/03 13:36:15 INFO: hash.csv "e2315a44c3d4c5d200477a109e079f29" up to date.
2016/10/03 13:36:15 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
2016/10/03 13:36:16 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
2016/10/03 13:36:16 INFO: Starting: gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip
2016/10/03 13:36:16 ERR: error processing gba\2 Game Pack! - Matchbox Missions - Emergency Response + Air, Land and Sea Rescue (USA).zip: image: unknown format
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0x407bc7]

goroutine 28 [running]:
panic(0x81cd20, 0xc0420040b0)
        /usr/local/go/src/runtime/panic.go:500 +0x1af
main.crawlROMs.func2(0xc04340c010, 0xc043408000, 0x0, 0xc042a8dfe0, 0xc0425e9980, 0xc042a971c0)
        /home/sselph/go/src/github.com/sselph/scraper/scraper.go:281 +0x247
created by main.crawlROMs
        /home/sselph/go/src/github.com/sselph/scraper/scraper.go:314 +0x8c7

lilbud

Can you enable scraping games that are on an external drive.

DrMaxwell

@lilbud Just put the scraper executable in the same folder as your roms; whether or not the drive is external is irrelevant.

sselph

@paradadf Thanks. I found the issue and have it fixed on github. I wasn't passing the error returned when the bad image was found and caused it to crash when using -missing. I can't reproduce the image error. Maybe that game was fixed? or it was temporary. I also only set this up to scrape console games. I didn't do arcade since it uses names instead of hashes and hadn't done any testing on that.

paradadf

@sselph the rom works perfectly, so I don't understand what's the problem.
Anyway, I just noticed that the letters with accents á, é, ó, etc... aren't being shown on the xml file (those letters are just erased).

Nightfire

Hi @sselph, happy to hear you're working on the scraper again, I've used it every time I've worked with RetroPie. Not sure if you had seen recent development news of RetroPie, but they are working on emulators for PSP (ppsspp), Nintendo DS (desmune) and GameCube (dolphin). So, I was wondering, will you be adding support for scraping PSP/NDS/GameCube games? Or is there a way to scrape for those systems that I'm just not aware of?

robertybob

@Nightfire said in Test a new data source for sselph/scraper:

Hi @sselph, happy to hear you're working on the scraper again, I've used it every time I've worked with RetroPie. Not sure if you had seen recent development news of RetroPie, but they are working on emulators for PSP (ppsspp), Nintendo DS (desmune) and GameCube (dolphin). So, I was wondering, will you be adding support for scraping PSP/NDS/GameCube games? Or is there a way to scrape for those systems that I'm just not aware of?

PSP:
https://github.com/sselph/scraper/issues/52

NDS:
https://github.com/sselph/scraper/issues/28

Gamecube would be possible, I think, as the hashes are listed on redump

Nightfire

@robertybob ahh, ok, thanks for the info, guess I never went to the 'issues' when I was there.

spannernick1

Will this scraper have UK box art as well,the ones that come with ES are not good at finding Mega Drive and CD Pal box art,had to end up adding them manually ,sometimes it adds a completely different game when its not that game.

robertybob

@spannernick1 Don't bother with the ES scraper, use Sselph's one.

Sselph's scraper should match (Europe) roms to Mega Drive games and (USA) roms to Genesis games. If not, first check whether the image uploaded to thegamesdb.net is actually for the correct region (more often that not it isn't). http://thegamesdb.net/browse/36/ If the Mega Drive game is listed there and the artwork is for the Europe copy, then check which region your roms are.

Failing that, segaretro.org has all the best quality Mega Drive scans :)

paradadf

If you use the flag -region with eu in the first place, it will find the megadrive box art, assuming it exists and, of course, it is pressent on www.screenscraper.fr
SS either has added to his database the media from the best possible sources, or the people has contributed creating it.

paradadf

I'm not really sure how retropie handles scummvm. I saw on the wiki that you use either .sh or .svm extensions (?). Anyway... this information is valid for recalbox, I suppose that you use files as shortname.svm (or .sh). If you put the fullname inside those files as shown here, you will be able to scrape scummvm using SS as well. I'm assuming that both extensions are already known by the scraper. @sselph Would you mind adding .scummvm to the list, please?

tien_huu_1408

@sselph : I have downloaded scraper code (rpi2 version). On Raspberry Pi 3.
How to use it by terminal?
Hope you could help me step by step, especially for scraping 3D image.
Thanks a lot. Hope to hear from you asap.

paradadf

hi @sselph! I found something weird with one specific Game Boy rom.
http://screenscraper.fr/rominfos.php?romid=309753

the scraper isn't finding the game at all... I asked about it on IRC and the API is apparently returning the right gameid, but still nothing

2016-10-29 13:15:42 | ip: ##.##.##.### | Aladdin (USA) (SGB Enhanced).gb -> gameid=3025

the sha1 is correct also.

Nismo

hi @sselph, the scraper works ok except doesn't downloading any image for any game. What i'm doing wrong?

I use windows version, this is my command line:

scraper -add_not_found=true -append=true -download_images=true -image_dir="P:\emulationstation\roms\megadrive\images" -image_path="P:\emulationstation\roms\megadrive\images" -image_suffix="" -img_format="png" -no_thumb=true -thumb_only=false -thumb_suffix="" -use_gdb=true -use_ss=true -use_ovgdb=true -use_nointro_name=false -workers=4 -missing="missing.txt" -console_img="3b" -region="us,eu,jp,fr,xx" -lang="en"

All metadata is ok on gamelist.xml but in missing.txt all games are "missing image" like this:

AAAHH!!! Real Monsters (UE) [!].zip,,fa579a7e47ee4a1bdb080bbdec1eef480a85293e,missing image
3 Ninjas Kick Back (U) [!].zip,,d634e3f04672b8a01c3a6e13f8722d5cbbc6b900,missing image

Any help whould be appreciated, thanks.