Versatile C++ game scraper: Skyscraper

muldjord

Hi guys, please check here: https://retropie.org.uk/forum/topic/19588/skyscraper-now-officially-part-of-retropie-please-test

:)

annomatik

ok, here are the check sums for the NES games... which database should I compare them to?

The MD5 sums:

8bff672e12a06787abfa3372f27f2ae4 Balloon Fight (USA) 2P.nes
a1db8a18bae78e276d22e0725062b559 Banana Prince (GER).nes
73fa5a34a5c1579161a8c1e8ef1b2afc Castlevania 2 (USA).nes
e824d5b1c62bc13de3d2716d29225371 Donkey Kong Jr. (World).nes
9ab02f9f92d67a7ae91ebf981c1e2adf Galaga (USA).nes
2e4a6b4553d1f47db8f0e484aa5cfa3e Ninja Gaiden 1 (USA).nes
d8cbc5f2dbf44fae5239a170ad6a092d SCAT (USA) 2P.nes
cd7816986cbae890e15debd5a06e6414 StarTropics 1 (USA).nes
5c4ab6b03431f7298b19bcab58e858f0 Tetris (Nintendo).nes

The SHA1 sums:

3031c47d4481360a911763213fc3ee4a94779d0a Balloon Fight (USA) 2P.nes
ef8214f4bad14443b37121091690416f8375fbfc Banana Prince (GER).nes
990f6b4c12812ab11f2716d82bfda59de742d636 Castlevania 2 (USA).nes
1e6fcc6af295a2e41f8370efcf2440da4f29d017 Donkey Kong Jr. (World).nes
f0c642b45424b85231e8fe678c9b741fbf16cce9 Galaga (USA).nes
82e8d9e63a2f7cdaefc89ca3a8946b793b15db76 Ninja Gaiden 1 (USA).nes
74a15693eebc2c01c7982c0c9a3f294bc26f53a7 SCAT (USA) 2P.nes
ec6ebf666e3871a8dd6f296a6e200a3ca3d82a87 StarTropics 1 (USA).nes
865849aa2064ab38a5959116e697ebbfda9f4795 Tetris (Nintendo).nes

mitu

@annomatik said in Versatile C++ game scraper: Skyscraper:

ok, here are the check sums for the NES games... which database should I compare them to?

I posted in your other topic how you can check the data on the Screenscraper.fr site, each game has it's own set of name/roms/hashes associated (in the 'ROMs/ISOs' tab of the game detail page)

0_1539342857947_986b9071-805d-4840-a8a0-a9d1eab95809-image.png

annomatik

Wow, that's a ton of games there... :-)

I just checked the first couple of games. Balloon Fight, Banana Prince, Castlevania 2, Donkey Kong Jr. : many versions are there, but not with the checksums I have.

So... I can think of at least four solutions:

Do I really have to keep looking for the right roms ? That would be probably a lot of effort.
I could try adding my own versions to the lists, but why, the meta-data is already there, the roms are working fine, there's already many different versions of the same game (why not just have a list of different checksums for the same game?)...
Could Skyscrapper just match filenames, not CRC sums? How would I enable that? and, if not
Can I maybe give Skyscrapper a pointer to the right meta-data?

In Kodi, it is possible to add a .nfo file, which can contain the link to the details-page of a movie / tv show episode. It is in the exact same solution like the video, just instead of mp4/mkv/whatever, it has the extension nfo. In its simplest form, it's just a simple link, but it can also be used for storing all meta-data, if wanted. And if combined with a link, it first takes the provided meta-data from the nfo and takes the rest via scrapping from the nfo.

Maybe, if that's not yet possible, it would be possible to provide direct links somehow?

Thanks!

tacodog

@muldjord Thanks for the replay about the psx games. I was able to write a script that would read the m3u file to get the cue file, and then grab the checksum of the bin/cue files and grab the correct images using --query. It works great.

I have another question/issue. I just scraped my nes roms and Super Mario 2 seems to grab the japanese wheel no matter what I do. I've tried --region 'us', 'eu', and 'wor', but I always get the "Super Mario USA" wheel.

Is this an issue with the way I'm running Skyscraper? an issue with the program itself? or an issue with the screenscraper database? Can you repeat this?

The wheel I want is located here:
https://screenscraper.fr/image.php?gameid=1248&media=wheel&hd=1&region=wor&num=&version=&maxwidth=600&maxheight=300

Skyscraper -p nes --refresh --region 'wor' --unpack -s screenscraper -i . -o . -g . --pretend /mnt/exhd0/roms/nes/Favorites/Super\ Mario\ Bros.\ 2\ \(USA\)\ \(Rev\ A\).zip

The md5 of my (unzipped) rom is d769d73689ea4684dbb0bd7cf3a24b2f.

Thanks

muldjord

@annomatik Skyscraper already matches just filenames when it looks for games at screenscraper. But it only works if the filename is unique in their database. So if they have 2 roms with the same filename, it won't work. This is not a Skyscraper thing, but a screenscraper thing, so not something I can change. You can try scraping the games with some of the other sources. They are filename search based.

You can, however, overwrite the checksums Skyscraper uses when looking for the games with the '--query' option. Read more about that here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

It works from 2.7.5 and newer. :)

muldjord

@tacodog I'll need to look into that when I get the time. It could be a bug of some sort. If the '--region us' doesn't pull it in, there might be a problem. If the wheel exists in their database, it should use it. If it didn't it will prioritize other regions until it finds one that does exist. So either there's a problem with the us wheel in their database, or there's an issue in Skyscraper. Should be easy to reproduce though.

tacodog

@muldjord One more request/suggestion. Would it make sense to divide the import directory by system? My understanding is that all images now go in import/screenshots. So if I wanted to use custom images for Mortal Kombat 2, I would have to remember to change the images in the import directory when building snes or megadrive databases. Not really a big problem, just an idea.

annomatik

Cool, will try the query option. Thanks!

annomatik

Didn't work :-( I used the following script, it said it scraped something, but when I started Emulationstation again, there still was no meta-data for the games I specified:

#!/bin/bash
./Skyscraper -p nes -s screenscraper --query sha1=F5E4CD9F287A8233EEE35AD1881B2770D9F2C453 "/home/pi/RetroPie/roms/nes/Balloon Fight (USA) 2P.nes"
./Skyscraper -p nes -s screenscraper --query sha1=74357C80B61FFD25AC2A802391068600F2646D98 "/home/pi/RetroPie/roms/nes/Banana Prince (GER).nes"
./Skyscraper -p nes -s screenscraper --query sha1=85793B0F013EC0F24F36FED0A2859B3B74131510 "/home/pi/RetroPie/roms/nes/Castlevania 2 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=02633E208732B598E3A8EB80B6E0E09926F25E83 "/home/pi/RetroPie/roms/nes/Donkey Kong Jr. (World).nes"
./Skyscraper -p nes -s screenscraper --query sha1=DA54C223D79FA59EB95437854B677CF69B5CAC8A "/home/pi/RetroPie/roms/nes/Galaga (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=CA513F841D75EFEB33BB8099FB02BEEB39F6BB9C "/home/pi/RetroPie/roms/nes/Ninja Gaiden 1 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=74A15693EEBC2C01C7982C0C9A3F294BC26F53A7 "/home/pi/RetroPie/roms/nes/SCAT (USA) 2P.nes"
./Skyscraper -p nes -s screenscraper --query sha1=FDD3D68C7B5B2F9DC7EC50D9888B0A8EDBAAEC28 "/home/pi/RetroPie/roms/nes/StarTropics 1 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=77747840541BFC62A28A5957692A98C550BD6B2B "/home/pi/RetroPie/roms/nes/Tetris (Nintendo).nes"

muldjord

@annomatik You need to always rescrape with:

$ Skyscraper -p [platform]

when you've added new data to the cache as you just did. That will regenerate the gamelist.xml using all of your cached data. Please read the github documentation thoroughly. It's all very well-documented including these features. :)

muldjord

@tacodog That's a good idea, I will add it to the roadmap. :)

annomatik

Thanks, that worked. To be honest, I'm a little overwhelmed by the doc, so if it's ok for you, I'll just ask my question here.

Next topic. After scraping, I now have a "Adventure Island 3", a "Adventure Island Classic" and a "Adventure Island II".

What I'd like to have instead: "Adventure Island 1", "Adventure Island 2" and "Adventure Island 3".

Where would I specify that so it does not get overwritten with the next db update? Thanks

alt text

annomatik

Also, is it possible to configure the console-colors used? The yellow is very hard to read on a white background. Thanks!

alt text

muldjord

@annomatik If you wish to use your own titles (and keep them) for any game, I suggest using the 'import' feature. Please read the documentation on how to use that. I am not gonna go over it here as it is already well-documented.

I will answer questions that aren't covered in the documentation, but I expect you to take your time and read it, overwhelming as it might be. Just read the sections you need to learn about. It'll be worth it in the end.

As for terminal colors, I don't know if you can change them in Putty. I am not a Windows user myself, so you will have to look into Putty's documentation. There is not a way to change them in Skyscraper.

annomatik

Ok, thanks.

martin_f

Just started using Skyscraper, excellent little tool muldjord ! Appreciate your time in putting this together. I had been using UXML for a little bit, but much prefer Skyscraper.

One question I have. I was reading dorkvader's comments regarding the "spine". Am I correct in assuming that at current time Skyscraper doesn't scrape "spine" art, such as that on https://www.screenscraper.fr/ ? If not, is there any plan to add this feature?

dorkvader, were you manually downloading the spine art, or are there packs floating about?

Anyway, keep up the good work, I am sure its appreciated by many.

muldjord

@martin_f I don't really plan to expand the types of resources that can be fetched currently. But you can sortof cheat and use the 'marquee' as the 'spine' art if you don't use it for anything already. I did this by downloading some of the spine art packs from emumovies, then importing them as 'marquee' and then added them on the side of the boxart effect. That turned out perfect.

martin_f

Nice one muldjord, I will give that a go.

And thanks again for such an awesome tool !

muldjord

@tacodog I have just implemented the feature to have subfolders in the import folder. So if you have 'import/snes/screenshots' it will look for specifically snes screenshots in there and so on.

It's not officially released yet, just wanted to let you know that the feature is done and will be in the next release. You can already try it out now if you'd like to, it's on the master branch on git.

EDIT: It is now released in 2.7.8