Versatile C++ game scraper: Skyscraper

tacodog

@muldjord Thanks for the replay about the psx games. I was able to write a script that would read the m3u file to get the cue file, and then grab the checksum of the bin/cue files and grab the correct images using --query. It works great.

I have another question/issue. I just scraped my nes roms and Super Mario 2 seems to grab the japanese wheel no matter what I do. I've tried --region 'us', 'eu', and 'wor', but I always get the "Super Mario USA" wheel.

Is this an issue with the way I'm running Skyscraper? an issue with the program itself? or an issue with the screenscraper database? Can you repeat this?

The wheel I want is located here:
https://screenscraper.fr/image.php?gameid=1248&media=wheel&hd=1&region=wor&num=&version=&maxwidth=600&maxheight=300

Skyscraper -p nes --refresh --region 'wor' --unpack -s screenscraper -i . -o . -g . --pretend /mnt/exhd0/roms/nes/Favorites/Super\ Mario\ Bros.\ 2\ \(USA\)\ \(Rev\ A\).zip

The md5 of my (unzipped) rom is d769d73689ea4684dbb0bd7cf3a24b2f.

Thanks

muldjord

@annomatik Skyscraper already matches just filenames when it looks for games at screenscraper. But it only works if the filename is unique in their database. So if they have 2 roms with the same filename, it won't work. This is not a Skyscraper thing, but a screenscraper thing, so not something I can change. You can try scraping the games with some of the other sources. They are filename search based.

You can, however, overwrite the checksums Skyscraper uses when looking for the games with the '--query' option. Read more about that here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

It works from 2.7.5 and newer. :)

muldjord

@tacodog I'll need to look into that when I get the time. It could be a bug of some sort. If the '--region us' doesn't pull it in, there might be a problem. If the wheel exists in their database, it should use it. If it didn't it will prioritize other regions until it finds one that does exist. So either there's a problem with the us wheel in their database, or there's an issue in Skyscraper. Should be easy to reproduce though.

tacodog

@muldjord One more request/suggestion. Would it make sense to divide the import directory by system? My understanding is that all images now go in import/screenshots. So if I wanted to use custom images for Mortal Kombat 2, I would have to remember to change the images in the import directory when building snes or megadrive databases. Not really a big problem, just an idea.

annomatik

Cool, will try the query option. Thanks!

annomatik

Didn't work :-( I used the following script, it said it scraped something, but when I started Emulationstation again, there still was no meta-data for the games I specified:

#!/bin/bash
./Skyscraper -p nes -s screenscraper --query sha1=F5E4CD9F287A8233EEE35AD1881B2770D9F2C453 "/home/pi/RetroPie/roms/nes/Balloon Fight (USA) 2P.nes"
./Skyscraper -p nes -s screenscraper --query sha1=74357C80B61FFD25AC2A802391068600F2646D98 "/home/pi/RetroPie/roms/nes/Banana Prince (GER).nes"
./Skyscraper -p nes -s screenscraper --query sha1=85793B0F013EC0F24F36FED0A2859B3B74131510 "/home/pi/RetroPie/roms/nes/Castlevania 2 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=02633E208732B598E3A8EB80B6E0E09926F25E83 "/home/pi/RetroPie/roms/nes/Donkey Kong Jr. (World).nes"
./Skyscraper -p nes -s screenscraper --query sha1=DA54C223D79FA59EB95437854B677CF69B5CAC8A "/home/pi/RetroPie/roms/nes/Galaga (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=CA513F841D75EFEB33BB8099FB02BEEB39F6BB9C "/home/pi/RetroPie/roms/nes/Ninja Gaiden 1 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=74A15693EEBC2C01C7982C0C9A3F294BC26F53A7 "/home/pi/RetroPie/roms/nes/SCAT (USA) 2P.nes"
./Skyscraper -p nes -s screenscraper --query sha1=FDD3D68C7B5B2F9DC7EC50D9888B0A8EDBAAEC28 "/home/pi/RetroPie/roms/nes/StarTropics 1 (USA).nes"
./Skyscraper -p nes -s screenscraper --query sha1=77747840541BFC62A28A5957692A98C550BD6B2B "/home/pi/RetroPie/roms/nes/Tetris (Nintendo).nes"

muldjord

@annomatik You need to always rescrape with:

$ Skyscraper -p [platform]

when you've added new data to the cache as you just did. That will regenerate the gamelist.xml using all of your cached data. Please read the github documentation thoroughly. It's all very well-documented including these features. :)

muldjord

@tacodog That's a good idea, I will add it to the roadmap. :)

annomatik

Thanks, that worked. To be honest, I'm a little overwhelmed by the doc, so if it's ok for you, I'll just ask my question here.

Next topic. After scraping, I now have a "Adventure Island 3", a "Adventure Island Classic" and a "Adventure Island II".

What I'd like to have instead: "Adventure Island 1", "Adventure Island 2" and "Adventure Island 3".

Where would I specify that so it does not get overwritten with the next db update? Thanks

alt text

annomatik

Also, is it possible to configure the console-colors used? The yellow is very hard to read on a white background. Thanks!

alt text

muldjord

@annomatik If you wish to use your own titles (and keep them) for any game, I suggest using the 'import' feature. Please read the documentation on how to use that. I am not gonna go over it here as it is already well-documented.

I will answer questions that aren't covered in the documentation, but I expect you to take your time and read it, overwhelming as it might be. Just read the sections you need to learn about. It'll be worth it in the end.

As for terminal colors, I don't know if you can change them in Putty. I am not a Windows user myself, so you will have to look into Putty's documentation. There is not a way to change them in Skyscraper.

annomatik

Ok, thanks.

martin_f

Just started using Skyscraper, excellent little tool muldjord ! Appreciate your time in putting this together. I had been using UXML for a little bit, but much prefer Skyscraper.

One question I have. I was reading dorkvader's comments regarding the "spine". Am I correct in assuming that at current time Skyscraper doesn't scrape "spine" art, such as that on https://www.screenscraper.fr/ ? If not, is there any plan to add this feature?

dorkvader, were you manually downloading the spine art, or are there packs floating about?

Anyway, keep up the good work, I am sure its appreciated by many.

muldjord

@martin_f I don't really plan to expand the types of resources that can be fetched currently. But you can sortof cheat and use the 'marquee' as the 'spine' art if you don't use it for anything already. I did this by downloading some of the spine art packs from emumovies, then importing them as 'marquee' and then added them on the side of the boxart effect. That turned out perfect.

martin_f

Nice one muldjord, I will give that a go.

And thanks again for such an awesome tool !

muldjord

@tacodog I have just implemented the feature to have subfolders in the import folder. So if you have 'import/snes/screenshots' it will look for specifically snes screenshots in there and so on.

It's not officially released yet, just wanted to let you know that the feature is done and will be in the next release. You can already try it out now if you'd like to, it's on the master branch on git.

EDIT: It is now released in 2.7.8

muldjord

Skyscraper version 2.7.8 released: https://github.com/muldjord/skyscraper

Added CRC32 checksum attribute to 'screenscraper' pass
Changed the 'import' folder to also look into 'import/[platform]' if it exists so users can create imports per platform. Old method also works (Thanks to tacodog for suggesting this)
Added 'mpixels' attribute to compositor layers that allows users to scale images to a desired area size instead of using width and height (Thank you to Vynce for suggesting this)

Next up is 2.8.0 which will implement some interesting preserve features for the current gamelists. For instance, 'hidden' and 'favorite' aren't preserved for the EmulationStation gamelist.xml currently. They will be from 2.8.0 and forward.

tacodog

Thanks for adding the import subdirectories. I just tested it and it's working well. Just a note for anyone trying this, 'definitions.dat' needs to go in the system subdirectory as well if you want to use textual data. This took me a little while to figure out.

muldjord

@tacodog I was actually wondering if people would have problems figuring that out as it isn't all that intuitive to also have to put it in the platform subfolder. So thank you for reporting this, I will consider making it work by only having it in the import base folder.

EDIT: This is fixed in the upcoming 2.8.0 release (not released yet). It will first look for 'definitions.dat' in the platform subfolder and then in the base import folder.

BiZzAr721

@muldjord said in Versatile C++ game scraper: Skyscraper:

@sammyboy It's a bit tricky to do. You have to compile it on a Linux machine that ISN'T the RetroPie. I would suggest Ubuntu since the installation procedure on there is exactly the same as on the pi. Then you need to create a rom folder with just 1 single rom inside. Then you need to scrape that rom with sources until you have all of the artwork types for it (start by trying screenscraper, it probably has all of them). Then you open a terminal and type in "watch -n2 Skyscraper -p [platform] -i /[full]/[rom]/[folder] -s localdb --unattend". Be sure to change [platform] and the rom folder. That command will run Skyscraper every 2 seconds, using the localdb without asking any questions, which will result in the exported media to be updated every 2 seconds from the '[homedir]/.skyscraper/artwork.xml' file. Now open the relevant output image in an image viewer that automatically reloads if the image changes. The default one in Ubuntu works.

Now all you need to do is open up an editor and edit '[homedir]/.skyscraper/artwork.xml' and the image will update with your changes every 2 seconds.

So yeah, not so simple unfortunately. It can be done on the pie itself too if you ssh connect to it from another machine through 2 different terminals. On one you run the above command. On the other you edit the artwork.xml. And then you open a file browser and browse to the export folder where the images reside. I DO NOT recommend doing this on the pi, as the command will save data to the SD card every 2 seconds while you're working on it. SD cards do not like that! You've been warned. :)

@muldjord Excellent work! I never realized just how customizable you made this (especially importing local artwork!) Since some time has passed, is there any easier ways to view the artwork while editing artwork.xml?