Versatile C++ game scraper: Skyscraper

timb

For what it’s worth, this is how I scrape multiple platforms:

Skyscraper -p nes -s screenscraper --unpack -u user:pass && \
Skyscraper -p snes -s screenscraper --unpack -u user:pass && \
Skyscraper -p n64 -s screenscraper -u user:pass && \
Skyscraper -p gb -s screenscraper --unpack -u user:pass && \
Skyscraper -p gbc -s screenscraper --unpack -u user:pass && \
Skyscraper -p gba -s screenscraper --unpack -u user:pass && \
Skyscraper -p megadrive -s screenscraper --unpack -u user:pass && \
Skyscraper -p sega32x -s screenscraper --unpack -u user:pass && \
Skyscraper -p segacd -s screenscraper --unpack -u user:pass

The && between commands tells the shell to run the next command if the previous command completed without errors. So, in the example above, if the NES, SNES and N64 instances all scrape fine, but the GB instance runs into a problem and quits, the chain will stop there. The \ allows you to split a single command among multiple lines. (It tells the shell to treat the newline as an actual newline, instead of executing the command.)

This is a quick and dirty way to do it, but works fine. I could whip up a short shell script that would be a lot cleaner and allow you to pass it a list of platforms to scrape and args to pass to Skyscraper if you’d like. (Something like this: ./skywrapper.sh -p ‘nes megdrive snes n64’ -wargs ‘-s screenscrapper -u user:pass’)

easye9inches

Ubuntu is new to me and driving me crazy, can someone help. I have my ROMs on an external. I want to scrape the data to the external.

My mount point is /media/usb0

The virtualboy for example, would be: /media/usb0/All ROMs/virtualboy

This is what is confusing me when I goto scrape:

Platform: 'virtualboy'
Scraper module: 'screenscraper'
Input folder: '/media/usb0/’All'
Game list folder: '/media/usb0/’All'
Covers folder: '/media/usb0/’All/covers'
Screenshots folder: '/media/usb0/’All/screenshots'
Wheels folder: '/media/usb0/’All/wheels'
Marquees folder: '/media/usb0/’All/marquees'
Videos folder: '/media/usb0/’All/videos'
Local db folder: 'dbs/virtualboy'

DID YOU KNOW: You can force a refresh of the locally cached data using the '--refresh' option. Skyscraper will then refetch the requested entries from the scraping sources, instead of loading it from cache. Sort of like Ctrl+F5 in a browser.

Forcing 1 threads as this is the anonymous limit in the ScreenScraper scraping module. Sign up for an account at https://www.screenscraper.fr and support them to gain more threads. Then use the credentials with Skyscraper using the '-u [user:password]' command line option or by setting 'userCreds=[user:password]' in '~/.skyscraper/config.ini'.

Looking for optional 'priorities.xml' file in local db folder... Found!
Priorities loaded successfully!

Input folder '/media/usb0/’All' doesn't exist or can't be seen by current user. Please check path and permissions.

muldjord

You need to quote the entire path since you have spaces in it. Otherwise it'll see All roms as two different paths and just stop at All. So basically put in:

$ Skyscraper -p virtualboy -i "/media/usb0/All ROMs/virtualboy" -s screenscraper

But instead of doing that, I would just add this to the ~/.skyscraper/config.ini

[virtualboy]
intputFolder="/media/usb0/All ROMs/virtualboy"

Then you don't have to type it in all the time. Check ~/.skyscraper/config.ini.examplefor more available options, and also check the output of $ Skyscraper --help for all command line options. :)

EDIT: I just realized I've forgotten to add inputFolder as a possible option of the [main] section of config.ini. I'll fix this in the next release. When that's fixed you can add it as:

[main]
intputFolder="/media/usb0/All ROMs"

Then it'll be used as the base for all platforms.

easye9inches

@muldjord Ahhhh. Thanks, it works flawlessly now!

Question? I know you could have only one picture as in a screen shot, cover, etc. I found in DBS folder if i wanted to copy covers, marquees, etc over to the external as well, why are those pics named like 0b809d4d49d064bd95d84b2865cc0f9304750b9d (3d Tetris COVER)? Must you go about in renaming everyone, lol?

muldjord

@easye9inches I've just released 2.7.4 earlier today. That allows you to add the 'inputFolder' variable to the [main] section in config.ini. Then it works for all platforms automatically.

Mick2K

I have a problem. I can't update. Any ideas?!

alt text

muldjord

@mick2k Please run the following and try again, I have been messing around with the 2.7.4 release because I found a silly bug, so I re-released.

Run this:

$ cd
$ cd skysource
$ rm VERSION
$ ./update_skyscraper.sh

That should fix it and update to 2.7.4. Let me know how it goes.

Mick2K

@muldjord said in Versatile C++ game scraper: Skyscraper:

./update_skyscraper.sh

It worked. Thank you very much.

easye9inches

Is there a certain way to name a sub-folder so that it also could get scrapped into the gamelist.xml? For instance, I added a "#JP Games" folder to the genesis library instead of having both a megadrive and genesis system selection. Just to condense the system selection down. But it did not scrape that sub-folder. Is that possible?

Edit: Nevermind. I see they were scraped and included in the gamelist.xml

muldjord

@analoghero Hey man, I've implemented the checksum option you suggested some time back. It's currently on the master branch and will be in 2.7.5 to be released soon. :)

It works by using the command line option '--query' which basically takes either a search query for the filename based scraping modules, or either 'sha1=[checksum]', 'md5=[checksum]', or 'romnom=[filename]' (rom name in French). It also requires a single rom to be passed on command line like so:

$ Skyscraper -p [platform] -s screenscraper --refresh --query sha1=[checksum] /[path]/[to]/[romfile.zip]

This will allow you to override the checksums used when searching for the game. So you can look one up at screenscraper, and just use that.

AnalogHero

@muldjord Nice! Thanks for putting your time into this. Hipe that others could use this option, too.

muldjord

@analoghero It's a niche feature for sure, but I got another request for it, and I figured out a way to implement it in a way I was satisfied with. :)

muldjord

Skyscraper version 2.7.5 released: https://github.com/muldjord/skyscraper

Fixed a bug where 'brackets="false"' in config.ini would be flipped (Thanks to Vynce for reporting this)
Completely refactored pass procedures for cleaner code and to enable '--query' option
Added '--query' command line option. This option requires a single rom file to be passed on command line aswell, otherwise it will be ignored (Thank you to AnalogHero and Vynce for suggesting this)
Added scrapers to 'psx' and 'pc' platforms when using Simple Mode

To elaborate on the "--query" option, this is how it works: For most modules a search query is sent to the scraping module in an URL format. That means that a filename such as "Rick Dangerous.lha" becomes "rick+dangerous". The '+' here means a space. You could probably also use the URL encoded space "rick%20dangerous" but my tests show that most modules expect spaces as '+'. And it is the "rick+dangerous" that you, as the user, can pass as the query, like so:

$ Skyscraper -p [platform] -s [module] --query "rick+dangerous" [filename]

Remember to also add a filename that you wish to use the override with. Otherwise the query will be ignored.

But, not all of the scraping modules are search name based. For instance, the "screenscraper" module can use a variety of different search methods. So for screenscraper, you also have the option of overriding the checksums it use to search for a game. This is especially convenient in cases where a filename exists multiple times in their database and your own local file doesn't match with any of the connected checksums (maybe you've compressed the rom yourself or whatever).
In this case you can look up one of the working checksums on "screenscrapers" website (screenscraper.fr) and override the checksum like these examples:

$ Skyscraper -p [platform] -s [module] --query sha1=[checksum] [filename]
$ Skyscraper -p [platform] -s [module] --query md5=[checksum] [filename]
$ Skyscraper -p [platform] -s [module] --query sha1=[checksum]&md5=[checksum]&romnom=[exact url encoded filename] [filename]

The last example combines two of the checksum options and even the "romnom" option which is "rom name" in French (this is a screenscraper thing, not a Skyscraper thing). You obviously only need one of the checksum options, it's just to show that you can combine them if you really need to.

The '--query' option is clearly an "experts only" option, but for those that like to go down the rabbit hole, I am your humble servant. Down you go... :D

And happy scraping! :)

AnalogHero

@muldjord Dont know if im doing something wrong, but the new query option isnt working for me. I tried this

$ Skyscraper -p amiga -s screenscraper --query md5=0D5E4770B34021A666E9CADF0F39DA75 /home/pi/RetroPie/roms/amiga/BlackViper_v1.0_AGA.lha

and it outputs
1/1 (T1) Pass 1 ---- Game 'BlackViper_v1.0_AGA' not found :( ----

What am i doing wrong?

EDIT: Ok. It works. Just used a different entry from screenscraper.fr md5=7858DFE8AE9A6725B6CAD18D67FEBA61 worked.

muldjord

@analoghero Yeah, I ran into this problem with amiga games aswell. It needs to be the same platform, otherwise it won't match them even though you put in a custom checksum. So for instance, if you copy a sha1 from a cd32 game and try to override it for a standard amiga game, it won't be found. It will be found if you use the amiga version. So this should not be an issue as I assume people will always be using sha1's from the actual platform.

Btw, Skyscraper differentiates the Amiga platforms from the filenames. If a filename has 'cd32' or 'cdtv' in it, it will switch to that platform for that rom.

screech

@muldjord Just checked for the Screenscraper DB ;) "Amiga CD32", "Amiga AGA", "Amiga CDTV",... are child of "Amiga" ;) so if you scrape a CD32 rom on "Amiga" system, it normally return the good game referenced on "Amiga CD32" ;)

It's the same for "MAME" for exemple, if you scrape a CPS1 game in the MAME System, it will return the good game (not needed to specify CPS1 ;) ).

muldjord

@screech Yes, this is true, but switching the platform (in Skyscraper) is necessary for the match to occur. If you scrape the 'amiga' platform with Skyscraper, it will not accept an entry for the 'Amiga CD32' platform because some scraping modules are name search based. So it needs to change the platform in order to accept the returned entry. :)

With Screenscraper this is less important since it's checksum based. But for the others, it's a necessity since many games are multiplatform, and Skyscraper needs to find the right one. :)

tacodog

I have a couple question about the usage of Skyscraper that I was hoping someone could help with.

My first issue is that I use m3u files for my psx games because it handles multi disc games organically giving me just one entry in emulationstation. My m3u file points to ../psx-discs/...(disc 1).bin ../psx-discs/...(disc 2).bin etc. If I scrape by m3u filename, nothing is found. If I check by disc, I get duplicates for each disk, and the gamelist file doesn't point to the m3u. Is there a way to search by filename rather than checksums?

Secondly, I have scraped all of my systems using screenscraper, and I am missing a handful of files.

Local database cache stats:
'screenscraper' module
  Titles       : 428
  Platforms    : 428
  Descriptions : 426
  Publishers   : 418
  Developers   : 412
  Ages         : 171
  Tags         : 414
  Ratings      : 279
  ReleaseDates : 316
  Covers       : 428
  Screenshots  : 428
  Wheels       : 423
  Marquees     : 423
  Videos       : 0

I'd like to fill in these small gaps (just 3 wheels) from a second source, but don't necessary need to grab everything all over again. Is there a way to tell Skyscraper to only get the missing files? Or can I see which files are missing so I can try to grab them manually?

Thanks

muldjord

@tacodog You can scrape single files simply by providing the full or partial path to the rom on command line. You can also use the '--startat' and '--endat' command line options to only scrape a span of roms. As for seeing exactly which roms are missing the data, that is currently not possible.

For the m3u issue, the reason it isn't working for those is that the source you are using (ScreenScraper) doesn't have those exact files in their database. You'd need to overwrite the checksum manually, which can be done with '--query'. Try checking out the release notes for 2.7.5 here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

It's described in there. You can also try scraping it with another source, such as "mobygames" or "thegamesdb". They search by name instead of checksum, so if your filenames are good, they should provide good data. Not for wheels though, as those sources don't support wheels.

hermit

would it be possible to scrape Mess roms?
example for Amstrad GX4000 :
barb2.zip = Barbarian 2
brubber.zip = Burnin' Rubber