Versatile C++ game scraper: Skyscraper

stoo

I was just making a suggestion on the GitHub of another scraping tool when I came across this one.

I noticed someone suggested adding .scummvm for scraping ScummVM. May I suggest instead...

For ScummVM, simply parse the scummvm.ini file (\retropie\configs\scummvm\scummvm.ini) which contains all the information about installed games. This way there is no need to create dummy files to "trick" the scraper.

It shouldn't be difficult to parse this ini file and grab all the game names and paths from there. It has the benefit that the descriptions are the same for everyone (ScummVM always uses the same description for each game variant) so matching by name alone should be no problem.

Hashing ScummVM games does not work well, in my experience. There are far too many different variants of each game (compressed/uncompressed/stripped of extra files, etc.). If you use the values from the .ini file you can forget hashing entirely as ScummVM automatically identifies the games when you add them.

stoo

@analoghero You can contact Screech and the rest of the screenscraper team on IRC. Use the CHAT IRC button at the top of the screenscraper.fr site.

They seem to speak pretty good English as well as French :)

bleuge

@muldjord Thanks!
Also, regarding the Skyscraper halting in my pi while scraping (the issue i opened in github). I've just finnished a total scrapping of all my platform (i built a big skyscript.sh as i told with all platforms). Worked perfectly, no more halts or whatever. Finnished every run ok.

muldjord

@stoo It's an interesting idea. But doesn't ES need the dummy files to have anything on the list at all? I assume it simply runs scummvm through runcommand with the dummy filename or similar.

stoo

@muldjord ES uses .svm as dummy files for generating the gamelist and, presumably, for the runcommand, and not .scummvm which is used, I believe, for Recalbox.

However, these files are empty by default, and the hack to get most scrapers to acknowledge them is to put the FULL NAME of the game as text into the dummy file.

This seems pointless, given that the .ini file is 100% accurate (ScummVM detects the games itself when you add them) and contains all the information you need already.

muldjord

@stoo Skyscraper looks for both .svm and .scummvm files.

It won't get rid of the dummy files, but what I can do is, for each dummy file, look that filename up in the scummvm.ini file and use the name from that file when scraping. Then you could call the dummy files whatever you like, they would be recognized regardless.

EDIT: 'players' have been fixed to always be a 1-digit or in some cases '4+' now. I currently look for the formats "1 Player", "1-3 Players", "1-3" and "1 - 2 (2 simultaneous)" and convert them to "1", "3", "3" and "2". If players exceeds 9 it handles that aswell.
If you happen upon more formats Skyscraper needs to understand, let me know in a comment. :)

hi_im_daft

@muldjord - I have seen a few 1 - 4 as well, might as well add those to cover bases, and since that's the maximum number of players (traditionally) supported by consoles.

muldjord

@hi_im_daft Yes, "1 - 4" is already covered by my current regexp. It can even do "1 - 10" without problems. I think that's it then. I will move on to improving the game result search match function in the coming week. I have plenty ideas on how to get better results from the sources that don't use checksumming. It's ok'ish right now, but I can imrove it at lot.

AnalogHero

@muldjord I went ahead and compiled 2.3.1 from master and can say it works as it should:)

muldjord

@analoghero Cool :) Glad it works. Thank you for testing it.

AnalogHero

@muldjord Strange thing is that the .hack-link something is gone now. I know its filtered now, but i rescraped my megadrive collection with exact the same command as yesterday and it worked flawless. Exeption seems amiga. When i scrape with screenscraper it guits :) after 30 fails.

muldjord

@analoghero They probably fixed their servers then. What files do you use for Amiga? If you use .uae files it won't work with screenscraper at all. If you have ADF's it might work for some of them. But I strongly recommend using openretro instead.

Skyscraper is set to quit if the first 30 consecutive entries don't return anything. It makes no sense to continue since something is obviously wrong at that point (unless you have some really rare roms as the first 30 games, which would be extremely rare).

AnalogHero

@muldjord Then its fine. Im also using .uae files with HoraceandSpiders whdload autobooter.

SammyBoy

This post is deleted!

dudleydes

@muldjord I am getting a build error when trying to compile on Linux Mint 17 Qiana Cinnamon 64-bit. Running dpkg-query -l qt5* confirms that I have qt5-default installed (version 5.2.1+dfsg-1). The error occurs when compiling src/fxshadow.cpp. Full output when running make with error message can be found at https://pastebin.com/b8805LVr.

mitu

@dudleydes Seems that the error comes from a function introduced in a newer version of Qt5 (5.3) - http://doc.qt.io/qt-5/qcolor.html#qPremultiply.

dudleydes

@mitu Thanks for the reply. Mint 17 appears to only have version 5.2.1 available in the Software Manager so I'll look to upgrade to Mint 18.

SammyBoy

@muldjord you should consider asking @herb_fargus to allow Skyscraper to be installed from inside the RetroPie menu

muldjord

@dudleydes Yes, Skyscraper requires Qt5.3. I'll add a note about that in the instructions.

@SammyBoy I would love to have Skyscraper in the setup script. I think with the new "simple mode" it would make sense. So it should just compile from source and run it in simple mode. I will look into it at some point.

SammyBoy

@muldjord Did you read my message on Reddit?