Versatile C++ game scraper: Skyscraper

muldjord

@retro81 In the current version it will fetch "flyer" for all arcade platforms including "neogeo". But I see that that might not be optimal. So I've removed "neogeo" from that list. From 3.6.7 it will use "box-2D" instead. Thanks for letting me know.

EDIT: 3.6.7 out now with this change included. Please update your Skyscraper version and rescrape the neogeo platform with cache refresh enabled (important!).

retro81

@muldjord thank you for both responses!

Rion

@muldjord Would you consider adding support for Batocera?

muldjord

@rion Just did a Google search and I assume you mean Batocera Linux, right? I won't personally support it, but I don't see why it wouldn't work already. As long as it has Qt5 included and a build chain, you should be able to install it using the description at the Skyscraper github. How to get Qt5 on there is up to the user to figure out. Maybe it's in the Batocera package manager already (if there is such a thing, I have never heard of the distribution before just now).

Rion

@muldjord You should really try it out. It's a great distribution both for sbc's but also for an x86_64 PC.

Skyscraper is not yet available in the Batocera pacman repository. But could probably be in the future.

What i wanted to know is if you would think/consider to implement support for more systems supported by Batocera. I'm completely fine if i could just use it on my Linux machine and scrape the roms there.

Cause i just love how the artwork looks with Skyscraper and there is no better alternative out there! 😃

https://wiki.batocera.org/emulators

muldjord

@rion If they are major platforms that are related to consoles or home computers I will consider it. But I do have a lot of requests to add obscure "homebrew" and or / sub-platforms that I do not wish to support as they are often either too obscure or already supported by "arcade" or others.

Mainly Skyscraper was made to support the RetroPie naming scheme and platforms. I have, however, added some platforms besides that that I felt made sense. I am quite restricting when it comes to adding more, as I don't want to bloat Skyscraper too much.

But if you have a couple of platforms that you feel could fall under the RetroPie umbrella, I'd need the same info as found on the RetroPie supported platform pages, which is:

Platform name and shortname (eg. "Nintendo Entertainment System" / "nes")
File extensions (*.this, *.that)

But no promises I'll add them. :)

Obi-Wan

Not sure where to post this, but I felt it should be mentioned somewhere.

Scraping does not work on Windows with version 3.6.7, but works fine with the previous version, 3.5.8.

I was so happy to find Skyscraper because it seemed like a really nice tool, so I was a bit bummed out about the lack of Windows-support and even more so when I couldn't get it to work. But the second I tried the older 3.5.8 version scraping worked immediately.

I get the same behavior on two separate Windows systems. This reproduces the behavior on both my systems:

Download Skyscraper and unzip to C:\work\
Copy contents of the deploy folder to my user-folder
Open Powershell in Skyscraper-folder (cmd has the same behavior)
In PS, run: .\Skyscraper.exe -s thegamesdb -p snes -i "C:/work/ROMS"

3.6.7 returns Game not found
3.5.8 works fine.

I have no idea if I've missed something else here, but with 3.6.7 I got nowhere and with 3.5.8 everything seems to work as intended.

muldjord

@obi-wan I might look into this one of the coming days. And thank you for the detailed description, that is highly appreciated.

muldjord

@obi-wan Please redownload it from the same link. It was missing an OpenSSL dll that I've now included. I've tested it on my Windows dev machine and it works now. Let me know if it works for you too.

Obi-Wan

@muldjord Thanks a bunch! :) I was thrown off my emulator project for a while, but I'm back at it and 3.6.7 seems to work perfectly well now! Appreciate it, especially because the Windows version is not really supported!

muldjord

@obi-wan Yeah, I don't really support it beyond releasing it. But I at least want it to work, so thanks for reporting that. I don't really use it myself so... :D

muldjord

Quick show of hands: Would anyone be against me removing the "Simple mode" from Skyscraper? The questionnaire mode that it runs if you run Skyscraper without command line parameters. I haven't updated it in ages and it seems that it's currently broken. I never liked it, and I kind of want to remove it completely.

Given that the RetroPie Skyscraper script is much better I feel like "Simple mode" is deprecated.

Clyde

@muldjord Since I never noticed it until you mentioned it now, I don't mind. 😉

muldjord

Thanks guys. I will remove it in the next release, unless someone stops by within the next week with compelling arguments to keep it.

johnodon

I've been reading through the docs and haven't found my answer (or if it is possible)...

I like to save all of my media to a single folder using file suffixes (like sselph scraper does). For example in /roms/arcade/images/ I would have:

zaxxon-image.png
zaxxon-video.mp4
zaxxon-marquee.png
zaxxon-launching.png

Is this possible with Skyscraper?

TIA.

John

muldjord

@johnodon Hi John, You can change the output folder with the option described here, but not the names of the media files themselves.

TomFury

Hi! First things first - I'm impressed with your work @muldjord and I really like Skyscraper a lot! I was looking for a way to handle one specific use case and was advised to post in this thread by @mitu (thank you). I would like to propose a feature suggestion for Skyscraper. I detailed my use case in this thread.

The ghist of it is a feature where one could provide a list of files that Skyscraper should avoid scraping completely, i.e.. some kind of "unallowed list". This is based on the fact that there are roms in the collection I know will not render any results, and I would like to avoid hammering the online services for these "known failures".

I realize after the replies in the thread that one could use the --excludefiles option. The caveat there is that one would have to assemble (and maintain!) a list of files manually into a string that is passed to the option (or, as also suggested in the thread - change the file names to match a certain pattern for files one would like to avoid).

It would be really neat if one could for example pass a "skipped-<platform>-<source>.txt"-list to Skyscraper when instructing it to scrape for a platform. Perhaps the --excludefiles option could be extended to also take such a file as input?

muldjord

@tomfury Hi, thanks for the suggestion. I've implemented this in my local development version now. I just need to test it some more, then I will release it.

It will work by using either the CLI parameter --excludefrom FILENAME or by setting excludeFrom="FILENAME" in the config.ini [main] section or a [PLATFORM] section.

The FILENAME must contain a list of filenames with full path.

I will post here when it is ready.

muldjord

Skyscraper 3.7.0 out now: https://github.com/muldjord/skyscraper

Moved '--fromfile' option to '--includefrom'. '--fromfile' still works, but is considered deprecated
Moved '--includefiles' option to '--includepattern'. '--includefiles' still works, but is considered deprecated
Moved '--excludefiles' option to '--excludepattern'. '--excludefiles' still works, but is considered deprecated
Added '--excludefrom' option similar to '--includefrom' only the opposite (Thank you to user 'TomFury' for suggesting this)
Skyscraper will now ignore any subfolders within the input folder where a file called '.skyscraperignore' is found (Thank you to user 'sromeroi' for suggesting this)
Added platform 'easyrpg', only usable using the 'screenscraper' scraping module (Thank you to user 'zerojay' for suggesting this)

Documentation for the CLI versions of the new options is here. config.ini documenation is here.

If you run into issues, please let me know. Thanks! :)

TomFury

@muldjord Thank you so much! This is a great add-on. I have tested it on version 3.7.2 and it looks like it works! I kind of had the expectation that one could use the generated "skip-list" as an input, but I think using full path works just as great (and also provide more control).

For my use case the only missing step now is to be able to convert the skipped-<platform>-<scraper>.txt files from a scraping session to an input file compatible with the --excludefrom flag (i.e. for every row change:

'<rom name>', No returned matches

... to:

/full/path/to/<rom name>.<extension>

... and populate my "final" list for excluded files. Perhaps that would be too much to ask for Skyscraper to handle. My use case might be a bit odd, and in that case it's probably better to create an external adapter-script that will do that kind of conversion for me. I mean, I guess most users want to have total control of a list which is used as input instead of relying on a list of unsuccessful scrapes (which might have failed for many reasons; not only that the rom is missing from the database).