Versatile C++ game scraper: Skyscraper

muldjord

@rion If they are major platforms that are related to consoles or home computers I will consider it. But I do have a lot of requests to add obscure "homebrew" and or / sub-platforms that I do not wish to support as they are often either too obscure or already supported by "arcade" or others.

Mainly Skyscraper was made to support the RetroPie naming scheme and platforms. I have, however, added some platforms besides that that I felt made sense. I am quite restricting when it comes to adding more, as I don't want to bloat Skyscraper too much.

But if you have a couple of platforms that you feel could fall under the RetroPie umbrella, I'd need the same info as found on the RetroPie supported platform pages, which is:

Platform name and shortname (eg. "Nintendo Entertainment System" / "nes")
File extensions (*.this, *.that)

But no promises I'll add them. :)

Obi-Wan

Not sure where to post this, but I felt it should be mentioned somewhere.

Scraping does not work on Windows with version 3.6.7, but works fine with the previous version, 3.5.8.

I was so happy to find Skyscraper because it seemed like a really nice tool, so I was a bit bummed out about the lack of Windows-support and even more so when I couldn't get it to work. But the second I tried the older 3.5.8 version scraping worked immediately.

I get the same behavior on two separate Windows systems. This reproduces the behavior on both my systems:

Download Skyscraper and unzip to C:\work\
Copy contents of the deploy folder to my user-folder
Open Powershell in Skyscraper-folder (cmd has the same behavior)
In PS, run: .\Skyscraper.exe -s thegamesdb -p snes -i "C:/work/ROMS"

3.6.7 returns Game not found
3.5.8 works fine.

I have no idea if I've missed something else here, but with 3.6.7 I got nowhere and with 3.5.8 everything seems to work as intended.

muldjord

@obi-wan I might look into this one of the coming days. And thank you for the detailed description, that is highly appreciated.

muldjord

@obi-wan Please redownload it from the same link. It was missing an OpenSSL dll that I've now included. I've tested it on my Windows dev machine and it works now. Let me know if it works for you too.

Obi-Wan

@muldjord Thanks a bunch! :) I was thrown off my emulator project for a while, but I'm back at it and 3.6.7 seems to work perfectly well now! Appreciate it, especially because the Windows version is not really supported!

muldjord

@obi-wan Yeah, I don't really support it beyond releasing it. But I at least want it to work, so thanks for reporting that. I don't really use it myself so... :D

muldjord

Quick show of hands: Would anyone be against me removing the "Simple mode" from Skyscraper? The questionnaire mode that it runs if you run Skyscraper without command line parameters. I haven't updated it in ages and it seems that it's currently broken. I never liked it, and I kind of want to remove it completely.

Given that the RetroPie Skyscraper script is much better I feel like "Simple mode" is deprecated.

Clyde

@muldjord Since I never noticed it until you mentioned it now, I don't mind. 😉

muldjord

Thanks guys. I will remove it in the next release, unless someone stops by within the next week with compelling arguments to keep it.

johnodon

I've been reading through the docs and haven't found my answer (or if it is possible)...

I like to save all of my media to a single folder using file suffixes (like sselph scraper does). For example in /roms/arcade/images/ I would have:

zaxxon-image.png
zaxxon-video.mp4
zaxxon-marquee.png
zaxxon-launching.png

Is this possible with Skyscraper?

TIA.

John

muldjord

@johnodon Hi John, You can change the output folder with the option described here, but not the names of the media files themselves.

TomFury

Hi! First things first - I'm impressed with your work @muldjord and I really like Skyscraper a lot! I was looking for a way to handle one specific use case and was advised to post in this thread by @mitu (thank you). I would like to propose a feature suggestion for Skyscraper. I detailed my use case in this thread.

The ghist of it is a feature where one could provide a list of files that Skyscraper should avoid scraping completely, i.e.. some kind of "unallowed list". This is based on the fact that there are roms in the collection I know will not render any results, and I would like to avoid hammering the online services for these "known failures".

I realize after the replies in the thread that one could use the --excludefiles option. The caveat there is that one would have to assemble (and maintain!) a list of files manually into a string that is passed to the option (or, as also suggested in the thread - change the file names to match a certain pattern for files one would like to avoid).

It would be really neat if one could for example pass a "skipped-<platform>-<source>.txt"-list to Skyscraper when instructing it to scrape for a platform. Perhaps the --excludefiles option could be extended to also take such a file as input?

muldjord

@tomfury Hi, thanks for the suggestion. I've implemented this in my local development version now. I just need to test it some more, then I will release it.

It will work by using either the CLI parameter --excludefrom FILENAME or by setting excludeFrom="FILENAME" in the config.ini [main] section or a [PLATFORM] section.

The FILENAME must contain a list of filenames with full path.

I will post here when it is ready.

muldjord

Skyscraper 3.7.0 out now: https://github.com/muldjord/skyscraper

Moved '--fromfile' option to '--includefrom'. '--fromfile' still works, but is considered deprecated
Moved '--includefiles' option to '--includepattern'. '--includefiles' still works, but is considered deprecated
Moved '--excludefiles' option to '--excludepattern'. '--excludefiles' still works, but is considered deprecated
Added '--excludefrom' option similar to '--includefrom' only the opposite (Thank you to user 'TomFury' for suggesting this)
Skyscraper will now ignore any subfolders within the input folder where a file called '.skyscraperignore' is found (Thank you to user 'sromeroi' for suggesting this)
Added platform 'easyrpg', only usable using the 'screenscraper' scraping module (Thank you to user 'zerojay' for suggesting this)

Documentation for the CLI versions of the new options is here. config.ini documenation is here.

If you run into issues, please let me know. Thanks! :)

TomFury

@muldjord Thank you so much! This is a great add-on. I have tested it on version 3.7.2 and it looks like it works! I kind of had the expectation that one could use the generated "skip-list" as an input, but I think using full path works just as great (and also provide more control).

For my use case the only missing step now is to be able to convert the skipped-<platform>-<scraper>.txt files from a scraping session to an input file compatible with the --excludefrom flag (i.e. for every row change:

'<rom name>', No returned matches

... to:

/full/path/to/<rom name>.<extension>

... and populate my "final" list for excluded files. Perhaps that would be too much to ask for Skyscraper to handle. My use case might be a bit odd, and in that case it's probably better to create an external adapter-script that will do that kind of conversion for me. I mean, I guess most users want to have total control of a list which is used as input instead of relying on a list of unsuccessful scrapes (which might have failed for many reasons; not only that the rom is missing from the database).

sleve_mcdichael

@muldjord said in Versatile C++ game scraper: Skyscraper:

Moved '--fromfile' option to '--includefrom'. '--fromfile' still works, but is considered deprecated

When I use this option with a relative path, no matter where from, it is expanded relative to ~/.skyscraper and not to {cwd}. This makes it impossible to use tab-completion unless I: 1) use absolute path, or 2) pushd or cd into the .skyscraper dir first.

(It's not new, --fromfile used to do this also.)

Is this by design?

muldjord

@sleve_mcdichael said in Versatile C++ game scraper: Skyscraper:

Is this by design?

Well, no, not really. And I can see how that is confusing. I'll change it.

@TomFury That was actually the idea and I thought I checked the skipped files, but I probably checked the report files instead. I think I will change the skipped files to include filenames instead and cross my fingers people won't miss the old format.

EDIT: 3.7.4 now out with both of these things fixed.

TomFury

@muldjord said in Versatile C++ game scraper: Skyscraper:

@TomFury That was actually the idea and I thought I checked the skipped files, but I probably checked the report files instead. I think I will change the skipped files to include filenames instead and cross my fingers people won't miss the old format.

EDIT: 3.7.4 now out with both of these things fixed.

Awesome! I will give it a try the nearest couple of days :-).

Clyde

@muldjord Thanks for the update! It's great that you still find the time to work on this. 👍

muldjord

Decided to remove the code that replaces : in the Pegasus command string. I can't remember why I put it in there, but things seem to work without it. So maybe I put it in there before I added the newer Pegasus parser (which seem to work with the : in place).

So 3.7.5 released where : is now allows in Pegasus launch command. It's been requested a few times.