Versatile C++ game scraper: Skyscraper

Clyde

(Sometimes, I want to say that literally instead "only" upvoting the helpful post. 😉 )

ecto

Hi, thanks a lot for your truly amazing and versarile scraper! I just wanted to let you know, that I just came across a small problem, which is related to my specific setup:

I start my Dosbox (pc) games with a custom made start script that launches certain *.init files in which the names of the executables for Dosbox are stored.

I noticed that in my gameslist.xml some games are named the same and also have the same artwork etc. Now I found out why: Skyscraper uses sha1 for cross referencing its database with the *.init files. But since these init files are small and some games use the same command to run (like game.exe or play.bat or ultima.com), they are treated as one by Skyscraper.

My workaround would now be to look for *.init files with the same content and maybe add a comment line to change the sha1 code... This of course would probably mean that I have to rescrape these games as the artwork uses the sha1 value as its name and thus is probably overwritten.

muldjord

@ecto said in Versatile C++ game scraper: Skyscraper:

Hi, thanks a lot for your truly amazing and versarile scraper!

Glad it proves useful! :)

I noticed that in my gameslist.xml some games are named the same and also have the same artwork etc. Now I found out why: Skyscraper uses sha1 for cross referencing its database with the *.init files.

Yes, this is correct. *.init files are not known by default by Skyscraper. I suggest using .sh or .bat instead. These are known by Skyscraper to be script files and the cache id will be calculated from the filename instead. I've added *.init files to this list and it will work for you in the next release (not yet released, no current ETA as I'm working on some Attract-Mode stuff as well). Just keep in mind that you'll need to redownload the data for those games as they are currently wrongly identified in your cache.

mitu

@muldjord I updated the Skyscraper RetroPie scriptmodule to add the priorities.xml.example to the cache folder.

ecto

@muldjord Oh wow! Thanks for this and the explanation!

I know, .init files are not the standard, that's why I included the extensions= *.init line in the [pc] section of the config.ini file of Skyscraper. It seems it would have been better to use .bat or something, but at the time I didn't think about that and now I don't want to go back.

Just so I get it right: The hash calculation is not used for .sh and .bat files. Are there other file types without hash calculation?

Anyways, I love scraping using Skyscraper and I "finished" my pc/dosbox games yesterday with the little workaround to change the sha1 values by adding a comment line in the .init file and then rescrape. Thanks again for your help!

muldjord

@ecto said in Versatile C++ game scraper: Skyscraper:

Just so I get it right: The hash calculation is not used for .sh and .bat files. Are there other file types without hash calculation?

The following is the code that shows the file extensions that are calculated from the filename instead of the contents:

  if(info.suffix() == "uae" || info.suffix() == "cue" ||
     info.suffix() == "sh" || info.suffix() == "svm" ||
     info.suffix() == "scummvm" || info.suffix() == "mds" ||
     info.suffix() == "zip" || info.suffix() == "7z" ||
     info.suffix() == "gdi" || info.suffix() == "ml" ||
     info.suffix() == "bat" || info.suffix() == "init") {

Anyways, I love scraping using Skyscraper and I "finished" my pc/dosbox games yesterday with the little workaround to change the sha1 values by adding a comment line in the .init file and then rescrape. Thanks again for your help!

Then I will remove *.init from the above code again. Otherwise I'll break your cache in the next version.

ecto

@muldjord Yes, of course, that makes sense! Thanks again!

muldjord

Skyscraper 3.3.4 released: https://github.com/muldjord/skyscraper

'igdb' module global key has been removed from Skyscraper. They have moved to free personal keys instead (this is good news!). To get one, register at https://api.igdb.com (Thank you to 'stlbluesfan' for letting me know)
Fixed disk space check so it also works for 'attractmode' frontend
Updated 'attractmode' to save overviews in correct location and format (Thanks to 'barolo' for letting me know, thanks to 'teeedubb' and 'barolo' for testing)
For 'attractmode' added semicolon detection for 'snaps'/'video' parsing for determining resource folders (Thanks to 'barolo' for pointing this out)

This is mostly a release to fix some AttractMode related export issues. But it also includes the change in the IGDB module that now craves a personal API key instead of the limited global one. This means that you can now get your own private key for IGDB and use that instead, giving everyone better access.

Clyde

@muldjord Great news! How do I store my personal IGDB key in the config files?

https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md#IGDB isn't updated yet in this matter.

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord Great news! How do I store my personal IGDB key in the config files?

Just like any of the other modules that need user credentials. In this case it's just a single key like this:

[igdb]
userCreds="<KEY>"

I will update the doc, thanks!

muldjord

Skyscraper 3.3.5 released: https://github.com/muldjord/skyscraper

Updated 'screenscraper' API to support new system name json format (Thank you to 'Micket' for letting me know)

This is an important release! The 'screenscraper' API has changed a tiny bit, which broke the Skyscraper implementation. So this release is needed to make it work again.

Please spread the word if you hear of anyone having 'screenscraper' issues with a pre-3.3.5 version. This version will fix it for them!

And thanks to 'Micket' for letting me know. I don't use Skyscraper on a daily basis, so I rely on users to report such API issues to me.

Clyde

@muldjord Thank you for your ongoing work.

One (seemingly) small feature request. 😌 Next to the --symlink option, a --hardlink option would be great. Hardlinks would combine the benefits of saved storage space like symlinks, but would also work like normal files if the roms and/or media directories are copied to another medium. Would that be feasible for a coming update?

mitu

Actually, using a hardlink has a few limitations and caveats:

a hardlink cannot be created across filesystems - it's only between files on the same FS.
if you remove the source, then re-create it (with the same name), the hardlink would still contain the 1st source data, while the symlink would still work.

muldjord

@Clyde The framework I use does not support creating hardlinks only symlinks.

Clyde

@mitu Yes, both variants have their pitfalls (like symlinks' dependence on the source). I just would have liked to have both as options. :)

@muldjord A pity, but thanks for clearing it up.

steptoe

Hi,

Just wondering if you have any plans to add support for pinball in your scraper

I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

Mainly Future Pinball and Visual Pinball/PinMAME

Thanks

Clyde

@muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

steptoe

I'm trying to compiler the latest Skyscraper 3.3.5 for Windows using QT as described here :

*You can easily cross-compile it for Windows. Just download and install Qt 5 from the official site (open source version). When installing, put a checkmark in the latest mingw32 version, I don't support MSVC++ although it might work, not sure. When it's installed you'll find a Qt specific command prompt in your start menu. From there navigate to the Skyscraper source folder and run:
$ qmake
$ mingw32-make release

That will give you a compiled version in a subfolder called "release". Then you need to copy the needed dll's from the qt5 install folder into the release folder and start an ordinary command prompt there. Then you should be able to run it.*

Running the QT Creator and trying various new projects and not finding how to point at the Skyscraper source folder to compile a windows version

My intention is then look into using other 3rd party software to create a simple GUI for us Windows users. I use SkraperUI but Skyscraper gets better and better as I've followed its development for RetroPie which I also use via RetroSMC on our Pi/KODI box running OSMC

Thanks

muldjord

@steptoe said in Versatile C++ game scraper: Skyscraper:

Just wondering if you have any plans to add support for pinball in your scraper

No current plans, no.

I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

Skyscraper doesn't support emumovies as they don't have an open API. At least not to my knowledge. I've been looking for one but never found one. It has to be some kind of REST-based API, not just an ftp with images or something like that.

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

That should be reported to the module that delivered the wrong data. I suggest registering with them and fixing it yourself if you can. That's the power of community based game info databases.