Versatile C++ game scraper: Skyscraper

[Clyde]

@muldjord That's what I feared after reading the docs, thanks again. One last question (I hope): If I purge the videos only, then only they would be re-scraped in a normal scraping run without --refresh, right?

If so, I'd rather use that, since I would very much like to have the normalized videos, but with as little stress to the SS servers as possible. I'm already donating 5€/month to them, but I don't see that as a free ticket to do whatever I like. 😇

If it ever happens that you don't know what else to do on Skyscraper, more control over refreshing the cache would be nice (if the online servers allow it). 😊

[muldjord]

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord That's what I feared after reading the docs, thanks again. One last question (I hope): If I purge the videos only, then only they would be re-scraped in a normal scraping run without --refresh, right?

No. ;) Then it will just fetch the game from the cache without the video. Entries are only ever redownloaded from the online sources if --refresh is set. Otherwise it fetches what it has from the cache.

If it ever happens that you don't know what else to do on Skyscraper, more control over refreshing the cache would be nice (if the online servers allow it). 😊

I am painfully aware :D But it's not a small change as my current implementation doesn't support this. I have ideas for it, and it might happen at some point, but don't hold your breath for it. :) Maybe for 4.0.0 at some point down the line.

[Clyde]

Thanks a final time in this matter. I'll register this as one of the many things we'll have to live with in an imperfect world, then. :)

[muldjord]

Skyscraper 3.3.1 released: https://github.com/muldjord/skyscraper

• Added new '--cache edit:new=<TYPE>' option for efficient batch adding of a resource of a certain type
• Improved ctrl+c handling
• Protected cache write calls from ctrl+c obstruction

If you've created reports with missing textual resources for certain types, it is now easy to add those resource using the Skyscraper -p <PLATFORM> --cache edit:new=<TYPE> --fromfile <REPORTFILE>. This will only edit the roms from the report, and move directly through each of them asking for that particular resource type to be added. Much easier and faster than before.

[Clyde]

@muldjord I just looked at your priorities.xml.example. The order of entries for each section seems rather random. Is it so for exemplary reasons, or what did you base the arrangements on? It doesn't always seem to follow your suggestions in https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md.

Also, this file doesn't seem to be installed with the Retropie build of Skyscraper. I had to get it directly from your Github repository. Is this intentional, and if so, why?

Thanks in advance for any light shed on these.

[muldjord]

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord ...what did you base the arrangements on?

It is based on a bunch of tests I did 2 and a half years back and some adjustments I've made to it since after adding new modules. Basically it is a subjective thing and people are supposed to define it themselves. It works well for me as it is but as it is a global default it will of course work better for some than others. Keep in mind that if modules are missing, it doesn't mean the data won't be used, then it's just decided by timestamp.

It doesn't always seem to follow your suggestions in https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md.

If you have suggestions for changes to the defaults I'll gladly hear about it. Just keep in mind that it has to work well for all platforms.

Also, this file doesn't seem to be installed with the Retropie build of Skyscraper. I had to get it directly from your Github repository. Is this intentional, and if so, why?

I thought it did. Maybe @mitu can shed some light on this. He's the script author.

[mitu]

@muldjord said in Versatile C++ game scraper: Skyscraper:

I thought it did. Maybe @mitu can shed some light on this. He's the script author.

It's not included because at that time it didn't exist (?), but since it should be copied to the cache folder and that folder might not exit during installation, I can maybe add it directly to the normal Skyscraper folder.

[muldjord]

@mitu Skyscraper looks for it in the cache folder, and copies it to the platform subfolders inside the cache folder as a default. If it doesn't exist in the cache folder it just ignores it and only ever prioritizes by timestamp. That can cause some weird behaviour. The best result would be achieved if a cache folder was created upon installation and the priorities.xml.example file was copied there.

EDIT: Alternatively I can make Skyscraper look for it in ~/.skyscraper directly as well and copy it from there. Let me know if creating a cache folder is an issue, then I will implement this.

[mitu]

@muldjord said in Versatile C++ game scraper: Skyscraper:

EDIT: Alternatively I can make Skyscraper look for it in ~/.skyscraper directly as well and copy it from there. Let me know if creating a cache folder is an issue, then I will implement this.

Don't need to modify the behaviour - I'll check again the scriptmodule and we'll find a way to add it.

[Clyde]

Thank you both very much. I'm happy that I could contribute a tiny little bit by mentioning the missing file. :) I stumbled upon it when I wondered how skyscraper would choose which artwork it will use if multiple sources have been scraped, which I did for the first time last weekend. I happily noticed that Skyscraper will fill the slots from different sources in case of missing content from the preferred one.

@muldjord Now that I know more thanks to you, I'll look into the priorities and decide if I want to change any of them. If I come up with anything I deem useful universally, I'll post it here.

[Clyde]

@muldjord said in Versatile C++ game scraper: Skyscraper:

Skyscraper looks for it in the cache folder, and copies it to the platform subfolders inside the cache folder as a default.

When does it do that? And does it update the copies if the example file was changed?

Thanks.

[muldjord]

@Clyde said in Versatile C++ game scraper: Skyscraper:

When does it do that?

Whenever you run it and it doesn't already have an existing priorities.xml in the platform cache/<platform> folder it will look for cache/priorities.xml.example and copy it to the folder as priorities.xml to have a decent default to use.

And does it update the copies if the example file was changed?

No. If one already exists, it never copies it.

[Clyde]

@muldjord Thanks!

(Sometimes, I want to say that literally instead "only" upvoting the helpful post. 😉 )

[ecto]

Hi, thanks a lot for your truly amazing and versarile scraper! I just wanted to let you know, that I just came across a small problem, which is related to my specific setup:

I start my Dosbox (pc) games with a custom made start script that launches certain *.init files in which the names of the executables for Dosbox are stored.

I noticed that in my gameslist.xml some games are named the same and also have the same artwork etc. Now I found out why: Skyscraper uses sha1 for cross referencing its database with the *.init files. But since these init files are small and some games use the same command to run (like game.exe or play.bat or ultima.com), they are treated as one by Skyscraper.

My workaround would now be to look for *.init files with the same content and maybe add a comment line to change the sha1 code... This of course would probably mean that I have to rescrape these games as the artwork uses the sha1 value as its name and thus is probably overwritten.

[muldjord]

@ecto said in Versatile C++ game scraper: Skyscraper:

Hi, thanks a lot for your truly amazing and versarile scraper!

Glad it proves useful! :)

I noticed that in my gameslist.xml some games are named the same and also have the same artwork etc. Now I found out why: Skyscraper uses sha1 for cross referencing its database with the *.init files.

Yes, this is correct. *.init files are not known by default by Skyscraper. I suggest using .sh or .bat instead. These are known by Skyscraper to be script files and the cache id will be calculated from the filename instead. I've added *.init files to this list and it will work for you in the next release (not yet released, no current ETA as I'm working on some Attract-Mode stuff as well). Just keep in mind that you'll need to redownload the data for those games as they are currently wrongly identified in your cache.

[mitu]

@muldjord I updated the Skyscraper RetroPie scriptmodule to add the priorities.xml.example to the cache folder.

[ecto]

@muldjord Oh wow! Thanks for this and the explanation!

I know, .init files are not the standard, that's why I included the extensions= *.init line in the [pc] section of the config.ini file of Skyscraper. It seems it would have been better to use .bat or something, but at the time I didn't think about that and now I don't want to go back.

Just so I get it right: The hash calculation is not used for .sh and .bat files. Are there other file types without hash calculation?

Anyways, I love scraping using Skyscraper and I "finished" my pc/dosbox games yesterday with the little workaround to change the sha1 values by adding a comment line in the .init file and then rescrape. Thanks again for your help!

[muldjord]

@ecto said in Versatile C++ game scraper: Skyscraper:

Just so I get it right: The hash calculation is not used for .sh and .bat files. Are there other file types without hash calculation?

The following is the code that shows the file extensions that are calculated from the filename instead of the contents:

  if(info.suffix() == "uae" || info.suffix() == "cue" ||
     info.suffix() == "sh" || info.suffix() == "svm" ||
     info.suffix() == "scummvm" || info.suffix() == "mds" ||
     info.suffix() == "zip" || info.suffix() == "7z" ||
     info.suffix() == "gdi" || info.suffix() == "ml" ||
     info.suffix() == "bat" || info.suffix() == "init") {

Anyways, I love scraping using Skyscraper and I "finished" my pc/dosbox games yesterday with the little workaround to change the sha1 values by adding a comment line in the .init file and then rescrape. Thanks again for your help!

Then I will remove *.init from the above code again. Otherwise I'll break your cache in the next version.

[ecto]

@muldjord Yes, of course, that makes sense! Thanks again!

[muldjord]

Skyscraper 3.3.4 released: https://github.com/muldjord/skyscraper

• 'igdb' module global key has been removed from Skyscraper. They have moved to free personal keys instead (this is good news!). To get one, register at https://api.igdb.com (Thank you to 'stlbluesfan' for letting me know)
• Fixed disk space check so it also works for 'attractmode' frontend
• Updated 'attractmode' to save overviews in correct location and format (Thanks to 'barolo' for letting me know, thanks to 'teeedubb' and 'barolo' for testing)
• For 'attractmode' added semicolon detection for 'snaps'/'video' parsing for determining resource folders (Thanks to 'barolo' for pointing this out)

This is mostly a release to fix some AttractMode related export issues. But it also includes the change in the IGDB module that now craves a personal API key instead of the limited global one. This means that you can now get your own private key for IGDB and use that instead, giving everyone better access.