Versatile C++ game scraper: Skyscraper

Clyde

@muldjord Excuse me for jumping in on that, but I just started using Skyscraper. Will it complete missing resources from already cached games? If, say, a game has everything cached but a Video, will SS download the Video on the next run (without refresh) if it becomes available?

I'm asking because AFAIK, Sselph's scraper which I used up until now doesn't do that for games that are already in the gamelist. It doesn't use a cache like SS does, though.

muldjord

@Clyde No, if a game has previously provided a result for any resource, it won't fill in the holes unless you run a refresh on it. This is how the databases work. If you look up a game on any of the databases, it will give you most, if not all, of the info in one request. At that point it only makes sense to use all of it or not look up the game again at all.

The idea is to use the --cache edit or the import module feature to fill in the holes. You can use the --cache report:missing=<RESOURCE> to figure out what your games are missing. And you can use the --cache edit --fromfile <REPORT FILE> afterwards to edit only the games missing any of the resources in the reports.

dudumaroja

@muldjord thanks very much.. the refresh issue is more of the retropie menu problem i guess.. it was enabled by default for me ( never messed if it.. ) and i didnt know about the option until you pointed it to me.. ( i was even adenventuring myself to learn how to use your program without the retropie-setup menu, and just leaned about --startat and was about to use it.

another question.. im seen to have problem with modified roms. ( translation and fixes.. ) but i can see to find a option to scrapt theses rom by name.

could you provide-me a example on how to do it? im using the proper no-intro names.. but the hashes as messed up..

and again... thanks for the great work. i loved the ideia on have a cache for personal re-scraping.. i set a network share so it wont waste space on my SD or usb drive. and it works amazing.. i wish there was a tool so we can acess the cache data.. and make alterations ouselfs.. ( and maybe send it back later to screenscraper. )

Clyde

@muldjord Thanks for the explanation. So, a game can only have all of its resources from one source, which will be overwritten by one single other source on a refresh?

The source in brackets when scraping subsequently (e.g. Screenshot: YES (screenscraper)) made me hope that you can have resources from different sources.

If I do understand you correctly, why can't SS fill in the gaps locally even if the online databases give you all resources in one request? The resources are saved in separate files, after all.

I'm still hoping that I misanderstand you. ;)

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord Thanks for the explanation. So, a game can only have all of its resources from one source, which will be overwritten by one single other source on a refresh?

No, it can have one of each resource per scraping module. Please check out the documentation it's all in there

The source in brackets when scraping subsequently (e.g. Screenshot: YES (screenscraper)) made me hope that you can have resources from different sources.

It does. :) It even allows for user resources on top of that with the --cache edit option where you can add new textual resources yourself that will be prioritized above all other resources. You can also prioritize the resources from all of the other databases with the priorities.xml files per platform. It's also documented, you'll find it. :)

If I do understand you correctly, why can't SS fill in the gaps locally even if the online databases give you all resources in one request? The resources are saved in separate files, after all.

It does, but there's no difference between filling in the holes and filling in everything. The point here being that if you are refreshing the data for a game, you will always be interested in filling in everything, since everything is already part of the data from the servers. Only filling in the holes would be a huge waste of database bandwidth.

muldjord

@dudumaroja said in Versatile C++ game scraper: Skyscraper:

@muldjord thanks very much.. the refresh issue is more of the retropie menu problem i guess.. it was enabled by default for me ( never messed if it.. ) and i didnt know about the option until you pointed it to me..

I'm pretty sure it's not enabled by default unless @mitu changed that. It shouldn't be.

another question.. im seen to have problem with modified roms. ( translation and fixes.. ) but i can see to find a option to scrapt theses rom by name.

Depends on the scraping module. Some of the modules use a file name search based query (and will just do so, you don't need to do anything special to enable it), others use checksums. You can read more about all of them here

and again... thanks for the great work. i loved the ideia on have a cache for personal re-scraping.. i set a network share so it wont waste space on my SD or usb drive. and it works amazing.. i wish there was a tool so we can acess the cache data.. and make alterations ouselfs.. ( and maybe send it back later to screenscraper. )

You're welcome. There's plenty of options to edit the cache. :) Check out the --cache help output from a terminal. It can't send it back to the sources though. You can also use the -s import module to add your own artwork resources. That's also well-documented.

Clyde

@muldjord Thanks again – for the information and your very polite RTFM. :) I admit that asking here seemed more convenient than looking it up by myself, which I will do now with the proper amount of guilty conscience.

I also want to thank you for this great piece of work and your ongoing support. Like @dudumaroja, I especially like the caching concept.

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

I'm pretty sure it's not enabled by default unless @mitu changed that. It shouldn't be.

Yes, it's not enabled by default - when the configuration is first generated, the refresh is disabled - https://github.com/RetroPie/RetroPie-Setup/blob/922149b09dfab04408e133114bcb2a97cf7ed87c/scriptmodules/supplementary/skyscraper.sh#L358.

muldjord

@Clyde Yes, caching the data is extremely important for any scraper if you ask me. :) Have fun with it!

@dudumaroja, see mitu's answer above. :)

muldjord

Skyscraper 3.3.0 released: https://github.com/muldjord/skyscraper

MAJOR: File identification now uses new quick id method for up to 75% faster processing (Thank you to 'langest' for finally making me look into this)
Added 'pc98' platform (Thank you to 'leosmeira' for suggesting it)
Added 'pokemini' platform (Thank you to 'leosmeira' for suggesting it)
Renamed all 'sha1' file id's to 'id' as sha1 was misleading
Changed relevant defines to constexpr
ScreenScraper now always prioritizes the 'video-normalized' above 'video' (Thank you to 'JuanVCS' for suggesting this)
Fixed bug in ScreenScraper retry code which made it retry more than necessary

Most prominent feature is the new quick id concept. So far Skyscraper has calculated the internal cache id's from the data of the roms every time you run Skyscraper. This seemed like a lot of work for files that rarely change, and takes a long time to do, especially for the larger roms such as n64 roms. So, I've made a new system called quick id which is basically an id caching that works seemlessly with the cache id system. Here's how it works:

Whenever a file processed for anything using the cache id, the cache id is calculated using the "slow" method. And at this time, the file path and file modified time is also saved in a quick id xml file. Then, the next time this particular file needs processing, it will first compare the file path and modified time from the quick id list to the actual file. If the file is still located at the same place and hasn't been modified since last run, it will look up the cache id from the quick id list.

So everything will be "as slow as usual" the first time you process any file, since it is then forced to use the "slow" id technique as it doesn't yet know the id. But for any subsequent processing of that particular file, you will see up to a 75% improvement in speed (the larger the roms, the speedier it will be). A great platform to test it on is the n64 platform. But remember, after updating, you won't see the speed improvement until you've processed the files one way or another once so the id's get cached in the quick id list.

A great way to test this is to update to 3.3.0, then regenerate the gamelist for any platform with Skyscraper -p <PLATFORM> and take note of how long it took . When that's done do it again and take note of the time again. It should have taken significantly shorter time the second time you ran it, as it was now able to use the quick id feature.
The quick id list can be vacuumed together with the cache with the --cache vacuum option to avoid having lingering quick id's if you move files around or rename them.

A lot of code has changed in this version. I've fixed some silly bugs in the ScreenScraper module aswell which made it retry more times than necessary. If you encounter any problems with any of the above, please let me know. As always I've been testing this quite a bit on my own systems, but it's impossible for me to test any use case.

Clyde

@muldjord said in Versatile C++ game scraper: Skyscraper:

ScreenScraper now always prioritizes the 'video-normalized' above 'video'

If I wanted to purge all videos from the cache to re-scrape their normalized versions, would this be the right command? (run from whichever location of the Skyscraper executable)

Skyscraper -p PLATFORM --cache purge:video

muldjord

@Clyde Almost. You need to run Skyscraper -p PLATFORM --cache purge:t=video,m=screenscraper to purge all videos from ScreenScraper. But you can just rescrape it with --refresh. Then it will overwrite the old videos automatically.

Clyde

@muldjord Thank you. But wouldn't --refresh re-scrape all resources instead of only the videos? Although the videos are the biggest resource, I would like to avoid re-scraping every other resource as well.

muldjord

@Clyde Long story short, that's not how it works. It always downloads "everything or nothing" due to my current implementation of the cache and how the online databases work.

Clyde

@muldjord That's what I feared after reading the docs, thanks again. One last question (I hope): If I purge the videos only, then only they would be re-scraped in a normal scraping run without --refresh, right?

If so, I'd rather use that, since I would very much like to have the normalized videos, but with as little stress to the SS servers as possible. I'm already donating 5€/month to them, but I don't see that as a free ticket to do whatever I like. 😇

If it ever happens that you don't know what else to do on Skyscraper, more control over refreshing the cache would be nice (if the online servers allow it). 😊

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord That's what I feared after reading the docs, thanks again. One last question (I hope): If I purge the videos only, then only they would be re-scraped in a normal scraping run without --refresh, right?

No. ;) Then it will just fetch the game from the cache without the video. Entries are only ever redownloaded from the online sources if --refresh is set. Otherwise it fetches what it has from the cache.

If it ever happens that you don't know what else to do on Skyscraper, more control over refreshing the cache would be nice (if the online servers allow it). 😊

I am painfully aware :D But it's not a small change as my current implementation doesn't support this. I have ideas for it, and it might happen at some point, but don't hold your breath for it. :) Maybe for 4.0.0 at some point down the line.

Clyde

Thanks a final time in this matter. I'll register this as one of the many things we'll have to live with in an imperfect world, then. :)

muldjord

Skyscraper 3.3.1 released: https://github.com/muldjord/skyscraper

Added new '--cache edit:new=<TYPE>' option for efficient batch adding of a resource of a certain type
Improved ctrl+c handling
Protected cache write calls from ctrl+c obstruction

If you've created reports with missing textual resources for certain types, it is now easy to add those resource using the Skyscraper -p <PLATFORM> --cache edit:new=<TYPE> --fromfile <REPORTFILE>. This will only edit the roms from the report, and move directly through each of them asking for that particular resource type to be added. Much easier and faster than before.

Clyde

@muldjord I just looked at your priorities.xml.example. The order of entries for each section seems rather random. Is it so for exemplary reasons, or what did you base the arrangements on? It doesn't always seem to follow your suggestions in https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md.

Also, this file doesn't seem to be installed with the Retropie build of Skyscraper. I had to get it directly from your Github repository. Is this intentional, and if so, why?

Thanks in advance for any light shed on these.

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord ...what did you base the arrangements on?

It is based on a bunch of tests I did 2 and a half years back and some adjustments I've made to it since after adding new modules. Basically it is a subjective thing and people are supposed to define it themselves. It works well for me as it is but as it is a global default it will of course work better for some than others. Keep in mind that if modules are missing, it doesn't mean the data won't be used, then it's just decided by timestamp.

It doesn't always seem to follow your suggestions in https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md.

If you have suggestions for changes to the defaults I'll gladly hear about it. Just keep in mind that it has to work well for all platforms.

Also, this file doesn't seem to be installed with the Retropie build of Skyscraper. I had to get it directly from your Github repository. Is this intentional, and if so, why?

I thought it did. Maybe @mitu can shed some light on this. He's the script author.