Versatile C++ game scraper: Skyscraper

muldjord

Skyscraper 3.2.0 released: https://github.com/muldjord/skyscraper

Improved output of edit mode to show queue size and initial prioritized resources
Now clearly shows missing resources when adding resources in editing mode
Made '--cache vacuum' more verbose at verbosity level 2 or higher
Removed platform check for screenscraper module as it wasn't really necessary
Added 'videoSizeLimit' config variable (Thank you to 'silverfr0st' for suggesting this)
Now also processes brackets for skipped games (Thanks to 'hexive' for pointing this out)
Added 'arcadia' and 'astrocde' MESS platforms (Thank you to 'raphkoster' for suggesting these)
Added '--cache report:missing=<GROUP or RESOURCE1,2,...>' option (Thank you to JaymesRS for suggesting this)
Added '--fromfile' option allowing you to work on a subset of files taken from a file. Compatible with files created using the '--cache report' option
MobyGames module now includes 'platform' id query when relevant. Should improve result accuracy
Added region support for MobyGames cover artwork
Added A LOT of regions to region autodetection in general
Now no longer creates media folders when in resource gathering mode
Moved help texts for '--cache' options to '--cache help'

Lots of interesting stuff in this release. The --cache edit function has been improved a bit to better show what resources are missing on each rom. I've also added a nifty --fromfile <FILENAME> option which allows you to add filenames, one per line, into a file and use it with this option. Skyscraper will then work on those files only. This brings me to the new --cache report:missing option which allows you to create file reports containing filenames of all files missing certain resources. When those are created you can then combine the edit option with the fromfile option and add the missing resources like so --cache edit --fromfile /home/pi/.skyscraper/reports/reportfile.txt.

I've also improved region autodetection quite a bit. It will now detect pretty much all relevant regions and / or countries from the rom filename. Also added region support to mobygames covers!

As for minor stuff, some new platforms were added and some requested features aswell. Check the list above for all of it.

Let me know if you run into problems.

Have fun!

A Bored Ninja

@muldjord said in Versatile C++ game scraper: Skyscraper:

new platforms were a

@muldjord Thank you!

I'm excited to now be able to scrape my PC Engine CD roms, as I was getting a "Platform not set" error before.

Keep up the good work!

muldjord

@A-Bored-Ninja This has not been addressed in this release. Otherwise it would have been part of the release notes, sorry. Or have you found some other way of doing it?

A Bored Ninja

@muldjord I assumed that when you took the platform option out of screenscraper it would sort of resolve itself for me.

Since I have a screenscraper account and get six threads that's always what I use.

What I not benefit from the screenscraper platform removal?

muldjord

@A-Bored-Ninja Ah, yes, that is correct. I thought you were talking about the "platform alias" option I have under consideration which is not yet implemented (and might not be as it a pretty big change for little gain).

OldSchool

Hi @muldjord, is it possible to "clean up" the media? Let's say I had a bunch of games that were scraped and looking good, but some time later they were deleted for being dupes, or I just didn't want them anymore. Is there an easy way to delete the media associated with those now deleted games without clearing all of the media folders and cache, and scraping everything again? And can the cache be emptied if the games already have their media in the media folders after generating the gsmelist? Because with the cache and media folders combined I believe there's a duplicate of every media file, correct? Thanks for the help.

muldjord

@OldSchool Yes, but only from the cache. Please read the documentation for the --cache command here

Particularly the --cache vacuum option.

Concerning the ES media folder Skyscraper can't and won't keep track of what is in it. Remember that Skyscraper supports multiple frontends. Keeping track of all that is mindbogglingly difficult. But as long as you haven't made manual changes and kept your Skyscraper cache, you can just delete the media folder and regenerate the gamelist and media files. Then it will be "clean" again.

OldSchool

@muldjord Great! Appreciate the reply.

muldjord

Found a bug in my mobygames cover download code that could result in an endless loop. 3.2.1 just released with this fixed, please update.

muldjord

Skyscraper 3.2.2 released: https://github.com/muldjord/skyscraper

Added 'bat' scripts to sha1 special handling list (please purge platforms using 'bat' files and rescrape)
Now discards 'ZZZ(notgame)' results from ScreenScraper
Fixed double-quote issue when reading titles from scummvm.ini
Made location of scummvm.ini configurable in config.ini
Fixed bug in roman and integer numeral conversion functions
Rewrote the entire 'screenscraper' module to use JSON instead of XML

Fixed some bugs and rewrote the ScreenScraper module from the ground up. I checked out their JSON response format and decided it was a lot cleaner to work with than the XML format. Obviously, with a total rewrite, there can be issues. I've tested it both with unregistered use and with registered use and haven't found any issues beyond what would have happened when using the XML format.

Let me know if you run into problems.

Have fun!

muldjord

Skyscraper 3.2.3 released: https://github.com/muldjord/skyscraper

MAJOR: Added support for the 'Pegasus' frontend (set with '-f pegasus')
Now checks for, and removes, double bracket notes in final game list title
Fixed minor bug in the 3D gamebox effect renderer
Completely transparent images are no longer saved when compositing (Thank you to metallkopf for getting me to finally fix this)
Optimized the final game list assembling code to use game entry references instead of copies
Optimized all cache resource iterations to use const references instead of copies
Optimized the entire codebase by removing all Qt-centric foreach iterate-by-copy to use references instead
Fixed bug where 'screenscraper' would only look for ESRB age classification
Potential faulty JSON replies from Screenscraper are now saved to '~/.skyscraper/screenscraper_error.json' for easier debugging

Most prominent new feature is the support for the Pegasus frontend. Use it with -f pegasus. Keep in mind that the default artwork.xml is not well suited for the default Pegasus theme, so I suggest using this:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640"/>
  <output type="cover" width="640" height="480">
    <layer resource="cover" height="480" align="center" valign="middle">
      <gamebox side="wheel" rotate="90"/>
    </layer>
  </output>
  <output type="wheel" height="200"/>
</artwork>

Which will give you this:

Please also remember to disable the EmulationStation linking in the Pegasus UI settings. And equally important, when a platform has been scraped, remember to add the metadata.pegasus.txt to Pegasus afterwards. The games won't show up until you do.

I've also done quite a bit of optimization resulting in a 10% speed boost when generating game lists. Oh, and completely empy and transparent images are no longer saved in the media folders.

Have fun! And please report any problems you might run into. There's a lot of code rewrite going on at the moment, so some bugs could have snuck in.

muldjord

A quick note to anyone reading this: There is currently a bug in the ScreenScraper API that causes some searches to return a faulty JSON document (the raw data format). I've reported this to ScreenScraper.

EDIT: They should have fixed this now, so if you still encounter JSON errors, please create a bug report report at the Skyscraper github and attach the screenscraper_error.json file as the Skyscraper output instructs. Thanks! :)

Clyde

@muldjord Yeah, I stumbled upon this yesterday. Thanks for the announcement.

dudumaroja

@muldjord hey. thanks for the great work on skyscrapper.. just a qick question.. is there a way to try scrapping only missing games? (Skiping games already in DB and cache )?

muldjord

@dudumaroja Just disable refresh, and it will load all existing games from the cache (which is quite fast). Other than that, you can also use the --startat and --endat command line options to narrow the scope of the scraping run. :) But the most common error people do is to enable refresh without knowing what it means. Enabling refresh means that it will always fetch the data from the online source, even though it has already been cached.

Clyde

@muldjord Excuse me for jumping in on that, but I just started using Skyscraper. Will it complete missing resources from already cached games? If, say, a game has everything cached but a Video, will SS download the Video on the next run (without refresh) if it becomes available?

I'm asking because AFAIK, Sselph's scraper which I used up until now doesn't do that for games that are already in the gamelist. It doesn't use a cache like SS does, though.

muldjord

@Clyde No, if a game has previously provided a result for any resource, it won't fill in the holes unless you run a refresh on it. This is how the databases work. If you look up a game on any of the databases, it will give you most, if not all, of the info in one request. At that point it only makes sense to use all of it or not look up the game again at all.

The idea is to use the --cache edit or the import module feature to fill in the holes. You can use the --cache report:missing=<RESOURCE> to figure out what your games are missing. And you can use the --cache edit --fromfile <REPORT FILE> afterwards to edit only the games missing any of the resources in the reports.

dudumaroja

@muldjord thanks very much.. the refresh issue is more of the retropie menu problem i guess.. it was enabled by default for me ( never messed if it.. ) and i didnt know about the option until you pointed it to me.. ( i was even adenventuring myself to learn how to use your program without the retropie-setup menu, and just leaned about --startat and was about to use it.

another question.. im seen to have problem with modified roms. ( translation and fixes.. ) but i can see to find a option to scrapt theses rom by name.

could you provide-me a example on how to do it? im using the proper no-intro names.. but the hashes as messed up..

and again... thanks for the great work. i loved the ideia on have a cache for personal re-scraping.. i set a network share so it wont waste space on my SD or usb drive. and it works amazing.. i wish there was a tool so we can acess the cache data.. and make alterations ouselfs.. ( and maybe send it back later to screenscraper. )

Clyde

@muldjord Thanks for the explanation. So, a game can only have all of its resources from one source, which will be overwritten by one single other source on a refresh?

The source in brackets when scraping subsequently (e.g. Screenshot: YES (screenscraper)) made me hope that you can have resources from different sources.

If I do understand you correctly, why can't SS fill in the gaps locally even if the online databases give you all resources in one request? The resources are saved in separate files, after all.

I'm still hoping that I misanderstand you. ;)

muldjord

@Clyde said in Versatile C++ game scraper: Skyscraper:

@muldjord Thanks for the explanation. So, a game can only have all of its resources from one source, which will be overwritten by one single other source on a refresh?

No, it can have one of each resource per scraping module. Please check out the documentation it's all in there

The source in brackets when scraping subsequently (e.g. Screenshot: YES (screenscraper)) made me hope that you can have resources from different sources.

It does. :) It even allows for user resources on top of that with the --cache edit option where you can add new textual resources yourself that will be prioritized above all other resources. You can also prioritize the resources from all of the other databases with the priorities.xml files per platform. It's also documented, you'll find it. :)

If I do understand you correctly, why can't SS fill in the gaps locally even if the online databases give you all resources in one request? The resources are saved in separate files, after all.

It does, but there's no difference between filling in the holes and filling in everything. The point here being that if you are refreshing the data for a game, you will always be interested in filling in everything, since everything is already part of the data from the servers. Only filling in the holes would be a huge waste of database bandwidth.