Versatile C++ game scraper: Skyscraper

muldjord

Hi guys, I'm just about ready to release Skyscraper 3.5.6 with the promised Pegasus game list parser. Currently awaiting some testing by users.

Another user requested --flags onlymissing to be added to the RetroPie Skyscraper script. @mitu I've added it in my fork. Let me know if you want me to PR it. I've tested it and it works without issue. Basically just added the option to the gather options menu.

mitu

@muldjord said in Versatile C++ game scraper: Skyscraper:

@mitu I've added it in my fork. Let me know if you want me to PR it

Nice, that's a good addition. You can send a PR if it's not too much trouble - I'll give it a test and maybe add the info to the wiki.

muldjord

@mitu No problem at all. I've created a PR for it.

muldjord

Skyscraper 3.5.6 released: https://github.com/muldjord/skyscraper

Added 'neogeocd' platform (Thank you to 'igno2k' and 'meveric' for suggesting it)
'screenscraper' module now supports region for screenshots (Thank you to 'cameronhimself' for pointing this out)
Low disk space check for cache is now calculated from cache folder location
Skyscraper initial output now shows frontend and extras when generating game lists
Changed 'gamelistFolder' config.ini option to 'gameListFolder' but kept old variable name for backwards compatibility
Added 'gameListBackup' config.ini option to allow for game list backups being created on each run (Thank you to 'igno2k' for suggesting it)
Attract-Mode now recognizes '$HOME' in emulator file
Refactored Attract-Mode media folder getters
Fixed rare faulty matches from 'esgamelist' module in cases where last part of 'path' was a match with filename (Thank you to 'akamming' for reporting this)
Exporting for Pegasus frontend now supports preservation of old game list key-value pairs (Thank you to user 'HugLifeTiZ' for suggesting this)
'--cache report' now also adheres so excludeFiles and includeFiles (Thank you to user 'spilinek' for pointing this out)

Skyscraper 3.5.6 is finally here! It contains the promised Pegasus metadata preservation parser. Basically it means that if you add custom fields to your metadata.pegasus.txt files in either the header or game entries, they will be preserved when regenerating the gamelist using Skyscraper.

A lot of other stuff has made it into the release. Some minor fixes to a bunch of stuff, and a new config.ini option that enables game list backup. Set gameListBackup="true" in the [main] or a [FRONTEND] section and instead of overwriting the game lists, it will add a timestamp to the current one before exporting the new one.

I've tested it of course, but please let me know if you run into issues with the new additions / changes.

Happy scraping!

muldjord

Just added a frontend overview, better explaining how frontend game list generation works in Skyscraper. Check it out here!.

Clyde

A great update with many useful additions! My personal favorites are the neogeocd support and the game list backup option.

That said, what is the difference of the latter's location in [main] or [FRONTEND]?

muldjord

@Clyde It's simply a matter of when it's used (this is true for many options by the way, it's all described here). You might only want game list backup to be enabled for certain frontends. In that case you can add the option under the desired frontend section like so:

[pegasus]
gameListBackup="true"

But if you prefer Skyscraper to create a backup no matter what frontend you are currently generating game lists for, you simply add it to the main section instead like so:

[main]
gameListBackup="true"

Clyde

Thanks for the explanation. 👍

stevas

@muldjord
Any reason why some games are being skipped for me on import (mainly unlicenced, but a couple of legit sonic the hedgehog too)? It says "no returned matches" in the skipped.txt. Am I enabling this in the config.ini somehow?
The thing that's confusing me is: I'm using screenscraper.fr, and I'd assume skyscraper is not seeing them there... But that's where I got the images from in the first place (via Skraper)? Is your app and the Skraper one using a different dB? What exactly is the import command matching the titles against?

These are up to date and clrmamepro'd no intro roms.

I've not updated skyscraper by the way, still on the previous version.

muldjord

@stevas Have you read the import documentation thoroughly? It's all in there: https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

stevas

I did, and I'm not seeing the section which covers my issue tbh. Thanks anyway.

muldjord

@stevas If you can provide a detailed example of an entry that should scrape I can look into it. What is the exact rom filename? What are the exact filenames of the artwork you have and where did you put them. What command do you use to import it would also be useful.

What exactly is the import command matching the titles against?

This is documented right here. If you feel it can be misinterpreted, please let me know.

stevas

@muldjord

Yeah... You know what, I think it's my bad here. Now, to be fair, the no intro sets have seemingly arbitrarily decided to change the format of their filenames (Sonic is a particularly good example of how this can be easily missed, as the "The" has become "the" - which is maybe more grammatically correct - but why did we need all the v1.1s to change to Rev A etc?)... and, of course, what's exacerbated my issue here is the fact Unix can have BOTH those Sonic files existing at the same time (which meant a process I had of making sure files matched by just copying them over existing ones on my windows PC was... fundamentally flawed).

Anyway, I've decided to rescrape my megadrive stuff again (sigh), and hopefully it'll all figure itself out.

Many thanks for the swift replies though, and I love the app; I just wish there was an easy way to get the composite screenshot images I like (recalbox mix V2, with the cartridge included) so I didn't have to use Skraper on the PC at all.

muldjord

@stevas said in Versatile C++ game scraper: Skyscraper:

I just wish there was an easy way to get the composite screenshot images I like (recalbox mix V2, with the cartridge included)

You can composite that with Skyscraper quite easily. If you import the cartridges into the marquee resource in Skyscraper and then set up the artwork.xml to composite it as you want it, you can basically make it look anyway you like. The Skyscraper compositor is very powerful and can do a lot of cool things. I assume you have seen the documentation for it, but here's the link just in case: https://github.com/muldjord/skyscraper/blob/master/docs/ARTWORK.md

stevas

Yeah, I tried it... is there a way to get it to grab 3d boxart? I mean with the correct spine, not a wheel instead?

muldjord

@stevas Yeah, but it needs some tinkering. I once did it for mastersystem. I simply imported all of the mastersystem spines into the marquee resource in Skyscraper, and then used the boxart effect with the marquee on the side/spine. Looked really good. But there's no automatic way of doing it. So it requires a bit of work to import all of those.

stevas

Actually, I just realised I can use the 3d box Skraper can get at... But now I can't scrape even just screenshots from screenscraper.fr through skyscraper (FFS? FFS).

o0alucard0o

I scraped NES roms for Attract-Mode with the latest Skyscraper version.
The process correctly fills standard cache folders (covers, marquees, screenshots, wheels) but exports only snaps to Attract-Mode.

NES.cfg

romext               .zip
system               Nintendo Entertainment System
artwork    flyer           /mnt/c/Games/Attract-Mode/scraper/NES/flyer
artwork    marquee         /mnt/c/Games/Attract-Mode/scraper/NES/marquee
artwork    snap            /mnt/c/Games/Attract-Mode/scraper/NES/snap
artwork    wheel           /mnt/c/Games/Attract-Mode/scraper/NES/wheel
artwork    boxart          /mnt/c/Games/Attract-Mode/scraper/NES/boxart

Skyscraper command(s):

Skyscraper -p nes -i /mnt/c/Games/roms/nes/ -g /mnt/c/Games/Attract-Mode/romlists/ -f attractmode -e /mnt/c/Games/Skyscraper/NES.cfg -s screenscraper -u username:password --flags unattend,skipped
Skyscraper -p nes -i /mnt/c/Games/roms/nes/ -g /mnt/c/Games/Attract-Mode/romlists/ -f attractmode -e /mnt/c/Games/Skyscraper/NES.cfg --flags unattend,skipped,forcefilename

muldjord

@o0alucard0o Yeah, the default Skyscraper /home/pi/.skyscraper/artwork.xml is set up to look good with EmulationStation. For Attract-Mode you can go ahead and adjust it to fit your needs. For instance:

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" width="640"/>
  <output type="cover" height="640"/>
  <output type="wheel" width="640"/>
  <output type="marquee" width="640"/>
</artwork>

When you've saved that, just regenerate the game list for Attract-Mode.

This will export all of the resources Skyscraper supports and scale them according to aspect ratio to 640 pixels height or width. This is the most basic example I can give you. But the compositor supports a lot more.

For the full documentation, go here.

o0alucard0o

@muldjord Thanks for the explanation. 👍