Versatile C++ game scraper: Skyscraper

martin_f

Just started using Skyscraper, excellent little tool muldjord ! Appreciate your time in putting this together. I had been using UXML for a little bit, but much prefer Skyscraper.

One question I have. I was reading dorkvader's comments regarding the "spine". Am I correct in assuming that at current time Skyscraper doesn't scrape "spine" art, such as that on https://www.screenscraper.fr/ ? If not, is there any plan to add this feature?

dorkvader, were you manually downloading the spine art, or are there packs floating about?

Anyway, keep up the good work, I am sure its appreciated by many.

muldjord

@martin_f I don't really plan to expand the types of resources that can be fetched currently. But you can sortof cheat and use the 'marquee' as the 'spine' art if you don't use it for anything already. I did this by downloading some of the spine art packs from emumovies, then importing them as 'marquee' and then added them on the side of the boxart effect. That turned out perfect.

martin_f

Nice one muldjord, I will give that a go.

And thanks again for such an awesome tool !

muldjord

@tacodog I have just implemented the feature to have subfolders in the import folder. So if you have 'import/snes/screenshots' it will look for specifically snes screenshots in there and so on.

It's not officially released yet, just wanted to let you know that the feature is done and will be in the next release. You can already try it out now if you'd like to, it's on the master branch on git.

EDIT: It is now released in 2.7.8

muldjord

Skyscraper version 2.7.8 released: https://github.com/muldjord/skyscraper

Added CRC32 checksum attribute to 'screenscraper' pass
Changed the 'import' folder to also look into 'import/[platform]' if it exists so users can create imports per platform. Old method also works (Thanks to tacodog for suggesting this)
Added 'mpixels' attribute to compositor layers that allows users to scale images to a desired area size instead of using width and height (Thank you to Vynce for suggesting this)

Next up is 2.8.0 which will implement some interesting preserve features for the current gamelists. For instance, 'hidden' and 'favorite' aren't preserved for the EmulationStation gamelist.xml currently. They will be from 2.8.0 and forward.

tacodog

Thanks for adding the import subdirectories. I just tested it and it's working well. Just a note for anyone trying this, 'definitions.dat' needs to go in the system subdirectory as well if you want to use textual data. This took me a little while to figure out.

muldjord

@tacodog I was actually wondering if people would have problems figuring that out as it isn't all that intuitive to also have to put it in the platform subfolder. So thank you for reporting this, I will consider making it work by only having it in the import base folder.

EDIT: This is fixed in the upcoming 2.8.0 release (not released yet). It will first look for 'definitions.dat' in the platform subfolder and then in the base import folder.

BiZzAr721

@muldjord said in Versatile C++ game scraper: Skyscraper:

@sammyboy It's a bit tricky to do. You have to compile it on a Linux machine that ISN'T the RetroPie. I would suggest Ubuntu since the installation procedure on there is exactly the same as on the pi. Then you need to create a rom folder with just 1 single rom inside. Then you need to scrape that rom with sources until you have all of the artwork types for it (start by trying screenscraper, it probably has all of them). Then you open a terminal and type in "watch -n2 Skyscraper -p [platform] -i /[full]/[rom]/[folder] -s localdb --unattend". Be sure to change [platform] and the rom folder. That command will run Skyscraper every 2 seconds, using the localdb without asking any questions, which will result in the exported media to be updated every 2 seconds from the '[homedir]/.skyscraper/artwork.xml' file. Now open the relevant output image in an image viewer that automatically reloads if the image changes. The default one in Ubuntu works.

Now all you need to do is open up an editor and edit '[homedir]/.skyscraper/artwork.xml' and the image will update with your changes every 2 seconds.

So yeah, not so simple unfortunately. It can be done on the pie itself too if you ssh connect to it from another machine through 2 different terminals. On one you run the above command. On the other you edit the artwork.xml. And then you open a file browser and browse to the export folder where the images reside. I DO NOT recommend doing this on the pi, as the command will save data to the SD card every 2 seconds while you're working on it. SD cards do not like that! You've been warned. :)

@muldjord Excellent work! I never realized just how customizable you made this (especially importing local artwork!) Since some time has passed, is there any easier ways to view the artwork while editing artwork.xml?

muldjord

@BiZzAr721 I'm afraid not. I would still do it the same way as described in your quote. If you create something awesome, feel free to share your artwork.xml.

BiZzAr721

If I do, I sure will.

movisman

Hi,

Can someone elaborate on the --skipped option for me?

I have read the help, however i'm building a command to use for future scrapes, and i'm not sure of the real difference / usage between --skipped and --unattend, except I know when using --skipped you get prompts, but it asks you anyway if you want to skip existing entries. So i'm not sure why you would use it?

Using --unattend removes the prompts and skips any existing entries when writing the gamelist, so again no need for --skipped, right?

I noticed when using the GUI via RetroPie -Setup, it uses --unattendskip and --skipped, but i'm still not sure why both are required.

In my case, I would use --unattend rather than --unattendskip because i'm renaming some of my files back to .chd after scraping them, and using --unattendskip wipes them off the gamelist if they've been renamed post-scrape.

Can anyone give me a heads up on the --skipped option and it's use case on it's own, as well as alongside --unattend and --unattendskip?

Just want to work out if it is an option I need to use or not. It seems like it's not necessary as the other options cover it, or the scraper asks you anyway when you don't run unattended.

Cheers

muldjord

@movisman For any confusion with these, please check the output of the "--help" option. This shows you all of the options and what they mean.

EDIT: Btw, if you are building a command line to use in general, remember that you can set most options permanently in the ~/.skyscraper/config.ini file. That way you won't have to add then to the command line all the time.

movisman

@muldjord

Hi,

Thanks for the help. I have read all the options on the command line, I always try to check stuff out myself first - but am still slightly confused by the usage of the --skipped option and when it's required? I must be missing something obvious, but I am not sure why it would be used when you run either interactively or with the --unattend option.

Are you able to elaborate a bit for me? Apologies if I am missing something here.

Understood about the config.ini file, I think once I have a command working the way i'd like, i'll transition those into a .ini file for ease of use.

Thanks

mitu

@movisman I think --unattend also adds --skipped, I just left it in the scriptmodule to be set anyways. The --unattendskip was removed in the last version of the scriptmodules.

movisman

@mitu

Cheers. I need to use --unattendskip so it doesn't overwrite my renamed files, so I can leave --skipped off then I guess?

Does --skipped have any use on it's own? Considering interactive asks whether you wish to skip anyway - and --unattend also adds it? Does it have any use when combined with --unattendskip? From what I can tell, it appears not?

Thanks!

movisman

Sorry, corrected my command above as it was wrong!
Cheers

muldjord

@movisman You seem to misunderstanding what --skipped does. The skipping prompt is not related to --skipped.

Answering yes to skipping at the beginning will skip all entries you already have in your gamelist, and therefore not try to rescrape them (and also not recreate the media for them). I never personally answer Y here as it makes more sense to let them be rescraped and regenerated from the cache. Answering Y will save you some time, but that's all. Basically it only makes sense to answer Y if you have manually changed some of the publishers, developers or data for some games in the gamelist.

Adding --skipped will make sure games that returned "not found" are still added as empty entries to the gamelist when it's generated at the end of a scraping run. That's all this option does.

I never personally use the --skipped option, but some people like to have those entries in the gamelist.

movisman

@muldjord said in Versatile C++ game scraper: Skyscraper:

Adding --skipped will make sure games that returned "not found" are still added as empty entries to the gamelist when it's generated at the end of a scraping run. That's all this option does.

This is exactly what I was looking for. That's what it does! Thank you! Maybe the help could be slightly tweaked to mention the behaviour in slightly more detail, it might just be me though!

I actually don't want that for my use case, so I can safely not use that option. I also have to answer Y to skip existing entries for certain systems (or use --unattendskip) rather than have them be rescraped. The reason for this, is after scraping successfully, I have to rename the files back to a .chd extension for some systems (and edit gamelist.xml to suit), and answering N means they get rescraped and not found due to unique checksums etc, thus overwriting their entry in the gamelist with nothing.

Unless you have any ideas on a better way for me to manage this, I think I am ok to just use --unattendskip, because if I add the odd game later, I can use this command along with a query (if needed) to append new entries to the gamelist, and it won't overwrite my CHD entries mentioned previously.

Some systems which use .zip, are of course a lot easier. Does that make sense?

Thanks a lot for your help as always, this scraper is really very flexible and powerful.

movisman

Me again :)

I see there is Amiga support, but should it work with CD32 games (this is a separate section on screenscraper)? Running a basic command doesn't seem to yield any results, however, when I run a query against a valid SHA1 checksum, it finds and scrapes the information to the DB.

However when running the command again but with -s localdb, it doesn't write anything at all to the gamelist.xml or copy the media across. I am not skipping any files or anything like that.

Using the same command with other system (Atari 2600) is fine.

Before I spend more time troubleshooting, are you able to confirm if the system is even supported?

Thanks!

muldjord

@movisman Yes, the system is supported, but the Amiga platform is a bit difficult to work with as it is basically more systems inside one system. So Skyscraper looks for the "aga", "cd32" and "cdtv" in the filenames to switch the platform when scraping them. Please try scraping them using the openretro source. It will only work with games that have 2 or less words in them due to a limitation on their website, but openretro is the best source for amiga games.
If it scrapes the data when using a custom sha1 and the data is entered into the local cache, it will use the data when scraping with '-s localdb' afterwards. If it doesn't something is wrong with your command or your config.ini.