Skyscraper now officially part of RetroPie, please test

Hyperlord

Hi guys,

great, I'm loving skyscraper!

I had it installed manually and ran it for a couple of month, now I installed the built-in version and just "copied" my db-folder from /home/pi/.skyscraper to /opt/.../skyscraper

Was this "right"? Am I missing something?
As I recently tripled the free space on the SD card I want to "rescrape" with videos this time. Am I able to do this or should I start from scratch?

mitu

@Hyperlord If you already had a Skyscraper installation, the configuration script in the module would have done this automatically - migrate your settings and move over your LocalDB. In fact, your $HOME/.skyscraper folder should be now symlinked to /opt/retropie/configs/all/skyscraper, so both versions should be running ok.
Do you have any problems after installing Skyscraper from the RetroPie-Setup script ?

muldjord

@movisman

Answer to first question:
I can add the .gdi extension to the list of file types that will do a sha1 checksum on the filename instead of the content. I already do this for script files that change often.

Answer to second question:
There is already a manual lookup although it isn't documented well, so I can easily understand why you haven't seen it. It's the "--query" option. Read more about it here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

movisman

@muldjord

Hi there,

Thanks for the heads up about the --query option, i'll check that out.

As for the .gdi thing, that sounds good, because quite a lot of the games (it seems) share the same checksum for the .gdi file.

Thanks!

movisman

@muldjord

Hi there,

Thanks for the heads up with the --query option. I had a play around with it and I was able to scrape one of the Gamecube games I had trouble with, by querying a recognised SHA1 on screenscraper.fr against my filename.

However, none of the gamelist or folders populate with the scraped media even though they are valid. I suspect it's probably this message which comes up after the command is run:

Pretend set! Not changing any files, but still caching resources in local database.

If I check the DB folder I can see the files there though, so the scrape happened, it just didn't write the media or populate the gamelist.xml.

I was testing outside of RetroPie on my laptop, so wasn't working within the system at all. Not sure if this affects anything? The command I ran for a test on that specific file was:

Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~/Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets --query sha1=F5ECEF62B8A6ED8F76EEC3319BE7168352C5D5AE ~/Documents/scraper/Capcom\ vs.\ SNK\ 2\ EO\ -\ Millionaire\ Fighting\ 2001\ (USA).iso

I tried various other commands, simplified etc, but the same issue occurs every time, and that message about 'pretend mode' always comes up.

Any idea?

Thanks!

mitu

@movisman As the message says, it's not modifying the gamelist, but just caching the resources in LocalDB. After you run this, scrape again using LocalDB to add the data to the gamelist.

movisman

@mitu

Thanks for your help with this.

I should have mentioned that in my previous post (it was late and I was getting tired!), I actually tried that already as I figured that would sort it (using the localdb as the scraping source after it is cached). I also looked at the script which gets created when you run Skyscraper with no switches and could see it runs twice, once to cache the resources, and then again to write the changes.

However if I swap out '-s screenscraper' to '-s localdb' on the above command, it is still giving me the message about 'pretend mode' and my gamelist.xml is still empty after running.

Not too sure what is occurring here.

Thanks for the help!

movisman

@mitu @muldjord

Okay, I think I have it now, on the second scrape I had to remove the query part of the command, then it wrote the changes. It didn't matter whether I used screenscraper or localdb as the source at this point.

As follows (one file in the directory as a test), first pass on the troublesome file:
Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~//Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets --query sha1=F5ECEF62B8A6ED8F76EEC3319BE7168352C5D5AE ~/Documents/scraper/Capcom\ vs.\ SNK\ 2\ EO\ -\ Millionaire\ Fighting\ 2001\ (USA).iso

And then on the second pass:
Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~/Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets

Basically same again without the query, it then wrote the gamelist and copied the media. If I change out screenscraper for localdb on the second pass it also works the same.

What i'm looking to get to, is once all my roms are scraped, i'd like to come up with a command which I can run when I add a single rom (or a couple of roms) to the system, but I don't want the scraper to overwrite or change any of the other entries (as i'm renaming them post-scrape back to .chd - in some cases). Need to experiment with this.

Cheers

brimby

Running my first test scrape using my Master System roms since I hadn't scraped for those yet.

First thing I noticed was that there are a lot more sources available than on the Selph Scraper. Does anyone know which source might have the most reliable "ratings"? The ratings I've gotten from screenscraper, vgdb and gamesdb have all been very poor. I'd really like to be able to order my games list by rating and get some sort of semblance of quality at the top.

Before you yell at me, yes of course I know ratings are "subjective". But there is a big difference between "What's the best game?" and "What's the best color?" You can gather a kind of truth out of the first question if you ask enough people. Not so much the second.

brimby

This is completely unrelated to my post above, and is about artwork.xml.

Is there a place where people are posting their artwork.xml code with an image of the output? I'd like to see what other individuals more creative than me have come up with. I know there are examples in the /.skyscraper folder, but I can't view the output without a bit of effort.

Secondly, I assume the artwork.xml is used to create a custom image at the moment the scraping happens, rather than it dynamically creating it every time the image is viewed in emulation station. If I'm correct in that assumption, is there a way that I could apply new artwork.xml settings to already-existing scraped screenshots? Like for instance if I take artwork.example1.xml and rename/replace it with artwork.xml: would it be applied automatically or do I have to do something to run all the images through the new settings?

mitu

@brimby The artwork.xml is used when producing the boxart and saving the image referenced in the gamelist.xml, to be shown in Emulationstation. Hence, you can change the artwork.xml and then re-scrape using only the LocalDB source, and the new image will be produced based on the new .xml file. This is the easiest way to test the new artwork or to change the output based on your new settings.

brimby

@mitu cool that's exactly what I needed to know. I only sort of get the purpose of the localdb source. What other reasons might someone have for using localdb source besides for rerunning their artwork.xml?

mitu

@brimby Read carefully the documentation over at github.com/muldjord/skyscraper, it's explained there what's the purpose of the Local DB and how it's used.

Hyperlord

@mitu Thanks for the fast reply!
I'll try this at the weekend since the scraping will take approx 2 days :-)

After I installed the skyscraper from the retropi-setup and moved the db I ran a quick scraping to see if it's still working. I guess I started this from the menu but currently I'm not sure.

But my questions has been answered so far by now ...

brimby

Is there a subreddit yet? I tried to find one but just found a lot of stuff about skyscrapers, heh.

If there is no subreddit, creating one might be something to consider so that people can share their opinions about priorities.xml, artwork.xml, or post their local dbs.

Or maybe there already is some other community hub in existence, like a discord or slack or idk? Something besides this single retropie.org.uk thread?

muldjord

@brimby yes, try r/retropie

yyandros

I've got a question about scraping images for Mega Drive/Genesis...is there a way to tell the scraper to grab the Genesis box art as opposed to the Mega Drive box art? I'm an absolute newb when it comes to this stuff, so I'm assuming I'm overlooking a way to do it since this scraper is really robust. Also, great work! This is by far my favorite scraper.

mitu

@yyandros You can try asking for the US region when scraping from the command lina, with the --region us parameter.

muldjord

@yyandros I just updated it yesterday to support auto-region detection, so if the filenames have (USA) in them, it will automatically try to grab the Genesis versions. Otherwise, if you wish to force a certain region for it (megadrive and genesis are usually seen as the same platform) you can use the "--region us" or set it in config.ini with "region=us" under the "[main]" section. Remember to uncomment the lines by removing the "#" in front of them, otherwise they won't apply.

muldjord

@mitu It has been asked that the "mobygames" scraping module is removed from the RetroPie script. This is a fair point, as the module is limited to 25 files at a time and only meant to be used for single file command line provided scrapings. Thus it is both pretty useless to have in the script and also not obvious to the user why it won't work for most scrapings.

I would therefore ask for it to be removed, unless you have an idea for how we can handle this and force the use of it to be restricted to just 1 file at a time.

The reason for the limit on the "mobygames" module is that the key I have for it is limiting the ENTIRE Skyscraper user base. So if one user is using it when you want to use it, it will run into issues. It is therefore imperative that it be used only for single game scrapings. It's basically meant to be used only for those few singular roms you have left that the other sources won't provide data for. :)