Skyscraper now officially part of RetroPie, please test

mitu

@IanDaemon What version of RetroPie do you have installed ? Did you scroll down in the experimental section to see all packages ?

movisman

@muldjord

Hi there,

Thanks for the reply! No worries at all, I understand. I thought perhaps if Skyscraper kept track of the media that it scraped itself, it would perhaps have an option to purge them at a later stage and ignore any user generated files. I get why this wouldn't really work though I think.

I guess the only other option is to have an overwrite switch, where it overwrites the destination media folder upon scrape, removing whatever was there before and starting afresh. But again, I understand why you perhaps wouldn't implement this, due to risk involved of users selecting this option accidentally or not understanding what it does.

It is fine for me, because if I change ROMs around at any stage, I can just delete the media folder and rescrape if I wish to purge the old files which are no longer referenced due to the ROM being deleted. It was just something I noticed and thought i'd ask if it was possible with a switch or could be a consideration for the future.

Cheers!

IanDaemon

@mitu I have RetroPie 4.4. I'll look again tonight.

mitu

@IanDaemon 4.4 is the major release - the current version is 4.4.4, if you have updated recently. You can upgrade just the RetroPie-Setup script (sans packages) to get Skyscraper as new package in the experimental section.

movisman

@muldjord

Hi there,

I have a quick question about the passes the scraper makes on each ROM and the ability to scrape / search manually. I recently converted my 21 Dreamcast roms to .chd format, which I will do with some other systems eventually. This just makes the folder view way more tidy and manageable, as I now just have a single file for each game. Now, I know scraping a user-made CHD will likely not scrape, as it'll have it's own attributes and filename. I was right, of course it didn't match with any games.

So I figured I would scrape the original .gdi files (pre-conversion), and then just adjust the gamelist to suit. It's just 21 games, so no problem.

However, I ran into a problem scraping some DC .gdi files - I discovered that quite a few games share the same SHA1 value, mainly because a lot of the .gdi files have the same content (as they don't have any mention of the game inside, they just point to 'track 1', 'track 2', etc).

Anyway, this led to some very interesting scraping results, with a load of games simply scraping as a game called 'D-2'! Also the gamelist.xml was a bit all over the place, I guess because of the same SHA1 being matched multiple times for multiple files.

However, I understand the last pass the scraper makes, is to check for a full filename match (must be exact), if it cannot find a match for SHA1, etc. So, I thought as a workaround for these files, I would find a supported file on screenscraper.fr, rename my .gdi ROM to that, and scrape again.

This didn't work because, of course, it would match this common SHA1 value first and always take that as a priority over filename. I should have known that.

Then I realised, I could just rename each CHD file to a filename match on the screenscraper.fr site, including changing the extension. Because the SHA1 value would be unique on my user made CHD, it should in theory go for the filename. This worked! Scraped the ROMs and got all screenshots and videos. I then however, had to go back in and rename the roms again back to .chd extensions, and adjust the gamelist.xml to suit. Bit of a pain, but it did work.

So this, as a workaround was fine, but I have two questions (one is more of an 'enhancement'):

Can the user adjust the priorities of what each 'pass' tries to match? Eg. Can I change it to match the filename first over say, SHA1? This would then solve the issue of the scraper matching with a common SHA1 for files like .gdi. I understand this is probably a rare occurrence and maybe only isolated to DC, and maybe it is already possible to adjust the order! Just to clarify i'm not suggesting any change to the default (which makes perfect sense), just the ability to change the priority and if that is currently possible?
Second one, which would help with all the manual work above, is would you consider a manual lookup for a future version, if there is no match found OR if the user just wants to confirm each scrape as it is performed? So, as it scrapes a game, it asks you to confirm, if you say 'no', you can select from a list returned from a search based on the filename, or maybe you can type in all or part of the game you think it should be. It would then come back with a list where you can choose the correct ROM. Something like that anyway.

Not sure how hard the above is to implement (I suspect time consuming) but it would all but solve the issues I had above. I could then simply feed unique CHD's to skyscraper, and although it wouldn't match on the automatic passes because there is no match on screenscraper.fr, it would prompt me each time to type in some words to help find a match, or perhaps it could do a wildcard search based on part of the filename minus extension and give you some potential results to choose from.

Sorry the above is a bit long winded, just making a possible couple of suggestions based on my user experience.

Cheers!

Hyperlord

Hi guys,

great, I'm loving skyscraper!

I had it installed manually and ran it for a couple of month, now I installed the built-in version and just "copied" my db-folder from /home/pi/.skyscraper to /opt/.../skyscraper

Was this "right"? Am I missing something?
As I recently tripled the free space on the SD card I want to "rescrape" with videos this time. Am I able to do this or should I start from scratch?

mitu

@Hyperlord If you already had a Skyscraper installation, the configuration script in the module would have done this automatically - migrate your settings and move over your LocalDB. In fact, your $HOME/.skyscraper folder should be now symlinked to /opt/retropie/configs/all/skyscraper, so both versions should be running ok.
Do you have any problems after installing Skyscraper from the RetroPie-Setup script ?

muldjord

@movisman

Answer to first question:
I can add the .gdi extension to the list of file types that will do a sha1 checksum on the filename instead of the content. I already do this for script files that change often.

Answer to second question:
There is already a manual lookup although it isn't documented well, so I can easily understand why you haven't seen it. It's the "--query" option. Read more about it here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

movisman

@muldjord

Hi there,

Thanks for the heads up about the --query option, i'll check that out.

As for the .gdi thing, that sounds good, because quite a lot of the games (it seems) share the same checksum for the .gdi file.

Thanks!

movisman

@muldjord

Hi there,

Thanks for the heads up with the --query option. I had a play around with it and I was able to scrape one of the Gamecube games I had trouble with, by querying a recognised SHA1 on screenscraper.fr against my filename.

However, none of the gamelist or folders populate with the scraped media even though they are valid. I suspect it's probably this message which comes up after the command is run:

Pretend set! Not changing any files, but still caching resources in local database.

If I check the DB folder I can see the files there though, so the scrape happened, it just didn't write the media or populate the gamelist.xml.

I was testing outside of RetroPie on my laptop, so wasn't working within the system at all. Not sure if this affects anything? The command I ran for a test on that specific file was:

Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~/Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets --query sha1=F5ECEF62B8A6ED8F76EEC3319BE7168352C5D5AE ~/Documents/scraper/Capcom\ vs.\ SNK\ 2\ EO\ -\ Millionaire\ Fighting\ 2001\ (USA).iso

I tried various other commands, simplified etc, but the same issue occurs every time, and that message about 'pretend mode' always comes up.

Any idea?

Thanks!

mitu

@movisman As the message says, it's not modifying the gamelist, but just caching the resources in LocalDB. After you run this, scrape again using LocalDB to add the data to the gamelist.

movisman

@mitu

Thanks for your help with this.

I should have mentioned that in my previous post (it was late and I was getting tired!), I actually tried that already as I figured that would sort it (using the localdb as the scraping source after it is cached). I also looked at the script which gets created when you run Skyscraper with no switches and could see it runs twice, once to cache the resources, and then again to write the changes.

However if I swap out '-s screenscraper' to '-s localdb' on the above command, it is still giving me the message about 'pretend mode' and my gamelist.xml is still empty after running.

Not too sure what is occurring here.

Thanks for the help!

movisman

@mitu @muldjord

Okay, I think I have it now, on the second scrape I had to remove the query part of the command, then it wrote the changes. It didn't matter whether I used screenscraper or localdb as the source at this point.

As follows (one file in the directory as a test), first pass on the troublesome file:
Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~//Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets --query sha1=F5ECEF62B8A6ED8F76EEC3319BE7168352C5D5AE ~/Documents/scraper/Capcom\ vs.\ SNK\ 2\ EO\ -\ Millionaire\ Fighting\ 2001\ (USA).iso

And then on the second pass:
Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~/Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets

Basically same again without the query, it then wrote the gamelist and copied the media. If I change out screenscraper for localdb on the second pass it also works the same.

What i'm looking to get to, is once all my roms are scraped, i'd like to come up with a command which I can run when I add a single rom (or a couple of roms) to the system, but I don't want the scraper to overwrite or change any of the other entries (as i'm renaming them post-scrape back to .chd - in some cases). Need to experiment with this.

Cheers

brimby

Running my first test scrape using my Master System roms since I hadn't scraped for those yet.

First thing I noticed was that there are a lot more sources available than on the Selph Scraper. Does anyone know which source might have the most reliable "ratings"? The ratings I've gotten from screenscraper, vgdb and gamesdb have all been very poor. I'd really like to be able to order my games list by rating and get some sort of semblance of quality at the top.

Before you yell at me, yes of course I know ratings are "subjective". But there is a big difference between "What's the best game?" and "What's the best color?" You can gather a kind of truth out of the first question if you ask enough people. Not so much the second.

brimby

This is completely unrelated to my post above, and is about artwork.xml.

Is there a place where people are posting their artwork.xml code with an image of the output? I'd like to see what other individuals more creative than me have come up with. I know there are examples in the /.skyscraper folder, but I can't view the output without a bit of effort.

Secondly, I assume the artwork.xml is used to create a custom image at the moment the scraping happens, rather than it dynamically creating it every time the image is viewed in emulation station. If I'm correct in that assumption, is there a way that I could apply new artwork.xml settings to already-existing scraped screenshots? Like for instance if I take artwork.example1.xml and rename/replace it with artwork.xml: would it be applied automatically or do I have to do something to run all the images through the new settings?

mitu

@brimby The artwork.xml is used when producing the boxart and saving the image referenced in the gamelist.xml, to be shown in Emulationstation. Hence, you can change the artwork.xml and then re-scrape using only the LocalDB source, and the new image will be produced based on the new .xml file. This is the easiest way to test the new artwork or to change the output based on your new settings.

brimby

@mitu cool that's exactly what I needed to know. I only sort of get the purpose of the localdb source. What other reasons might someone have for using localdb source besides for rerunning their artwork.xml?

mitu

@brimby Read carefully the documentation over at github.com/muldjord/skyscraper, it's explained there what's the purpose of the Local DB and how it's used.

Hyperlord

@mitu Thanks for the fast reply!
I'll try this at the weekend since the scraping will take approx 2 days :-)

After I installed the skyscraper from the retropi-setup and moved the db I ran a quick scraping to see if it's still working. I guess I started this from the menu but currently I'm not sure.

But my questions has been answered so far by now ...

brimby

Is there a subreddit yet? I tried to find one but just found a lot of stuff about skyscrapers, heh.

If there is no subreddit, creating one might be something to consider so that people can share their opinions about priorities.xml, artwork.xml, or post their local dbs.

Or maybe there already is some other community hub in existence, like a discord or slack or idk? Something besides this single retropie.org.uk thread?