Skyscraper now officially part of RetroPie, please test

mitu

@BuZz said in Skyscraper now officially part of RetroPie, please test:

To make the module work from a binary install, everything needed will need to be in /opt/retropie/supplementary/skyscraper.

Submitted https://github.com/RetroPie/RetroPie-Setup/pull/2520 to fix this. Now all config files are pulled from $md_inst and the install has all the necessary files included.

@muldjord I've incorporated almost all your suggestions - what's left is the config.ini settings for the input, roms and dbs (I removed the artwork.xml). We can probably remove them in a subsequent modifications.
One thing I added as default is --unpack, I think it should be a default since most ROMs are zipped and it should make problems like I renamed by ROM to XYZ.zip be less prevalent.
I also fixed the Couldn't calculate sha1 hash sum of rom file '', please check permissions and try again, now exiting..." error, reported in

https://retropie.org.uk/forum/topic/19739/ and I think also in
https://old.reddit.com/r/RetroPie/comments/9qqi23/help_skyscraper_not_working/

I've encountered this error when testing, but I assumed it was because some of my systems had bogus zip files and Skyscraper couldn't unpack them (seemed to work fine with other scraping sources that don't use checksums). But it turned out it's a shell params expansion thing and I've found a simple workaround for it.

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

One thing I added as default is --unpack, I think it should be a default since most ROMs are zipped and it should make problems like I renamed by ROM to XYZ.zip be less prevalent.

Nonono, please remove that again. --unpack is implemented in a way that could give users problems with memory (and a performance hit) since it doesn't feed the checksum calculation data in segments. It works just fine without it as most zips are already in the screenscraper db, and all other sources are name based so they don't use it anyway.

I general anything that isn't default shouldn't be added by default. I usually add options to Skyscraper in a way that things that should be default, I just enable them, and then I add the opposite bool option to allow users to disable it. So, for instance, for --unpack, if it should be enabled by default, I would add '--nounpack'.

mitu

@muldjord OK, no problem.

movisman

@muldjord @mitu

Hi,

Just tried using this scraper for the first time when chopping down my ROMs to a smaller number, and so far, so good! I have only done a few systems, but it works exceptionally well - the RetroPie-Setup integration is also a massive plus for convenience.

I have one question/suggestion though, I noticed that when doing re-scrapes on a platform, if any roms had been deleted since the last scrape, the gamelist.xml would be updated to reflect that (eg. old game removed), but the media will always stay there.

I assume this is intentional so it is safer for the end user, but is there a switch or a way to have it so the old orphaned media is purged when a scrape takes place?

I thought the --cleandb switch might do this (which isn't available in the GUI) but when I tried to run it via CLI, it didn't seem to remove the media.

Do either of you have any suggestions to help me keep the media tidy/in sync with the roms and gamelist.xml?

Cheers!

mitu

@movisman

I have one question/suggestion though, I noticed that when doing re-scrapes on a platform, if any roms had been deleted since the last scrape, the gamelist.xml would be updated to reflect that (eg. old game removed), but the media will always stay there.

That's the behavior of every scraper, it cannot erase what was downloaded through another program.

I assume this is intentional so it is safer for the end user, but is there a switch or a way to have it so the old orphaned media is purged when a scrape takes place?

Not from the scraper, since they're out the actual gamelist and they could be - potentially - everywhere. Furthermore, it could cause problems for users that are just trying out Skyscraper and might have custom artwork/screenshots - we don't want that to get erased just from running Skyscraper.

I thought the --cleandb switch might do this (which isn't available in the GUI) but when I tried to run it via CLI, it didn't seem to remove the media.

It's normal, since --cleandb cleans the Local DB and not the output of the scraping (i.e. arwork referenced in gamelist).

However, since Skyscraper caches the media, you can remove (or just move out of the way) the old media(s) folder (used by the old gamelists) and re-scrape again, it will re-generate a clean gamelist and media folder without downloading anything :).

movisman

@mitu

Hi there,

Ah, just to clarify, there was no media or gamelist present before starting to use Skyscraper. I deleted all of the files for that system before scraping. So in my test case, I removed all of the media and the gamelist.xml from Atari 2600 and 7800 before scraping.

With this in mind, should it be cleaning up media files if a ROM is removed and a rescrape is performed (bearing in mind the first scrape is done by Skyscraper)?

Thanks!

brimby

@mitu I'm seeing it in "experimental packages" not optional. Was this changed?

muldjord

@brimby Yes, it was moved to Experimental as we found some problems with it. They have been fixed now though, so it might be moved back to Optional again soon.

muldjord

@movisman If you mean the cached data, then yes, this will be possible in a future version of Skyscraper. It's not possible currently though. But it's on my roadmap. Skyscraper won't ever remove any media from the media folder. As @mitu mentioned it is not possible to know what files the user wants to keep and which to delete. So that will never be an option.

IanDaemon

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@brimby Yes, it was moved to Experimental as we found some problems with it. They have been fixed now though, so it might be moved back to Optional again soon.

I can't find it in either Optional nor Experimental. Was this removed temporarily?

mitu

@IanDaemon What version of RetroPie do you have installed ? Did you scroll down in the experimental section to see all packages ?

movisman

@muldjord

Hi there,

Thanks for the reply! No worries at all, I understand. I thought perhaps if Skyscraper kept track of the media that it scraped itself, it would perhaps have an option to purge them at a later stage and ignore any user generated files. I get why this wouldn't really work though I think.

I guess the only other option is to have an overwrite switch, where it overwrites the destination media folder upon scrape, removing whatever was there before and starting afresh. But again, I understand why you perhaps wouldn't implement this, due to risk involved of users selecting this option accidentally or not understanding what it does.

It is fine for me, because if I change ROMs around at any stage, I can just delete the media folder and rescrape if I wish to purge the old files which are no longer referenced due to the ROM being deleted. It was just something I noticed and thought i'd ask if it was possible with a switch or could be a consideration for the future.

Cheers!

IanDaemon

@mitu I have RetroPie 4.4. I'll look again tonight.

mitu

@IanDaemon 4.4 is the major release - the current version is 4.4.4, if you have updated recently. You can upgrade just the RetroPie-Setup script (sans packages) to get Skyscraper as new package in the experimental section.

movisman

@muldjord

Hi there,

I have a quick question about the passes the scraper makes on each ROM and the ability to scrape / search manually. I recently converted my 21 Dreamcast roms to .chd format, which I will do with some other systems eventually. This just makes the folder view way more tidy and manageable, as I now just have a single file for each game. Now, I know scraping a user-made CHD will likely not scrape, as it'll have it's own attributes and filename. I was right, of course it didn't match with any games.

So I figured I would scrape the original .gdi files (pre-conversion), and then just adjust the gamelist to suit. It's just 21 games, so no problem.

However, I ran into a problem scraping some DC .gdi files - I discovered that quite a few games share the same SHA1 value, mainly because a lot of the .gdi files have the same content (as they don't have any mention of the game inside, they just point to 'track 1', 'track 2', etc).

Anyway, this led to some very interesting scraping results, with a load of games simply scraping as a game called 'D-2'! Also the gamelist.xml was a bit all over the place, I guess because of the same SHA1 being matched multiple times for multiple files.

However, I understand the last pass the scraper makes, is to check for a full filename match (must be exact), if it cannot find a match for SHA1, etc. So, I thought as a workaround for these files, I would find a supported file on screenscraper.fr, rename my .gdi ROM to that, and scrape again.

This didn't work because, of course, it would match this common SHA1 value first and always take that as a priority over filename. I should have known that.

Then I realised, I could just rename each CHD file to a filename match on the screenscraper.fr site, including changing the extension. Because the SHA1 value would be unique on my user made CHD, it should in theory go for the filename. This worked! Scraped the ROMs and got all screenshots and videos. I then however, had to go back in and rename the roms again back to .chd extensions, and adjust the gamelist.xml to suit. Bit of a pain, but it did work.

So this, as a workaround was fine, but I have two questions (one is more of an 'enhancement'):

Can the user adjust the priorities of what each 'pass' tries to match? Eg. Can I change it to match the filename first over say, SHA1? This would then solve the issue of the scraper matching with a common SHA1 for files like .gdi. I understand this is probably a rare occurrence and maybe only isolated to DC, and maybe it is already possible to adjust the order! Just to clarify i'm not suggesting any change to the default (which makes perfect sense), just the ability to change the priority and if that is currently possible?
Second one, which would help with all the manual work above, is would you consider a manual lookup for a future version, if there is no match found OR if the user just wants to confirm each scrape as it is performed? So, as it scrapes a game, it asks you to confirm, if you say 'no', you can select from a list returned from a search based on the filename, or maybe you can type in all or part of the game you think it should be. It would then come back with a list where you can choose the correct ROM. Something like that anyway.

Not sure how hard the above is to implement (I suspect time consuming) but it would all but solve the issues I had above. I could then simply feed unique CHD's to skyscraper, and although it wouldn't match on the automatic passes because there is no match on screenscraper.fr, it would prompt me each time to type in some words to help find a match, or perhaps it could do a wildcard search based on part of the filename minus extension and give you some potential results to choose from.

Sorry the above is a bit long winded, just making a possible couple of suggestions based on my user experience.

Cheers!

Hyperlord

Hi guys,

great, I'm loving skyscraper!

I had it installed manually and ran it for a couple of month, now I installed the built-in version and just "copied" my db-folder from /home/pi/.skyscraper to /opt/.../skyscraper

Was this "right"? Am I missing something?
As I recently tripled the free space on the SD card I want to "rescrape" with videos this time. Am I able to do this or should I start from scratch?

mitu

@Hyperlord If you already had a Skyscraper installation, the configuration script in the module would have done this automatically - migrate your settings and move over your LocalDB. In fact, your $HOME/.skyscraper folder should be now symlinked to /opt/retropie/configs/all/skyscraper, so both versions should be running ok.
Do you have any problems after installing Skyscraper from the RetroPie-Setup script ?

muldjord

@movisman

Answer to first question:
I can add the .gdi extension to the list of file types that will do a sha1 checksum on the filename instead of the content. I already do this for script files that change often.

Answer to second question:
There is already a manual lookup although it isn't documented well, so I can easily understand why you haven't seen it. It's the "--query" option. Read more about it here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

movisman

@muldjord

Hi there,

Thanks for the heads up about the --query option, i'll check that out.

As for the .gdi thing, that sounds good, because quite a lot of the games (it seems) share the same checksum for the .gdi file.

Thanks!

movisman

@muldjord

Hi there,

Thanks for the heads up with the --query option. I had a play around with it and I was able to scrape one of the Gamecube games I had trouble with, by querying a recognised SHA1 on screenscraper.fr against my filename.

However, none of the gamelist or folders populate with the scraped media even though they are valid. I suspect it's probably this message which comes up after the command is run:

Pretend set! Not changing any files, but still caching resources in local database.

If I check the DB folder I can see the files there though, so the scrape happened, it just didn't write the media or populate the gamelist.xml.

I was testing outside of RetroPie on my laptop, so wasn't working within the system at all. Not sure if this affects anything? The command I ran for a test on that specific file was:

Skyscraper --unattend --skipped -i ~/Documents/scraper/ -g ~/Documents/scraper -o ~/Documents/scraper/media --relative -p gc -s screenscraper --videos --nobrackets --query sha1=F5ECEF62B8A6ED8F76EEC3319BE7168352C5D5AE ~/Documents/scraper/Capcom\ vs.\ SNK\ 2\ EO\ -\ Millionaire\ Fighting\ 2001\ (USA).iso

I tried various other commands, simplified etc, but the same issue occurs every time, and that message about 'pretend mode' always comes up.

Any idea?

Thanks!