Skyscraper now officially part of RetroPie, please test

mitu · 24 Oct 2018, 17:42

@muldjord said in Skyscraper now officially part of RetroPie, please test:

Sorry to lay all this on you, this is the first time I've had the opportunity to check it out on a fresh install. Just got an extra SD card so I could test it. :)

No problem. Just do a source install and check other files that may be missing. Btw, you can do this on your Ubuntu install, its not restricted to the PI.

@buzz - I think you created a binary package for the skyscraper module, but I haven't accounted for that in the scriptmodule - so configuration expects some files in the build folder to be present so it fails.
Can you please remove the binary package until I figure this out ? This way the source install would work fine.

BuZz · 24 Oct 2018, 18:20

@mitu hmm ok. The install stage should put everything that's needed into the install folder. Anything else outside of that needs to be in the configure stage. Maybe it's best to put this in experimental for now as I don't build binaries for those. Probably should have put it in exp initially in hindsight as its a new module etc.

mitu · 24 Oct 2018, 18:52

@BuZz said in Skyscraper now officially part of RetroPie, please test:

Probably should have put it in exp initially in hindsight as its a new module etc.

Yes, I didn't know about binaries built automatically != exp or the nobin flag. I should have picked one of them. Thanks. I'll incorporate some feedback from @muldjord and do a PR then probably add another change later to accomodate the binary install.

BuZz · 24 Oct 2018, 19:29

@mitu the nobin flag was only just re-introduced again so it wasn't there when you built the module. I don't build binaries automatically for experimental packages though. I have moved the module to experimental for now.

To make the module work from a binary install, everything needed will need to be in /opt/retropie/supplementary/skyscraper.

the configure script can copy data from this location (eg default configs), later. Then it will work from a binary install.

mitu · 25 Oct 2018, 06:22

@BuZz said in Skyscraper now officially part of RetroPie, please test:

To make the module work from a binary install, everything needed will need to be in /opt/retropie/supplementary/skyscraper.

Submitted https://github.com/RetroPie/RetroPie-Setup/pull/2520 to fix this. Now all config files are pulled from $md_inst and the install has all the necessary files included.

@muldjord I've incorporated almost all your suggestions - what's left is the config.ini settings for the input, roms and dbs (I removed the artwork.xml). We can probably remove them in a subsequent modifications.
One thing I added as default is --unpack, I think it should be a default since most ROMs are zipped and it should make problems like I renamed by ROM to XYZ.zip be less prevalent.
I also fixed the Couldn't calculate sha1 hash sum of rom file '', please check permissions and try again, now exiting..." error, reported in

https://retropie.org.uk/forum/topic/19739/ and I think also in
https://old.reddit.com/r/RetroPie/comments/9qqi23/help_skyscraper_not_working/

I've encountered this error when testing, but I assumed it was because some of my systems had bogus zip files and Skyscraper couldn't unpack them (seemed to work fine with other scraping sources that don't use checksums). But it turned out it's a shell params expansion thing and I've found a simple workaround for it.

muldjord · 25 Oct 2018, 06:28

@mitu said in Skyscraper now officially part of RetroPie, please test:

One thing I added as default is --unpack, I think it should be a default since most ROMs are zipped and it should make problems like I renamed by ROM to XYZ.zip be less prevalent.

Nonono, please remove that again. --unpack is implemented in a way that could give users problems with memory (and a performance hit) since it doesn't feed the checksum calculation data in segments. It works just fine without it as most zips are already in the screenscraper db, and all other sources are name based so they don't use it anyway.

I general anything that isn't default shouldn't be added by default. I usually add options to Skyscraper in a way that things that should be default, I just enable them, and then I add the opposite bool option to allow users to disable it. So, for instance, for --unpack, if it should be enabled by default, I would add '--nounpack'.

mitu · 25 Oct 2018, 06:33

@muldjord OK, no problem.

movisman · 25 Nov 2018, 21:22

@muldjord @mitu

Hi,

Just tried using this scraper for the first time when chopping down my ROMs to a smaller number, and so far, so good! I have only done a few systems, but it works exceptionally well - the RetroPie-Setup integration is also a massive plus for convenience.

I have one question/suggestion though, I noticed that when doing re-scrapes on a platform, if any roms had been deleted since the last scrape, the gamelist.xml would be updated to reflect that (eg. old game removed), but the media will always stay there.

I assume this is intentional so it is safer for the end user, but is there a switch or a way to have it so the old orphaned media is purged when a scrape takes place?

I thought the --cleandb switch might do this (which isn't available in the GUI) but when I tried to run it via CLI, it didn't seem to remove the media.

Do either of you have any suggestions to help me keep the media tidy/in sync with the roms and gamelist.xml?

Cheers!

mitu · 25 Nov 2018, 22:01

@movisman

I have one question/suggestion though, I noticed that when doing re-scrapes on a platform, if any roms had been deleted since the last scrape, the gamelist.xml would be updated to reflect that (eg. old game removed), but the media will always stay there.

That's the behavior of every scraper, it cannot erase what was downloaded through another program.

I assume this is intentional so it is safer for the end user, but is there a switch or a way to have it so the old orphaned media is purged when a scrape takes place?

Not from the scraper, since they're out the actual gamelist and they could be - potentially - everywhere. Furthermore, it could cause problems for users that are just trying out Skyscraper and might have custom artwork/screenshots - we don't want that to get erased just from running Skyscraper.

I thought the --cleandb switch might do this (which isn't available in the GUI) but when I tried to run it via CLI, it didn't seem to remove the media.

It's normal, since --cleandb cleans the Local DB and not the output of the scraping (i.e. arwork referenced in gamelist).

However, since Skyscraper caches the media, you can remove (or just move out of the way) the old media(s) folder (used by the old gamelists) and re-scrape again, it will re-generate a clean gamelist and media folder without downloading anything :).

movisman · 25 Nov 2018, 22:19

@mitu

Hi there,

Ah, just to clarify, there was no media or gamelist present before starting to use Skyscraper. I deleted all of the files for that system before scraping. So in my test case, I removed all of the media and the gamelist.xml from Atari 2600 and 7800 before scraping.

With this in mind, should it be cleaning up media files if a ROM is removed and a rescrape is performed (bearing in mind the first scrape is done by Skyscraper)?

Thanks!

brimby · 27 Nov 2018, 22:22

@mitu I'm seeing it in "experimental packages" not optional. Was this changed?

muldjord · 28 Nov 2018, 14:03

@brimby Yes, it was moved to Experimental as we found some problems with it. They have been fixed now though, so it might be moved back to Optional again soon.

muldjord · 28 Nov 2018, 14:06

@movisman If you mean the cached data, then yes, this will be possible in a future version of Skyscraper. It's not possible currently though. But it's on my roadmap. Skyscraper won't ever remove any media from the media folder. As @mitu mentioned it is not possible to know what files the user wants to keep and which to delete. So that will never be an option.

IanDaemon · 29 Nov 2018, 19:59

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@brimby Yes, it was moved to Experimental as we found some problems with it. They have been fixed now though, so it might be moved back to Optional again soon.

I can't find it in either Optional nor Experimental. Was this removed temporarily?

mitu · 29 Nov 2018, 20:09

@IanDaemon What version of RetroPie do you have installed ? Did you scroll down in the experimental section to see all packages ?

movisman · 29 Nov 2018, 21:13

@muldjord

Hi there,

Thanks for the reply! No worries at all, I understand. I thought perhaps if Skyscraper kept track of the media that it scraped itself, it would perhaps have an option to purge them at a later stage and ignore any user generated files. I get why this wouldn't really work though I think.

I guess the only other option is to have an overwrite switch, where it overwrites the destination media folder upon scrape, removing whatever was there before and starting afresh. But again, I understand why you perhaps wouldn't implement this, due to risk involved of users selecting this option accidentally or not understanding what it does.

It is fine for me, because if I change ROMs around at any stage, I can just delete the media folder and rescrape if I wish to purge the old files which are no longer referenced due to the ROM being deleted. It was just something I noticed and thought i'd ask if it was possible with a switch or could be a consideration for the future.

Cheers!

IanDaemon · 30 Nov 2018, 16:06

@mitu I have RetroPie 4.4. I'll look again tonight.

mitu · 30 Nov 2018, 17:10

@IanDaemon 4.4 is the major release - the current version is 4.4.4, if you have updated recently. You can upgrade just the RetroPie-Setup script (sans packages) to get Skyscraper as new package in the experimental section.

movisman · 4 Dec 2018, 10:55

@muldjord

Hi there,

I have a quick question about the passes the scraper makes on each ROM and the ability to scrape / search manually. I recently converted my 21 Dreamcast roms to .chd format, which I will do with some other systems eventually. This just makes the folder view way more tidy and manageable, as I now just have a single file for each game. Now, I know scraping a user-made CHD will likely not scrape, as it'll have it's own attributes and filename. I was right, of course it didn't match with any games.

So I figured I would scrape the original .gdi files (pre-conversion), and then just adjust the gamelist to suit. It's just 21 games, so no problem.

However, I ran into a problem scraping some DC .gdi files - I discovered that quite a few games share the same SHA1 value, mainly because a lot of the .gdi files have the same content (as they don't have any mention of the game inside, they just point to 'track 1', 'track 2', etc).

Anyway, this led to some very interesting scraping results, with a load of games simply scraping as a game called 'D-2'! Also the gamelist.xml was a bit all over the place, I guess because of the same SHA1 being matched multiple times for multiple files.

However, I understand the last pass the scraper makes, is to check for a full filename match (must be exact), if it cannot find a match for SHA1, etc. So, I thought as a workaround for these files, I would find a supported file on screenscraper.fr, rename my .gdi ROM to that, and scrape again.

This didn't work because, of course, it would match this common SHA1 value first and always take that as a priority over filename. I should have known that.

Then I realised, I could just rename each CHD file to a filename match on the screenscraper.fr site, including changing the extension. Because the SHA1 value would be unique on my user made CHD, it should in theory go for the filename. This worked! Scraped the ROMs and got all screenshots and videos. I then however, had to go back in and rename the roms again back to .chd extensions, and adjust the gamelist.xml to suit. Bit of a pain, but it did work.

So this, as a workaround was fine, but I have two questions (one is more of an 'enhancement'):

Can the user adjust the priorities of what each 'pass' tries to match? Eg. Can I change it to match the filename first over say, SHA1? This would then solve the issue of the scraper matching with a common SHA1 for files like .gdi. I understand this is probably a rare occurrence and maybe only isolated to DC, and maybe it is already possible to adjust the order! Just to clarify i'm not suggesting any change to the default (which makes perfect sense), just the ability to change the priority and if that is currently possible?
Second one, which would help with all the manual work above, is would you consider a manual lookup for a future version, if there is no match found OR if the user just wants to confirm each scrape as it is performed? So, as it scrapes a game, it asks you to confirm, if you say 'no', you can select from a list returned from a search based on the filename, or maybe you can type in all or part of the game you think it should be. It would then come back with a list where you can choose the correct ROM. Something like that anyway.

Not sure how hard the above is to implement (I suspect time consuming) but it would all but solve the issues I had above. I could then simply feed unique CHD's to skyscraper, and although it wouldn't match on the automatic passes because there is no match on screenscraper.fr, it would prompt me each time to type in some words to help find a match, or perhaps it could do a wildcard search based on part of the filename minus extension and give you some potential results to choose from.

Sorry the above is a bit long winded, just making a possible couple of suggestions based on my user experience.

Cheers!

Hyperlord · 4 Dec 2018, 14:52

Hi guys,

great, I'm loving skyscraper!

I had it installed manually and ran it for a couple of month, now I installed the built-in version and just "copied" my db-folder from /home/pi/.skyscraper to /opt/.../skyscraper

Was this "right"? Am I missing something?
As I recently tripled the free space on the SD card I want to "rescrape" with videos this time. Am I able to do this or should I start from scratch?