Skyscraper now officially part of RetroPie, please test

Silent

Also, for the sake of "noob friendliness" I think the nice end goal would be to see Skyscraper as a backend for ES scraping. Keep in mind however that I am saying this without checking how that ES feature works at all, so perhaps it's never going to be possible.

mitu

@muldjord PR submitted - https://github.com/RetroPie/RetroPie-Setup/pull/2584. Since it should work with both versions, can be merged before you release 3.x.
Fixed both bugs ~~fixed~~ reported so far (the one with the ROM folder containing only sub-folders and the one with the symlink-ed ROMs folder).

muldjord

@mitu Great! I am working hard on the documentation at this very moment. I will try to squeeze in some time to test the script later tonight. I'll ping you if I run into anything that needs fixing.

mitu

@muldjord Take your time, documentation is always hard to write (for your own applications :) ).

muldjord

@mitu Just testing the script from the PR. For some reason the 3 Gather source: ... Option doesn't show up on my menu. Any idea why that is?

mitu

@muldjord I think I renamed it to 3 Gather source - <source> -->.

muldjord

@mitu Yeah, but it doesn't show up on my menu. I just have Option 1, 2, 4, 5 and so on. No 3.

mitu

@muldjord What's the md5 sum of the script ?

 $ md5sum `pwd`/scriptmodules/supplementary/skyscraper.sh 
7070a79beb0629a70478d36ff56e8742  /home/pi/RetroPie-Setup/scriptmodules/supplementary/skyscraper.sh

muldjord

@mitu

md5sum `pwd`/scriptmodules/supplementary/skyscraper.sh
84e97f7fd7fc6d4ab6701203d83e8482  /home/pi/RetroPie-Setup/scriptmodules/supplementary/skyscraper.sh

I got it by wgetting the raw file from the PR. Do I need to fetch it from somewhere else?

mitu

No, it should be ok. What do you have in the /opt/retropie/configs/all/skyscraper.cfg file ?

muldjord

@mitu

rom_name = "0"
use_rom_folder = "1"
download_videos = "1"
cache_marquees = "0"
cache_covers = "0"
cache_wheels = "0"
cache_screenshots = "1"
scrape_source = "localdb"
remove_brackets = "0"
force_refresh = "1"

It has "localdb" from an old installation. Might that be the reason?

EDIT: Yep, that's it. Changed it to "thegamesdb" and the menu option shows up. This should be handled somehow. Check if it has "localdb" and just set it to "screenscraper" or so if it is.

mitu

@muldjord Yes, because now localdb is no longer an option - I'll fix it. You can just nuke the .cfg file and start with the defaults (just need to handle this for upgrades with the fix).

muldjord

@mitu Tested the rest and it works really well. I think the workflow is a lot better now. In my opinion it is ready to be merged.

EDIT:

Minor. The ArcadeDB source is noted as just "Arcade" right now. It should be called "ArcadeDB".
IGDB should be removed. It has a 5 max rom limit in the vein of mobygames, so it's not really usable from the menu. The reasoning behind the limit is that it know has my user KEY added and it has a global limit of 10000 requests per month for all Skyscraper users.

mitu

@muldjord Should be fixed now - scraping source/removed IGDB/fixed spelling.

muldjord

Just a sec, found another error. Gonna describe it in a minute.

EDIT: Ah, nevermind. It's an error that I have already fixed in 3.0.0. It's not related to the script. Basically it's related to the --relative option when users have their roms in subfolders it would add the wrong rom location to the gamelist. But this is fixed in 3.0.0.

I have nothing further to add. Works really well! I'm pretty excited about this. :)

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I have nothing further to add. Works really well! I'm pretty excited about this. :)

Alright - I'll buzz it the morning for the final review before merge. I still have to modify the wiki after the merge, but shouldn't be take long.

mitu

@muldjord Here's an updated Wiki page (minus the screenshot for the UI) at https://gist.github.com/cmitu/8d22c2e9548de8ff405ec977cff49fa0.

muldjord

@mitu Awesome, couple notes:

LOCAL: esgamelist - Scrapes and caches data from an EmulationStation gamelist.xml located at $HOME/RetroPie/roms/[platform]/gamelist.xml or ~/.skyscraper/import/gamelist.xml

The last path should be ~/.skyscraper/import/[platform]/gamelist.xml

... local cache

Could you search replace all instances "local cache" to "resource cache" instead? That's what it's referred to in the Skyscraper documentation.

use this with care, since it removes any cached resource. Any subsequent scraping will have to re-download the media files again from the online sources.

Could you change this to "use this with care, it will completely remove the cached resources used when generating the game lists for the chosen platform(s). Generating game lists after you've cleared the cache requires you to re-download the media from the scraping sources all over again."

This will generate the Emulationstation game list(s) for the systems you choose. Press Space to select each system you wish to incluse and OK to start scraping.

Should be "This will generate the Emulationstation game list(s) for the systems you choose using all previously cached resources. Press Space to select the system(s) you wish to include and OK to start the generator."

Edit the config.ini file.

Add link to: https://github.com/muldjord/skyscraper/blob/master/docs/CONFIGINI.md

Edit the artwork.xml file.

Add link to: https://github.com/muldjord/skyscraper/blob/master/docs/ARTWORK.md

Consult the official import documentation to understand how to use this feature.

Broken link. Change to: https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

Lastly, please add a link to the CLI options after this line:

but if you wish to use it for generating AttractMode metadata, you can use Skyscraper from the command line.

Add "Check the official documentation for a list of all command line options." with a link to: https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

I think that's it.

mitu

@muldjord

Thanks for the review, I've incorporated the suggestions.

The last path should be ~/.skyscraper/import/[platform]/gamelist.xml

Maybe needs to be correct also in the README.md of the repository (from where I got it). The link to the import module in the previous line is also not pointing inside the docs now.

muldjord

@mitu Fixed, thanks for notifying me.

EDIT: Btw, the screenshot you have in the wiki is currently missing option 3.