Skyscraper now officially part of RetroPie, please test

muldjord

@mitu Tested the rest and it works really well. I think the workflow is a lot better now. In my opinion it is ready to be merged.

EDIT:

Minor. The ArcadeDB source is noted as just "Arcade" right now. It should be called "ArcadeDB".
IGDB should be removed. It has a 5 max rom limit in the vein of mobygames, so it's not really usable from the menu. The reasoning behind the limit is that it know has my user KEY added and it has a global limit of 10000 requests per month for all Skyscraper users.

mitu

@muldjord Should be fixed now - scraping source/removed IGDB/fixed spelling.

muldjord

Just a sec, found another error. Gonna describe it in a minute.

EDIT: Ah, nevermind. It's an error that I have already fixed in 3.0.0. It's not related to the script. Basically it's related to the --relative option when users have their roms in subfolders it would add the wrong rom location to the gamelist. But this is fixed in 3.0.0.

I have nothing further to add. Works really well! I'm pretty excited about this. :)

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I have nothing further to add. Works really well! I'm pretty excited about this. :)

Alright - I'll buzz it the morning for the final review before merge. I still have to modify the wiki after the merge, but shouldn't be take long.

mitu

@muldjord Here's an updated Wiki page (minus the screenshot for the UI) at https://gist.github.com/cmitu/8d22c2e9548de8ff405ec977cff49fa0.

muldjord

@mitu Awesome, couple notes:

LOCAL: esgamelist - Scrapes and caches data from an EmulationStation gamelist.xml located at $HOME/RetroPie/roms/[platform]/gamelist.xml or ~/.skyscraper/import/gamelist.xml

The last path should be ~/.skyscraper/import/[platform]/gamelist.xml

... local cache

Could you search replace all instances "local cache" to "resource cache" instead? That's what it's referred to in the Skyscraper documentation.

use this with care, since it removes any cached resource. Any subsequent scraping will have to re-download the media files again from the online sources.

Could you change this to "use this with care, it will completely remove the cached resources used when generating the game lists for the chosen platform(s). Generating game lists after you've cleared the cache requires you to re-download the media from the scraping sources all over again."

This will generate the Emulationstation game list(s) for the systems you choose. Press Space to select each system you wish to incluse and OK to start scraping.

Should be "This will generate the Emulationstation game list(s) for the systems you choose using all previously cached resources. Press Space to select the system(s) you wish to include and OK to start the generator."

Edit the config.ini file.

Add link to: https://github.com/muldjord/skyscraper/blob/master/docs/CONFIGINI.md

Edit the artwork.xml file.

Add link to: https://github.com/muldjord/skyscraper/blob/master/docs/ARTWORK.md

Consult the official import documentation to understand how to use this feature.

Broken link. Change to: https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

Lastly, please add a link to the CLI options after this line:

but if you wish to use it for generating AttractMode metadata, you can use Skyscraper from the command line.

Add "Check the official documentation for a list of all command line options." with a link to: https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

I think that's it.

mitu

@muldjord

Thanks for the review, I've incorporated the suggestions.

The last path should be ~/.skyscraper/import/[platform]/gamelist.xml

Maybe needs to be correct also in the README.md of the repository (from where I got it). The link to the import module in the previous line is also not pointing inside the docs now.

muldjord

@mitu Fixed, thanks for notifying me.

EDIT: Btw, the screenshot you have in the wiki is currently missing option 3.

bleuge

I was downloading today the libretro thumbnails packs, http://thumbnailpacks.libretro.com/

And was thinking if it could be useful to import all these images to the localdb in any way, I know you provide a way to import external data, I read it someday ago, need to refresh it.

I mean, importing these, i could mitigate a lot downloading and scrapping from other sites?

I'll test later and see if i can do importing external data, and report if this could be useful

muldjord

@bleuge Check this: https://github.com/muldjord/skyscraper/blob/master/docs/IMPORT.md

EDIT: I used the emumovies packs for the same thing a while back. Worked really well too. It's easiest if the pack you get your hands on is for the rom pack you have. Then the filenames are properly correct from the get-go, ready to use with the -s import module in Skyscraper.

mitu

@muldjord Yes, I see now - I'll retake the screenshot :).

hhromic

@bleuge didn't know about the thumbnails pack from Libretro, very interesting resource! Thanks for the tip.

muldjord

@mitu Someone noted on reddit that

After scraping the reminder to Generate Game Lists has a spelling error. "Generate Same Lists" lol. Again not a biggie but thought i'd let you know.

He also mentions that:

Just did a trial run. Generate games lists and Gather for systems takes a long time to progress to the next screen. Not a biggie obviously but might be something to optimise.

I'm not sure what exactly that means and I haven't verified it myself.

mitu

@muldjord Yes, the spelling is in the message shown after gathering chosen systems. I'll fix this.

For the other part, I'll do some more test runs, but it sounds similar to the old buffering problem (which was fixed and the fix is still in). I ran a few test (gather and generate) and it doesn't seem to be a difference between the scriptmodule command and a manual (CLI) run.

muldjord

@mitu Awesome, that's what I would have guessed. Thanks.

bleuge

@muldjord Thanks for confirmation,

I guess I'll write a script for importing all those images for every platform, I'll do this:

Thumbnail folder - Skyscraper import
named_boxarts -> /import/covers
named_snaps -> /import/screenshots
named_titles -> import/wheels?

Let's see how it works...

Thanks for 3.0!!

muldjord

@mitu A user reports:

I tried to update from source and got the following error:
Could not successfully install Scraper for EmulationStation by Lars Muldjord (/home/pi/RetroPie-Setup/tmp/build/skyscraper/ARTWORK.md not found)
I've never mucked directly with that folder, so I'm not sure why the file would be missing? I'll give it another go and see if I can get it
[EDIT] Crashed the 2nd time as well, but the package says 3.0.0 when I load it up now... Weird

I then asked if he updated the script prior to updating Skyscraper. He then proceeded to uninstall Skyscraper, update the script, and then reinstall Skyscraper from source, but still get this error. I'm thinking maybe it's because some of the files inside of the skyscraper.pro file have changed location?

mitu

@muldjord I've seen that. The error is from using an older version of the script - should have updated first. In the new version, ARTWORK.md is no longer checked by the install script. First thing I've done is update from the GUI with the new option and didn't get any errors :).

EDIT: fixed the spelling in the latest version. I still don't see a difference between scraping from the GUI and from the CLI in my tests - maybe there is a difference with the 2.x version that the user is actually seeing. I'll keep poking around.

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

@muldjord I've seen that. The error is from using an older version of the script - should have updated first. In the new version, ARTWORK.md is no longer checked by the install script. First thing I've done is update from the GUI with the new option and didn't get any errors :).

Good to know, thanks.

Stuffu

First off, I’ve tried skyscraper before without success.

Gave it a new try today and amiga section went flawlessly, awesome!

Now where I’m stuck. I was scraping c64 with screenscraper, everything as expected. When doing the same with thegamesdb, it just quits immediately with “forcing thread x to stop”.

Edit: screenscraper left out a lot, but I noticed that thegamesdb starts with 0 requests remaining, which it shouldn’t be.

Any suggestions here?

And thanks for a great scraper!