Skyscraper now officially part of RetroPie, please test

muldjord

UPDATED Jan 27th 2019:
Skyscraper is available from the RetroPie-Setup menu. Find it under "Experimental" packages. It will be moved to "Optional" packages at some point.

Alternate installation method

If you've already installed Skyscraper by using the installation guide at the official Skyscraper github and wish to keep using it like that, feel free to do so. That method is not deprecated. Installing through the RetroPie-Setup script is an additional method, not a substitute.

Basic usage suggestion

Check the RetroPie readme on how to use it through the script here.

Full documentation

Skyscraper is documented down to the last detail. Check the official docs for it here.

A huge thanks to @mitu for creating the RetroPie setup script. And thank you to @BuZz for reviewing the script. If you have any issues, please let us know so we can look into it. Please check it out if you feel like it, and feedback is of course more than welcome!

Happy scraping!

mitu

Just to add to @muldjord's announcement

the new skyscraper module is added in the Optional section of the packages. Make sure you first update your RetroPie-Setup script to get the new module.
the Skyscraper usage from the RetroPie-Setup has a gui (similar to Steven Selph's scraper) that can be accessed either from the Configuration/Tools menu of the RetroPie-Setup or using the Configuration/Options menu of the package.

0_1539341753292_2d4d68da-bea6-4f9c-a44f-bee87d3d5ecd-image.png

More information will be added in the Wiki to explain better how to use Skyscraper from the RetroPie-Setup's interface.

cyperghost

@muldjord Finally done ;) This is really a nice piece of software - valueable use!
@mitu Thank you for writing the script module and the bash script so we have a text inputs and less work with CLI.

mitu

@cyperghost said in Skyscraper now officially part of RetroPie, please test:

@mitu Thank you for writing the script module and the bash script so we have a text inputs ;)

What ? There are no input boxes in the scriptmodule :)

cyperghost

@mitu Text Input via menu selections ;)

mitu

@muldjord - I have a WIP draft for the Wiki page at Scraping. I've re-organized the page and added sections for each scraper, keeping the old information and adding a section for Skyscraper.
I've not added info for the Artwork and Import module and I have to re-organize the section about the location of metadata/media files on the system (Markdown is quite limited when it comes to tables). I think you're changing some options for the Import module, so we should incorporate that.
Feel free to add any commends or suggest modifications.

robertvb83

@muldjord great great work. Even though i did not try it yet, I like your concept of local-db very much. this makes this scraper really stand out. Over time you can improve the local-db (even manually) and minimize traffic for the online dbs. If I don't like the art or need to change some little things there is no need to scrape a lot of data again and again, Genius! thank you!

@mitu thanks for improving the wiki already. makes it easy to get startet!

I have one suggestion. instead of everyone making there own artwork.xml adaptions it would be nice to have some predefined options to chose from. e.g. 3D cover boxart or mixed with sreenshot, marquee, box.
would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

muldjord

@mitu Looks good, here are my comments:

Skyscraper by Lars Muldjord is a powerful and versatile yet easy to use game scraper written in C++ for use with multiple frontends running on a Linux system. It scrapes and caches various game resources from various web sources, including media such as screenshot, cover and video. It then gives you the option to combine all of these resources into the most complete results by using the provided 'localdb' scraping module.

Could you add this line beneath this: "For a more thorough description of any functionality described in this howto, please check out the official Skyscraper documentation here: https://github.com/muldjord/skyscraper

NOTE: Skyscraper can be used to generate scraping information for both [FIXME:Emulationstation][Emulationstation] and [FixMelAttractMode][AttractMode]. Installing Skyscraper as a RetroPie package will automatically configure it for Emulationstation, but if you wish to use it for generating AttachMode metadata, you can use Skyscraper from the command line.

"AttachMode" should be "AttractMode".

Can you also add a section just below this with: "Recommended usage is to scrape your platform(s) with as many online sources as you'd like. All of the data will be cached while doing so. Then, when you have gathered enough data, be sure to always scrape it one last time using the 'Local: LocalDb' source. This will combine all of your cached data into the most complete results for each rom."

Remove bracket info: Choose to remove (Enabled) or keep (Disabled) the text between brackets when using the Filename option for ROM Names. If the name of the ROM is Super Mario World (USA)[!].sfc, enabling this option will make the ROM show as Super Mario World in Emulationstation.

This will actually also happen when using the source titles. It strips the bracket info from the filename and adds it back to the frontend name no matter which one you've chosen to use.

Force cache refresh: Skyscraper caches the media retrieved from online sources locally to speed up scraping and to combine multiple sources of information to give you the best scraped information available. This option forces Skyscraper to bypass the locally cached resources and re-download the media already cached from the internet.

Can you change "...bypass the locally cached resources and re-download the media already cached from the internet." to "...bypass the locally cached resources and re-download the media from the online sources."

Download Videos: Choose to toggle the download and caching of videos for the ROM names. This also enables adding the video to the gamelist in Emulationstation.

Could you add "NOTE: Be aware that fetching and caching videos takes up a lot of disk space."

Lastly, I think for artwork, just point users to my official docs here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

I've spent a lot of time on that including image examples and even a video (more to come when I get the time).

Same thing about the import module. The doc for this is here: https://github.com/muldjord/skyscraper/blob/master/import/README.md

And lastly, the doc for localdb: https://github.com/muldjord/skyscraper/blob/master/dbs/README.md

I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

muldjord

@robertvb83 That is actually a good idea with the different artwork.xml configs. I think we'll let this settle first though and see how users react to it.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

Sure thing. For the Artwork/Import my idea was just to say a few words and point to the official docs anyway, there's too much to cover and it's just silly to duplicate things instead of using the source.

@robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

Is this a template in XML scraper (I assume you're referring to UXS) ?

mitu

@muldjord I've modified the WIP wiki page (link here) and added your suggestions. For the advanced part I've added your descriptions for Import and Artwork and linked to the official docs. I've removed the CLI usage since this is going to evolve and will probably be outdated.

muldjord

@mitu Awesome! Love the artwork example, looks good. I just went over it all real quick, I don't have further comments, looks good to me.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

Love the artwork example, looks good

I couldn't resist putting it there, I initially thought of adding a table with each example artwork and it's preview, but I think that one table is suggestive enough to grasp the 'magic' behind the artwork compositor.

muldjord

@mitu Agreed, it's nice to have it there as a teaser for what people can do and it works well to have the individual pieces first, and then the result. I think that explains how it works well. We can add more examples if we create a bunch of default artwork.xml files for people to use.

mitu

Page is now live at https://retropie.org.uk/docs/Scraper

muldjord

Posted on reddit now: https://www.reddit.com/r/RetroPie/comments/9p1zqq/skyscraper_now_officially_part_of_retropie/

robertvb83

@mitu said in Skyscraper now officially part of RetroPie, please test:

@robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

Is this a template in XML scraper (I assume you're referring to UXS) ?

Yes i am referring to UXS. They actually have a similar art output mixer but they also have the option to show a little arcade cabinet with screenshot and marquee as you can see here (scroll down a little)

muldjord

@robertvb83 This can actually be achieved relatively easily using the Skyscraper compositor. It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet. I will consider creating an "arcade cabinet" artwork.xml and we can add that on a list of artwork templates. But I need a completely royalty free arcade cabinet overlay to do that. Creating one from scratch is a bit much for my artistic skills.

Clyde

@muldjord said in Skyscraper now officially part of RetroPie, please test:

It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet.

So it can put scanlines in the videos? That would be great, since ES and the OMX Player can't do that on the fly like Attract Mode can, for example.

muldjord

@Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.