Skyscraper now officially part of RetroPie, please test

robertvb83

@muldjord great great work. Even though i did not try it yet, I like your concept of local-db very much. this makes this scraper really stand out. Over time you can improve the local-db (even manually) and minimize traffic for the online dbs. If I don't like the art or need to change some little things there is no need to scrape a lot of data again and again, Genius! thank you!

@mitu thanks for improving the wiki already. makes it easy to get startet!

I have one suggestion. instead of everyone making there own artwork.xml adaptions it would be nice to have some predefined options to chose from. e.g. 3D cover boxart or mixed with sreenshot, marquee, box.
would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

muldjord

@mitu Looks good, here are my comments:

Skyscraper by Lars Muldjord is a powerful and versatile yet easy to use game scraper written in C++ for use with multiple frontends running on a Linux system. It scrapes and caches various game resources from various web sources, including media such as screenshot, cover and video. It then gives you the option to combine all of these resources into the most complete results by using the provided 'localdb' scraping module.

Could you add this line beneath this: "For a more thorough description of any functionality described in this howto, please check out the official Skyscraper documentation here: https://github.com/muldjord/skyscraper

NOTE: Skyscraper can be used to generate scraping information for both [FIXME:Emulationstation][Emulationstation] and [FixMelAttractMode][AttractMode]. Installing Skyscraper as a RetroPie package will automatically configure it for Emulationstation, but if you wish to use it for generating AttachMode metadata, you can use Skyscraper from the command line.

"AttachMode" should be "AttractMode".

Can you also add a section just below this with: "Recommended usage is to scrape your platform(s) with as many online sources as you'd like. All of the data will be cached while doing so. Then, when you have gathered enough data, be sure to always scrape it one last time using the 'Local: LocalDb' source. This will combine all of your cached data into the most complete results for each rom."

Remove bracket info: Choose to remove (Enabled) or keep (Disabled) the text between brackets when using the Filename option for ROM Names. If the name of the ROM is Super Mario World (USA)[!].sfc, enabling this option will make the ROM show as Super Mario World in Emulationstation.

This will actually also happen when using the source titles. It strips the bracket info from the filename and adds it back to the frontend name no matter which one you've chosen to use.

Force cache refresh: Skyscraper caches the media retrieved from online sources locally to speed up scraping and to combine multiple sources of information to give you the best scraped information available. This option forces Skyscraper to bypass the locally cached resources and re-download the media already cached from the internet.

Can you change "...bypass the locally cached resources and re-download the media already cached from the internet." to "...bypass the locally cached resources and re-download the media from the online sources."

Download Videos: Choose to toggle the download and caching of videos for the ROM names. This also enables adding the video to the gamelist in Emulationstation.

Could you add "NOTE: Be aware that fetching and caching videos takes up a lot of disk space."

Lastly, I think for artwork, just point users to my official docs here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

I've spent a lot of time on that including image examples and even a video (more to come when I get the time).

Same thing about the import module. The doc for this is here: https://github.com/muldjord/skyscraper/blob/master/import/README.md

And lastly, the doc for localdb: https://github.com/muldjord/skyscraper/blob/master/dbs/README.md

I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

muldjord

@robertvb83 That is actually a good idea with the different artwork.xml configs. I think we'll let this settle first though and see how users react to it.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

Sure thing. For the Artwork/Import my idea was just to say a few words and point to the official docs anyway, there's too much to cover and it's just silly to duplicate things instead of using the source.

@robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

Is this a template in XML scraper (I assume you're referring to UXS) ?

mitu

@muldjord I've modified the WIP wiki page (link here) and added your suggestions. For the advanced part I've added your descriptions for Import and Artwork and linked to the official docs. I've removed the CLI usage since this is going to evolve and will probably be outdated.

muldjord

@mitu Awesome! Love the artwork example, looks good. I just went over it all real quick, I don't have further comments, looks good to me.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

Love the artwork example, looks good

I couldn't resist putting it there, I initially thought of adding a table with each example artwork and it's preview, but I think that one table is suggestive enough to grasp the 'magic' behind the artwork compositor.

muldjord

@mitu Agreed, it's nice to have it there as a teaser for what people can do and it works well to have the individual pieces first, and then the result. I think that explains how it works well. We can add more examples if we create a bunch of default artwork.xml files for people to use.

mitu

Page is now live at https://retropie.org.uk/docs/Scraper

muldjord

Posted on reddit now: https://www.reddit.com/r/RetroPie/comments/9p1zqq/skyscraper_now_officially_part_of_retropie/

robertvb83

@mitu said in Skyscraper now officially part of RetroPie, please test:

@robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

Is this a template in XML scraper (I assume you're referring to UXS) ?

Yes i am referring to UXS. They actually have a similar art output mixer but they also have the option to show a little arcade cabinet with screenshot and marquee as you can see here (scroll down a little)

muldjord

@robertvb83 This can actually be achieved relatively easily using the Skyscraper compositor. It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet. I will consider creating an "arcade cabinet" artwork.xml and we can add that on a list of artwork templates. But I need a completely royalty free arcade cabinet overlay to do that. Creating one from scratch is a bit much for my artistic skills.

Clyde

@muldjord said in Skyscraper now officially part of RetroPie, please test:

It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet.

So it can put scanlines in the videos? That would be great, since ES and the OMX Player can't do that on the fly like Attract Mode can, for example.

muldjord

@Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.

mitu

@muldjord

So it can put scanlines in the videos?

Does it work for videos ?

muldjord

@mitu Sorry, no, I was just editing my post while you were posting. :D No, it can't process videos.

Clyde

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

:)

No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.

So, it can put scanlines over the videos? That's what I actually meant, since Attract Mode doesn't modify the videos themselves either.

Just to be sure I understand you correctly: I would create a scanline image with transparent lines between the scanlines, and use that as a "frame" although its no actual frame?

muldjord

@Clyde Yes, you can do that for images, but not for videos unfortunately.

Clyde

@muldjord Damn, just when I saw a silver lining on the horizon.

Thanks for the clarification, though.

muldjord

@mitu Two users just asked on reddit why they got "File not found" when trying to run Skyscraper from command line after installing it with the setup script. Could we add a sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper line to the installation script somewhere so users can run it easily from command line?