Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

Skyscraper now officially part of RetroPie, please test



  • UPDATED 28th of Nov 2018: This has been requested many, many times and it has finally happened! Skyscraper can now be installed and used directly from the RetroPie-Setup menu. Find it under "Experimental" packages (It was under "Optional" previously. Will probably be moved to "Optional" when we regard it as stable).

    A few disclaimers for current Skyscraper users:

    1. You don't need to install Skyscraper through the Retropie-Setup if you don't want to. It will continue to work just fine using the github readme installation method. This new method is a convenience for RetroPie users who don't want the joy of working on the command line (that was not sarcasm). We do strongly recommend choosing one method or the other and stick with that. Don't mix it! The executables for the two versions are located in different places, this could potentially create compatibility issues if you remember to update one and not the other.
    2. Remember to quit Emulationstation before using Skyscraper through the RetroPie-Setup script. You can install it while Emulationstation is running though. Quit EmulationStation by pressing F4, then run "sudo ~/Retropie-Setup/retropie-setup.sh" to run the setup.
    3. When installing Skyscraper through the setup script it will move the entire '~/.skyscraper' folder to the '/opt/retropie/configs/all/skyscraper' folder and symlink it back to '~/.skyscraper'. So basically it works just as it does now, but via a symlink instead. I recommend making a backup of your current '~/.skyscraper' folder before installing Skyscraper through the RetroPie setup script. If you mess this up, don't come crying to us, you have been warned! :)

    Basic usage suggestion

    The script is best used in the following way:

    1. Select one or more platforms you wish to scrape
    2. Select the online source you wish to scrape them / it with (be aware that some sources need further setup in config.ini with key or user:id)
    3. Toggle / set any options you want enabled / disabled
    4. Scrape the platform(s)
    5. Repeat this with further online sources until you feel like you have enough data for the platform(s). Each time you scrape with a new source, the data is cached locally.
    6. When you are ready, choose the "Local: LocalDb" source and rescrape the platform(s) one last time. 'LocalDb' will scrape from ALL of your cached data and combine them into the most complete results.
    7. You're done! Fire up EmulationStation and enjoy the results.

    Advanced usage suggestion

    • You can edit the config.ini file through the advanced menu and add any sections and attributes you want. These will then be used when running Skyscraper. This is needed for the more advanced features such as setting key or user:id for some of the online sources.
    • You can also edit the artwork.xml file through the advanced menu to set up the artwork compositor. Check the full documentation for the format here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

    For all features, please refer to the github readme for details. Find all of it here: https://github.com/muldjord/skyscraper

    A huge thanks to @mitu for creating the RetroPie setup script. And thank you to @BuZz for reviewing the script. If you have any issues, please let us know so we can look into it. Please check it out if you feel like it, and feedback is of course more than welcome!

    Happy scraping!



  • Just to add to @muldjord's announcement

    • the new skyscraper module is added in the Optional section of the packages. Make sure you first update your RetroPie-Setup script to get the new module.
    • the Skyscraper usage from the RetroPie-Setup has a gui (similar to Steven Selph's scraper) that can be accessed either from the Configuration/Tools menu of the RetroPie-Setup or using the Configuration/Options menu of the package.

    0_1539341753292_2d4d68da-bea6-4f9c-a44f-bee87d3d5ecd-image.png

    More information will be added in the Wiki to explain better how to use Skyscraper from the RetroPie-Setup's interface.



  • @muldjord Finally done ;) This is really a nice piece of software - valueable use!
    @mitu Thank you for writing the script module and the bash script so we have a text inputs and less work with CLI.



  • @cyperghost said in Skyscraper now officially part of RetroPie, please test:

    @mitu Thank you for writing the script module and the bash script so we have a text inputs ;)

    What ? There are no input boxes in the scriptmodule :)



  • @mitu Text Input via menu selections ;)



  • @muldjord - I have a WIP draft for the Wiki page at Scraping. I've re-organized the page and added sections for each scraper, keeping the old information and adding a section for Skyscraper.
    I've not added info for the Artwork and Import module and I have to re-organize the section about the location of metadata/media files on the system (Markdown is quite limited when it comes to tables). I think you're changing some options for the Import module, so we should incorporate that.
    Feel free to add any commends or suggest modifications.



  • @muldjord great great work. Even though i did not try it yet, I like your concept of local-db very much. this makes this scraper really stand out. Over time you can improve the local-db (even manually) and minimize traffic for the online dbs. If I don't like the art or need to change some little things there is no need to scrape a lot of data again and again, Genius! thank you!

    @mitu thanks for improving the wiki already. makes it easy to get startet!

    I have one suggestion. instead of everyone making there own artwork.xml adaptions it would be nice to have some predefined options to chose from. e.g. 3D cover boxart or mixed with sreenshot, marquee, box.
    would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?



  • @mitu Looks good, here are my comments:

    Skyscraper by Lars Muldjord is a powerful and versatile yet easy to use game scraper written in C++ for use with multiple frontends running on a Linux system. It scrapes and caches various game resources from various web sources, including media such as screenshot, cover and video. It then gives you the option to combine all of these resources into the most complete results by using the provided 'localdb' scraping module.

    Could you add this line beneath this: "For a more thorough description of any functionality described in this howto, please check out the official Skyscraper documentation here: https://github.com/muldjord/skyscraper

    NOTE: Skyscraper can be used to generate scraping information for both [FIXME:Emulationstation][Emulationstation] and [FixMelAttractMode][AttractMode]. Installing Skyscraper as a RetroPie package will automatically configure it for Emulationstation, but if you wish to use it for generating AttachMode metadata, you can use Skyscraper from the command line.

    "AttachMode" should be "AttractMode".

    Can you also add a section just below this with: "Recommended usage is to scrape your platform(s) with as many online sources as you'd like. All of the data will be cached while doing so. Then, when you have gathered enough data, be sure to always scrape it one last time using the 'Local: LocalDb' source. This will combine all of your cached data into the most complete results for each rom."

    Remove bracket info: Choose to remove (Enabled) or keep (Disabled) the text between brackets when using the Filename option for ROM Names. If the name of the ROM is Super Mario World (USA)[!].sfc, enabling this option will make the ROM show as Super Mario World in Emulationstation.

    This will actually also happen when using the source titles. It strips the bracket info from the filename and adds it back to the frontend name no matter which one you've chosen to use.

    Force cache refresh: Skyscraper caches the media retrieved from online sources locally to speed up scraping and to combine multiple sources of information to give you the best scraped information available. This option forces Skyscraper to bypass the locally cached resources and re-download the media already cached from the internet.

    Can you change "...bypass the locally cached resources and re-download the media already cached from the internet." to "...bypass the locally cached resources and re-download the media from the online sources."

    Download Videos: Choose to toggle the download and caching of videos for the ROM names. This also enables adding the video to the gamelist in Emulationstation.

    Could you add "NOTE: Be aware that fetching and caching videos takes up a lot of disk space."

    Lastly, I think for artwork, just point users to my official docs here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

    I've spent a lot of time on that including image examples and even a video (more to come when I get the time).

    Same thing about the import module. The doc for this is here: https://github.com/muldjord/skyscraper/blob/master/import/README.md

    And lastly, the doc for localdb: https://github.com/muldjord/skyscraper/blob/master/dbs/README.md

    I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?



  • @robertvb83 That is actually a good idea with the different artwork.xml configs. I think we'll let this settle first though and see how users react to it.



  • @muldjord said in Skyscraper now officially part of RetroPie, please test:

    I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

    Sure thing. For the Artwork/Import my idea was just to say a few words and point to the official docs anyway, there's too much to cover and it's just silly to duplicate things instead of using the source.

    @robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

    would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

    Is this a template in XML scraper (I assume you're referring to UXS) ?



  • @muldjord I've modified the WIP wiki page (link here) and added your suggestions. For the advanced part I've added your descriptions for Import and Artwork and linked to the official docs. I've removed the CLI usage since this is going to evolve and will probably be outdated.



  • @mitu Awesome! Love the artwork example, looks good. I just went over it all real quick, I don't have further comments, looks good to me.



  • @muldjord said in Skyscraper now officially part of RetroPie, please test:

    Love the artwork example, looks good

    I couldn't resist putting it there, I initially thought of adding a table with each example artwork and it's preview, but I think that one table is suggestive enough to grasp the 'magic' behind the artwork compositor.



  • @mitu Agreed, it's nice to have it there as a teaser for what people can do and it works well to have the individual pieces first, and then the result. I think that explains how it works well. We can add more examples if we create a bunch of default artwork.xml files for people to use.







  • @mitu said in Skyscraper now officially part of RetroPie, please test:

    @robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

    would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

    Is this a template in XML scraper (I assume you're referring to UXS) ?

    Yes i am referring to UXS. They actually have a similar art output mixer but they also have the option to show a little arcade cabinet with screenshot and marquee as you can see here (scroll down a little)
    0_1539810210734_sf2ce-image.png



  • @robertvb83 This can actually be achieved relatively easily using the Skyscraper compositor. It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet. I will consider creating an "arcade cabinet" artwork.xml and we can add that on a list of artwork templates. But I need a completely royalty free arcade cabinet overlay to do that. Creating one from scratch is a bit much for my artistic skills.



  • @muldjord said in Skyscraper now officially part of RetroPie, please test:

    It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet.

    So it can put scanlines in the videos? That would be great, since ES and the OMX Player can't do that on the fly like Attract Mode can, for example.



  • @Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

    No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.



Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.