RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Skyscraper now officially part of RetroPie, please test

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscraperretropie-setupoptionalexperimental
    275 Posts 28 Posters 112.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robertvb83R
      robertvb83 @mitu
      last edited by

      @muldjord great great work. Even though i did not try it yet, I like your concept of local-db very much. this makes this scraper really stand out. Over time you can improve the local-db (even manually) and minimize traffic for the online dbs. If I don't like the art or need to change some little things there is no need to scrape a lot of data again and again, Genius! thank you!

      @mitu thanks for improving the wiki already. makes it easy to get startet!

      I have one suggestion. instead of everyone making there own artwork.xml adaptions it would be nice to have some predefined options to chose from. e.g. 3D cover boxart or mixed with sreenshot, marquee, box.
      would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

      My full size arcade cabinet Robotron vs. Octolyzer

      muldjordM 1 Reply Last reply Reply Quote 1
      • muldjordM
        muldjord @mitu
        last edited by

        @mitu Looks good, here are my comments:

        Skyscraper by Lars Muldjord is a powerful and versatile yet easy to use game scraper written in C++ for use with multiple frontends running on a Linux system. It scrapes and caches various game resources from various web sources, including media such as screenshot, cover and video. It then gives you the option to combine all of these resources into the most complete results by using the provided 'localdb' scraping module.

        Could you add this line beneath this: "For a more thorough description of any functionality described in this howto, please check out the official Skyscraper documentation here: https://github.com/muldjord/skyscraper

        NOTE: Skyscraper can be used to generate scraping information for both [FIXME:Emulationstation][Emulationstation] and [FixMelAttractMode][AttractMode]. Installing Skyscraper as a RetroPie package will automatically configure it for Emulationstation, but if you wish to use it for generating AttachMode metadata, you can use Skyscraper from the command line.

        "AttachMode" should be "AttractMode".

        Can you also add a section just below this with: "Recommended usage is to scrape your platform(s) with as many online sources as you'd like. All of the data will be cached while doing so. Then, when you have gathered enough data, be sure to always scrape it one last time using the 'Local: LocalDb' source. This will combine all of your cached data into the most complete results for each rom."

        Remove bracket info: Choose to remove (Enabled) or keep (Disabled) the text between brackets when using the Filename option for ROM Names. If the name of the ROM is Super Mario World (USA)[!].sfc, enabling this option will make the ROM show as Super Mario World in Emulationstation.

        This will actually also happen when using the source titles. It strips the bracket info from the filename and adds it back to the frontend name no matter which one you've chosen to use.

        Force cache refresh: Skyscraper caches the media retrieved from online sources locally to speed up scraping and to combine multiple sources of information to give you the best scraped information available. This option forces Skyscraper to bypass the locally cached resources and re-download the media already cached from the internet.

        Can you change "...bypass the locally cached resources and re-download the media already cached from the internet." to "...bypass the locally cached resources and re-download the media from the online sources."

        Download Videos: Choose to toggle the download and caching of videos for the ROM names. This also enables adding the video to the gamelist in Emulationstation.

        Could you add "NOTE: Be aware that fetching and caching videos takes up a lot of disk space."

        Lastly, I think for artwork, just point users to my official docs here: https://github.com/muldjord/skyscraper/blob/master/ARTWORK.md

        I've spent a lot of time on that including image examples and even a video (more to come when I get the time).

        Same thing about the import module. The doc for this is here: https://github.com/muldjord/skyscraper/blob/master/import/README.md

        And lastly, the doc for localdb: https://github.com/muldjord/skyscraper/blob/master/dbs/README.md

        I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

        mituM 1 Reply Last reply Reply Quote 0
        • muldjordM
          muldjord @robertvb83
          last edited by

          @robertvb83 That is actually a good idea with the different artwork.xml configs. I think we'll let this settle first though and see how users react to it.

          1 Reply Last reply Reply Quote 0
          • mituM
            mitu Global Moderator @muldjord
            last edited by

            @muldjord said in Skyscraper now officially part of RetroPie, please test:

            I think for those 3 parts (artwork, import, localdb), just link to my docs. It's great to have the RetroPie specific things documented in the RetroPie wiki and the overall explanations of course. But I feel like we should stick with my updated docs for the rest. Agree?

            Sure thing. For the Artwork/Import my idea was just to say a few words and point to the official docs anyway, there's too much to cover and it's just silly to duplicate things instead of using the source.

            @robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

            would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

            Is this a template in XML scraper (I assume you're referring to UXS) ?

            robertvb83R 1 Reply Last reply Reply Quote 1
            • mituM
              mitu Global Moderator
              last edited by mitu

              @muldjord I've modified the WIP wiki page (link here) and added your suggestions. For the advanced part I've added your descriptions for Import and Artwork and linked to the official docs. I've removed the CLI usage since this is going to evolve and will probably be outdated.

              muldjordM 1 Reply Last reply Reply Quote 2
              • muldjordM
                muldjord @mitu
                last edited by

                @mitu Awesome! Love the artwork example, looks good. I just went over it all real quick, I don't have further comments, looks good to me.

                mituM 1 Reply Last reply Reply Quote 0
                • mituM
                  mitu Global Moderator @muldjord
                  last edited by

                  @muldjord said in Skyscraper now officially part of RetroPie, please test:

                  Love the artwork example, looks good

                  I couldn't resist putting it there, I initially thought of adding a table with each example artwork and it's preview, but I think that one table is suggestive enough to grasp the 'magic' behind the artwork compositor.

                  muldjordM 1 Reply Last reply Reply Quote 1
                  • muldjordM
                    muldjord @mitu
                    last edited by

                    @mitu Agreed, it's nice to have it there as a teaser for what people can do and it works well to have the individual pieces first, and then the result. I think that explains how it works well. We can add more examples if we create a bunch of default artwork.xml files for people to use.

                    1 Reply Last reply Reply Quote 0
                    • mituM
                      mitu Global Moderator
                      last edited by

                      Page is now live at https://retropie.org.uk/docs/Scraper

                      1 Reply Last reply Reply Quote 1
                      • muldjordM
                        muldjord
                        last edited by

                        Posted on reddit now: https://www.reddit.com/r/RetroPie/comments/9p1zqq/skyscraper_now_officially_part_of_retropie/

                        1 Reply Last reply Reply Quote 1
                        • robertvb83R
                          robertvb83 @mitu
                          last edited by robertvb83

                          @mitu said in Skyscraper now officially part of RetroPie, please test:

                          @robertvb83 Thanks for the ideas - will probably add it as an advanced option in one of the next iterations. Can you elaborate a bit on the:

                          would it be possible to even do something like xml-scraper with the arcade cabinet with marquee and screenshot?

                          Is this a template in XML scraper (I assume you're referring to UXS) ?

                          Yes i am referring to UXS. They actually have a similar art output mixer but they also have the option to show a little arcade cabinet with screenshot and marquee as you can see here (scroll down a little)
                          0_1539810210734_sf2ce-image.png

                          My full size arcade cabinet Robotron vs. Octolyzer

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord @robertvb83
                            last edited by

                            @robertvb83 This can actually be achieved relatively easily using the Skyscraper compositor. It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet. I will consider creating an "arcade cabinet" artwork.xml and we can add that on a list of artwork templates. But I need a completely royalty free arcade cabinet overlay to do that. Creating one from scratch is a bit much for my artistic skills.

                            ClydeC 1 Reply Last reply Reply Quote 0
                            • ClydeC
                              Clyde @muldjord
                              last edited by

                              @muldjord said in Skyscraper now officially part of RetroPie, please test:

                              It has the option to do masks and frame overlays and can rotate the resources to fit the different sections of the cabinet.

                              So it can put scanlines in the videos? That would be great, since ES and the OMX Player can't do that on the fly like Attract Mode can, for example.

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @Clyde
                                last edited by muldjord

                                @Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

                                No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.

                                mituM ClydeC 2 Replies Last reply Reply Quote 0
                                • mituM
                                  mitu Global Moderator @muldjord
                                  last edited by

                                  @muldjord

                                  So it can put scanlines in the videos?

                                  Does it work for videos ?

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @mitu
                                    last edited by

                                    @mitu Sorry, no, I was just editing my post while you were posting. :D No, it can't process videos.

                                    1 Reply Last reply Reply Quote 0
                                    • ClydeC
                                      Clyde @muldjord
                                      last edited by

                                      @muldjord said in Skyscraper now officially part of RetroPie, please test:

                                      @Clyde OMG IT'S CLYDE FROM PAC-MAN! I'm a big fan :D

                                      :)

                                      No, it can't add effects to videos unfortunately, but you could add a scanline effect to the graphics using a frame effect with transparency applied to a layer.

                                      So, it can put scanlines over the videos? That's what I actually meant, since Attract Mode doesn't modify the videos themselves either.

                                      Just to be sure I understand you correctly: I would create a scanline image with transparent lines between the scanlines, and use that as a "frame" although its no actual frame?

                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @Clyde
                                        last edited by

                                        @Clyde Yes, you can do that for images, but not for videos unfortunately.

                                        ClydeC 1 Reply Last reply Reply Quote 0
                                        • ClydeC
                                          Clyde @muldjord
                                          last edited by

                                          @muldjord Damn, just when I saw a silver lining on the horizon.

                                          Thanks for the clarification, though.

                                          1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord
                                            last edited by muldjord

                                            @mitu Two users just asked on reddit why they got "File not found" when trying to run Skyscraper from command line after installing it with the setup script. Could we add a sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper line to the installation script somewhere so users can run it easily from command line?

                                            mituM 1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.