RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ByteThisB
      ByteThis @mitu
      last edited by ByteThis

      @mitu Yes..the error is "Could not install package(s): rapidjson.dev". I updated ES some time ago to install the Screenscraper scraper within Retropie and tried to re-update to include TheGAMESDB but got that error.

      mituM 1 Reply Last reply Reply Quote 0
      • mituM
        mitu Global Moderator @ByteThis
        last edited by

        @ByteThis I posted a workaround in the sticky topic with the ES scraper problems in the Help sections. If you could give it a try and see if it works, that would be great.

        ByteThisB 1 Reply Last reply Reply Quote 0
        • ByteThisB
          ByteThis @mitu
          last edited by

          @mitu Sure. I will try it. BRB.

          1 Reply Last reply Reply Quote 0
          • ByteThisB
            ByteThis
            last edited by

            @mitu The dependency couldn't be installed : https://pastebin.com/zknu3NVV

            mituM 1 Reply Last reply Reply Quote 0
            • mituM
              mitu Global Moderator @ByteThis
              last edited by

              @ByteThis The workaround I posted involves installing the rapidjson-dev package manually before updating

              wget http://ftp.us.debian.org/debian/pool/main/r/rapidjson/rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb
              sudo dpkg -i rapidjson-dev_1.1.0+dfsg2-3~bpo9+1_all.deb
              
              ByteThisB 1 Reply Last reply Reply Quote 0
              • ByteThisB
                ByteThis @mitu
                last edited by ByteThis

                @mitu That's the one I tried to install. I tried both WinSCP (through the terminal) and Putty.

                1 Reply Last reply Reply Quote 0
                • ByteThisB
                  ByteThis
                  last edited by ByteThis

                  @mitu I tried it again a couple of times and ES is building from source..stay tuned.

                  1 Reply Last reply Reply Quote 0
                  • ByteThisB
                    ByteThis
                    last edited by ByteThis

                    @mitu It works.Not sure what initially happened but the dependency installed and ES updated with both Screenscraper and THEGAMESDB are both there. Sorry for any confusion. Seems like some kind of hiccup installing rapidjson-dev.

                    mituM 1 Reply Last reply Reply Quote 0
                    • mituM
                      mitu Global Moderator @ByteThis
                      last edited by

                      @ByteThis You're supposed to execute those commands from a SSH session (Putty). Glad it works, thank you for the confirmation.

                      ByteThisB 1 Reply Last reply Reply Quote 1
                      • ByteThisB
                        ByteThis @mitu
                        last edited by

                        @mitu Yea..as I mentioned above I tried both Putty and the terminal in WinSCP. Thanks for your hard work ! Again apologies for the hiccup. Am glad to be of some help...lol.

                        1 Reply Last reply Reply Quote 1
                        • S
                          semakelh50
                          last edited by semakelh50

                          @muldjord The documentation contains a misspelling in SCRAPINGMODULES.md. In the OpenRetro section -> platform support, "aswell" should be "as well." It's two words, not one.

                          1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord
                            last edited by muldjord

                            Just a quick update: The screenscraper module is currently having issues with their internal server cache (not related to the Skyscraper cache, just to be clear) when you scrape without user credentials. It will sometimes (quite often) return faulty error messages even though Skyscraper uses the API correctly. I've contacted them and they verified that this is an issue. They've been working on fixing it for the past couple of days, but their cache (again, not the Skyscraper cache, their internal server cache) needs to catch up to the fixes. So hopefully this issue will resolve itself over the coming week.

                            1 Reply Last reply Reply Quote 0
                            • muldjordM
                              muldjord
                              last edited by muldjord

                              The culprit has been found for the screenscraper issues. This stems from their server CPU's usage being too high. When it reaches a certain limit, it starts rejecting requests from unregistered users (this is by design, it's not a bug).

                              3.0.5 has just been released which adds a note about this when the error occurs. It also fixes a bug in the metadata preservation code, so please update.

                              Once again, please consider supporting screenscraper when you go to register a user. And also consider supporting RetroPie while you are at it.

                              ByteThisB 1 Reply Last reply Reply Quote 0
                              • ByteThisB
                                ByteThis @muldjord
                                last edited by ByteThis

                                @muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

                                mituM muldjordM 2 Replies Last reply Reply Quote 0
                                • mituM
                                  mitu Global Moderator @ByteThis
                                  last edited by

                                  @ByteThis That thought crossed my mind, but that can only be answered by the ScreenScraper admins.

                                  1 Reply Last reply Reply Quote 1
                                  • muldjordM
                                    muldjord @ByteThis
                                    last edited by

                                    @ByteThis said in Versatile C++ game scraper: Skyscraper:

                                    @muldjord Is this due to ScreenScraper's database (servers) being overwhelmed by it's existence as an internal main scraper as well ?

                                    All I know is that this is only an issue for unregistered users. But in general it is slightly worrying that their servers are reaching these usage levels. Skyscraper does a lot to alleviate it by caching the data. But I'm not sure other scrapers do this.

                                    ByteThisB 1 Reply Last reply Reply Quote 1
                                    • ByteThisB
                                      ByteThis @muldjord
                                      last edited by ByteThis

                                      @muldjord They definitely have a lot on their plate.

                                      EDIT: This is posted on their site : This month: 4382 euros collected. / 265 euros for a dedicated server.

                                      Maybe they can afford another server ;-)

                                      1 Reply Last reply Reply Quote 1
                                      • chubstaC
                                        chubsta
                                        last edited by

                                        I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

                                        Raspberry Pi 400 - overclocked to 2.3GHz
                                        Theme - Comic book
                                        Emulators: All of them...
                                        Roms: Pretty much all of them...
                                        Favourite games: Circus Charlie, Gorf, Gauntlet Legends
                                        Controllers: Mayflash F101

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @chubsta
                                          last edited by

                                          @chubsta said in Versatile C++ game scraper: Skyscraper:

                                          I am currently trying to scrape all my systems and it is working well, but I cannot seem to scrape famicom or megadrive-japan. They are not listed in the supported systems but if I were to transfer the files over to my new and mega drive folders would it scrape them then?

                                          Famicom roms goes in "nes" folder and megadrive-japan roms go in "megadrive" og "genesis" folder. If the filenames have region indication in them such as "(japan)" then Skyscraper will autodetect the region and prioritize it when looking for the result.

                                          quicksilverQ chubstaC 2 Replies Last reply Reply Quote 1
                                          • quicksilverQ
                                            quicksilver @muldjord
                                            last edited by

                                            @muldjord Just started using skyscraper over sselphs scraper (great work btw! Looks fantastic!). Everything works easily enough but I have noticed that sometimes it will find a screenshot and wheel for example but not the box art even though when I look up the entry on screenscraper all info is there (box art, screenshot etc). For example, when it scrapes Spiderman - web of fire 32x it says it cant find the box art but it gets everything else.

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.