RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • muldjordM
      muldjord @ecto
      last edited by

      @ecto said in Versatile C++ game scraper: Skyscraper:

      Just so I get it right: The hash calculation is not used for .sh and .bat files. Are there other file types without hash calculation?

      The following is the code that shows the file extensions that are calculated from the filename instead of the contents:

        if(info.suffix() == "uae" || info.suffix() == "cue" ||
           info.suffix() == "sh" || info.suffix() == "svm" ||
           info.suffix() == "scummvm" || info.suffix() == "mds" ||
           info.suffix() == "zip" || info.suffix() == "7z" ||
           info.suffix() == "gdi" || info.suffix() == "ml" ||
           info.suffix() == "bat" || info.suffix() == "init") {
      

      Anyways, I love scraping using Skyscraper and I "finished" my pc/dosbox games yesterday with the little workaround to change the sha1 values by adding a comment line in the .init file and then rescrape. Thanks again for your help!

      Then I will remove *.init from the above code again. Otherwise I'll break your cache in the next version.

      ectoE 1 Reply Last reply Reply Quote 1
      • ectoE
        ecto @muldjord
        last edited by

        @muldjord Yes, of course, that makes sense! Thanks again!

        1 Reply Last reply Reply Quote 1
        • muldjordM
          muldjord
          last edited by

          Skyscraper 3.3.4 released: https://github.com/muldjord/skyscraper

          • 'igdb' module global key has been removed from Skyscraper. They have moved to free personal keys instead (this is good news!). To get one, register at https://api.igdb.com (Thank you to 'stlbluesfan' for letting me know)
          • Fixed disk space check so it also works for 'attractmode' frontend
          • Updated 'attractmode' to save overviews in correct location and format (Thanks to 'barolo' for letting me know, thanks to 'teeedubb' and 'barolo' for testing)
          • For 'attractmode' added semicolon detection for 'snaps'/'video' parsing for determining resource folders (Thanks to 'barolo' for pointing this out)

          This is mostly a release to fix some AttractMode related export issues. But it also includes the change in the IGDB module that now craves a personal API key instead of the limited global one. This means that you can now get your own private key for IGDB and use that instead, giving everyone better access.

          ClydeC o0alucard0oO 2 Replies Last reply Reply Quote 2
          • ClydeC
            Clyde @muldjord
            last edited by

            @muldjord Great news! How do I store my personal IGDB key in the config files?

            https://github.com/muldjord/skyscraper/blob/master/docs/SCRAPINGMODULES.md#IGDB isn't updated yet in this matter.

            muldjordM 1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @Clyde
              last edited by

              @Clyde said in Versatile C++ game scraper: Skyscraper:

              @muldjord Great news! How do I store my personal IGDB key in the config files?

              Just like any of the other modules that need user credentials. In this case it's just a single key like this:

              [igdb]
              userCreds="<KEY>"
              

              I will update the doc, thanks!

              1 Reply Last reply Reply Quote 1
              • muldjordM
                muldjord
                last edited by muldjord

                Skyscraper 3.3.5 released: https://github.com/muldjord/skyscraper

                • Updated 'screenscraper' API to support new system name json format (Thank you to 'Micket' for letting me know)

                This is an important release! The 'screenscraper' API has changed a tiny bit, which broke the Skyscraper implementation. So this release is needed to make it work again.

                Please spread the word if you hear of anyone having 'screenscraper' issues with a pre-3.3.5 version. This version will fix it for them!

                And thanks to 'Micket' for letting me know. I don't use Skyscraper on a daily basis, so I rely on users to report such API issues to me.

                ClydeC 1 Reply Last reply Reply Quote 3
                • ClydeC
                  Clyde @muldjord
                  last edited by

                  @muldjord Thank you for your ongoing work.

                  One (seemingly) small feature request. 😌 Next to the --symlink option, a --hardlink option would be great. Hardlinks would combine the benefits of saved storage space like symlinks, but would also work like normal files if the roms and/or media directories are copied to another medium. Would that be feasible for a coming update?

                  muldjordM 1 Reply Last reply Reply Quote 0
                  • mituM
                    mitu Global Moderator
                    last edited by

                    Actually, using a hardlink has a few limitations and caveats:

                    • a hardlink cannot be created across filesystems - it's only between files on the same FS.
                    • if you remove the source, then re-create it (with the same name), the hardlink would still contain the 1st source data, while the symlink would still work.
                    1 Reply Last reply Reply Quote 1
                    • muldjordM
                      muldjord @Clyde
                      last edited by

                      @Clyde The framework I use does not support creating hardlinks only symlinks.

                      ClydeC 1 Reply Last reply Reply Quote 0
                      • ClydeC
                        Clyde @muldjord
                        last edited by

                        @mitu Yes, both variants have their pitfalls (like symlinks' dependence on the source). I just would have liked to have both as options. :)

                        @muldjord A pity, but thanks for clearing it up.

                        1 Reply Last reply Reply Quote 0
                        • S
                          steptoe
                          last edited by

                          Hi,

                          Just wondering if you have any plans to add support for pinball in your scraper

                          I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

                          Mainly Future Pinball and Visual Pinball/PinMAME

                          Thanks

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • ClydeC
                            Clyde
                            last edited by

                            @muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

                            muldjordM 1 Reply Last reply Reply Quote 0
                            • S
                              steptoe
                              last edited by

                              I'm trying to compiler the latest Skyscraper 3.3.5 for Windows using QT as described here :

                              *You can easily cross-compile it for Windows. Just download and install Qt 5 from the official site (open source version). When installing, put a checkmark in the latest mingw32 version, I don't support MSVC++ although it might work, not sure. When it's installed you'll find a Qt specific command prompt in your start menu. From there navigate to the Skyscraper source folder and run:
                              $ qmake
                              $ mingw32-make release

                              That will give you a compiled version in a subfolder called "release". Then you need to copy the needed dll's from the qt5 install folder into the release folder and start an ordinary command prompt there. Then you should be able to run it.*

                              Running the QT Creator and trying various new projects and not finding how to point at the Skyscraper source folder to compile a windows version

                              My intention is then look into using other 3rd party software to create a simple GUI for us Windows users. I use SkraperUI but Skyscraper gets better and better as I've followed its development for RetroPie which I also use via RetroSMC on our Pi/KODI box running OSMC

                              Thanks

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @steptoe
                                last edited by

                                @steptoe said in Versatile C++ game scraper: Skyscraper:

                                Just wondering if you have any plans to add support for pinball in your scraper

                                No current plans, no.

                                I'm aware not many sites have pinball data on their videos or images. emumovies seems to be about the only one that has some videos/images

                                Skyscraper doesn't support emumovies as they don't have an open API. At least not to my knowledge. I've been looking for one but never found one. It has to be some kind of REST-based API, not just an ftp with images or something like that.

                                1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @Clyde
                                  last edited by

                                  @Clyde said in Versatile C++ game scraper: Skyscraper:

                                  @muldjord Happy holidays! I recently noticed a wrongly scraped game. Would it help if I reported this to you? If so, how should I do it?

                                  That should be reported to the module that delivered the wrong data. I suggest registering with them and fixing it yourself if you can. That's the power of community based game info databases.

                                  ClydeC 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @steptoe
                                    last edited by

                                    @steptoe said in Versatile C++ game scraper: Skyscraper:

                                    I'm trying to compiler the latest Skyscraper 3.3.5 for Windows

                                    The howto describes how to compile it under Windows. You shouldn't use QT Creator to do that, just the Qt command prompt as the text you pasted also tells you to. But you're basically on your own here. I don't use Windows myself and quite frankly don't have much interest in that system. Sorry.

                                    Remember that Skyscraper is licensed under the GPL if you end up using any of the Skyscraper code for any of your projects. :) Good luck with it!

                                    1 Reply Last reply Reply Quote 0
                                    • ClydeC
                                      Clyde @muldjord
                                      last edited by Clyde

                                      @muldjord said in Versatile C++ game scraper: Skyscraper:

                                      That should be reported to the module that delivered the wrong data. I suggest registering with them and fixing it yourself if you can. That's the power of community based game info databases.

                                      Ah, I thought that Skyscraper mistook one game for another (but on the second thought, only getting the video wrong then is illogical). I didn't think about checking the data on screenscraper.fr, though. And there it is: the wrong video (from Dangerous Xmas instead of Zettaizetsumei Dangerous Jiisan). I'll report it immediately, I have already an account there. edit: Done.

                                      Thanks for pointing me to something obvious, I plead insanity because of a heavy cold I'm having at the moment. 🥴

                                      1 Reply Last reply Reply Quote 1
                                      • ClydeC
                                        Clyde
                                        last edited by Clyde

                                        FYI: TheGamesDB is back online, but it needs a prefix /v1/ to work now.

                                        (Thanks to @maserspark for reporting this.)

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @Clyde
                                          last edited by muldjord

                                          @Clyde Thanks for the update. It seems they are running from an old backup, but I don't quite understand what they mean by

                                          the API is also back online,
                                          but using the prefix /v1/ which is both to get devs to explicitly make sure old data doesn't just crash their apps,
                                          but this prefix will remain as our main endpoint for this api going forward

                                          Sounds to me like it's only the old API that's online but it doesn't say so specifically.

                                          Either way, great to see that they are slowly getting back online.

                                          ClydeC 1 Reply Last reply Reply Quote 0
                                          • ClydeC
                                            Clyde @muldjord
                                            last edited by

                                            @muldjord Indeed, it is great. Do you have to update SS to include the prefix?

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.