RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      Panther7
      last edited by

      If you need any legal help (based on prior posts), feel free to PM me. I am a lawyer and would be willing to help you out.

      1 Reply Last reply Reply Quote 0
      • muldjordM
        muldjord @hi-ban
        last edited by

        @hi-ban Yes, please read the readme at github, it's all in there.

        @Panther7, Thank you, appeciate the sentiment. No need though, and there's no need for further elaboration on that point. I was simply asked to take out support for certain web sources and I complied promptly.

        The new version 2.0.0 only has support for modules I have acquired permission to use.

        At the end of the day, we're all here to support retro gaming. Some do it by creating game databases, some do it by creating emulator capable distributions and frontends. And some of us write tools to ease the flow and usage of game information to make it all a little bit more enjoyable.

        Skyscraper is free and open source and was developed in my spare time to help fellow retro'ers. This still holds true for the new version.

        H 1 Reply Last reply Reply Quote 1
        • H
          hi-ban @muldjord
          last edited by hi-ban

          @muldjord i already read the readme. It just shows how to load a config ini files, but it does not explain what the parameters from the config ini files exactly do.
          Also, there is no config or parameter for "dont-modify-image-at-all".
          And if you dont load any config at all, then the scraper applies the "small boxart and big screenshot" parameters as default.
          I cannot seem to find any way to make the scraper just take the raw image file without autto-editing it or processing it at all.

          By the way, the screenscraper.fr scraper is grabbing the wrong boxart images. Instead of grabbing the "Case: front" images, it's grabbing the "Case: texture" images instead.

          muldjordM 1 Reply Last reply Reply Quote 0
          • P
            Panther7
            last edited by

            Had a chance to toy with it. Seems really powerful, but it could REALLY use some sort of interface. It would do it wonders for usability.

            muldjordM 1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @hi-ban
              last edited by

              @hi-ban The screenscraper has been fixed in the version I am working on. The boxart image will always be modified in some way to resize it to fit EmulationStation or whatever frontend you are using. But you can set the size of it using the ini parameters, just do like this:
              finalImageWidth="281"
              finalImageHeight="375"
              coverWidth="281"
              coverHeight="375"
              coverX="0"
              coverY="0"
              coverShadowEnabled="false"
              screenshotEnabled="false"

              Set the widths and heights to whatever you want the cover size to be. There's no way of telling Skyscraper to not manipulate it somehow. But by using the above config, you can sortof trick it into doing so.

              1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord @Panther7
                last edited by

                @Panther7 I completely agree. I'd like to create a gui for it, but it won't be anytime soon.

                1 Reply Last reply Reply Quote 0
                • H
                  hi-ban
                  last edited by

                  @muldjord thanks for the tips.

                  It would be great if your scraper had the option to choose from different "possible" matches when the scraper is unable to find the correct match for the rom name, much like the default scraper from emulationstation does.

                  1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord @incunabula
                    last edited by

                    This post is deleted!
                    1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord
                      last edited by

                      Skyscraper 2.0.1 released: https://github.com/muldjord/skyscraper

                      • Slightly changed help text for scraping modules
                      • 'thegamesdb' now properly uses Qt's XML parser
                      • 'screenscraper' now properly uses Qt's XML parser
                      • Started implementing region and lang support for 'screenscraper', but still not enabled

                      Not the most interesting release from a user perspective. I am basically rewriting some stuff to make it work more standardized. But hey, it's still quite important work, so here it is! :D

                      screechS 1 Reply Last reply Reply Quote 0
                      • screechS
                        screech @muldjord
                        last edited by

                        Hi @muldjord,

                        I'm thinking about something. not sure I tell it to you :

                        Do you know there is an API V2 on Screenscraper ?
                        It's not "the official" one for now, but soon ;)
                        It work exactly the same as the API V1, but the XML generated is much much easier to use ;)

                        to use it simply put a "2" just after "www.screenscraper.fr/api" in the URL like this :

                        https://www.screenscraper.fr/api2/jeuInfos.php?devid=xxx&devpassword=yyy&softname=zzz&output=xml&ssid=test&sspassword=test&crc=50ABC90A&systemeid=1&romtype=rom&romnom=Sonic The Hedgehog 2 (World).zip&romtaille=749652

                        not so much documentation, but if you are working on the country/language data, you must see this ;) maybe it will be easier for you to use the new API ;)

                        muldjordM 1 Reply Last reply Reply Quote 0
                        • muldjordM
                          muldjord @screech
                          last edited by muldjord

                          @screech Haha, interesting! No, I didn't know it was online anywhere. I knew they were working on it, but not that it was already available. Seems like a bunch of the stuff I had gripes about with v1 has been fixed. Really cool!

                          Thanks for the heads-up! Do you have any idea how stable the format is? I mean, would it make sense to implement it now, or should I wait until xml format freeze?

                          EDIT: Especially I like that they now make use of attributes properly. It makes much more sense than before.

                          1 Reply Last reply Reply Quote 0
                          • screechS
                            screech
                            last edited by

                            Normally it's pretty stable. In the worst case more metada/média will be added. But on the same format. And Things already in it won't change ;)

                            1 Reply Last reply Reply Quote 1
                            • BladeHunterB
                              BladeHunter
                              last edited by

                              @screech, I just noticed that V2 doesn't list all the roms in a game:

                              https://www.screenscraper.fr/api/jeuInfos.php?devid=<removed>&devpassword=<removed>&devsoftname=RMS01output=xml&ssid=<removed>&sspassword=<removed>&md5=45e12558e1b9eda1b66b86f68f995f2a&sha1=533c497761cc802624708108bc9ef6e7c4f40296

                              Look at the bottom, there are 3 roms listed, 2 zip and 1 j64

                              https://www.screenscraper.fr/api2/jeuInfos.php?devid=<removed>&devpassword=<removed>&devsoftname=RMS01output=xml&ssid=<removed>&sspassword=<removed>&md5=45e12558e1b9eda1b66b86f68f995f2a&sha1=533c497761cc802624708108bc9ef6e7c4f40296

                              Only the j64 is listed.

                              Whilst this is the ROM I have specified in the url from the MD5 and SHA-1 shouldn't it still list the whole collection because it's the gameinfo part of the API not a ROMinfo part.

                              1 Reply Last reply Reply Quote 0
                              • AnalogHeroA
                                AnalogHero
                                last edited by AnalogHero

                                Soooooo...... i fried my 2 month old 64 gb sd card (stupid), which gave me the "opportunity" to rescrape my systems with the latest version.

                                And i must say that the old versions were better due to more scrape-sources. I know what the issues were, but i wish that i had a backup of either the old Skyscraper or the localdb.

                                Other than that it works flawless. :)

                                One thing what could be added is that if it doesnt find a match in the current module, try another one, as an option.

                                Edit: Scraping psx with screenscraper aborts with std::bad_alloc

                                1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord
                                  last edited by muldjord

                                  Skyscraper 2.0.2 released: https://github.com/muldjord/skyscraper

                                  • Updated 'arcadedb' result parsing to fit new format
                                  • Now scrapes 'msx' platform families correctly with the 'screenscraper' module
                                  • Changed limit for iso checksumming to 20 megs to avoid running out of memory.

                                  Mostly minor stuff and a bug fix. :)

                                  1 Reply Last reply Reply Quote 1
                                  • muldjordM
                                    muldjord
                                    last edited by muldjord

                                    @AnalogHero, The crash comes from running out of memory when trying to do a sha1 checksum of the iso files. I've changed the limit in 2.0.2 so please try that and see if it works. Either way, the psx platform won't work with the screenscraper module, simply because the raspberry pi doesn't have enough memory to checksum the iso files. So you'll have to use thegamesdb for those. :)

                                    Concerning trying another module while scraping if it doesn't find one with the current one: This won't be implemented, simply because it's already possible by scraping with another scraper afterwards. Just choose to "skip existing". :) And always remember to finish off by scraping with 'localdb' to get the best result from all cached data. :) Have fun!

                                    mituM 1 Reply Last reply Reply Quote 0
                                    • muldjordM
                                      muldjord
                                      last edited by

                                      Skyscraper 2.0.3 released: https://github.com/muldjord/skyscraper

                                      • Added '--lang' and '--region' command line options. Defaults to 'us' and 'wor' if none is provided.
                                      • Language and region support now implemented for 'screenscraper' module.
                                      • Removed 'stats' option as it didn't make much sense anymore.

                                      You can now set language and region which will be used currently by the 'screenscraper' module only. In a later release I will implement the option to get a list of supported languages and regions. But for now I just wanted to enable it. It defaults to language 'us' and region 'wor' if none is provided.

                                      1 Reply Last reply Reply Quote 0
                                      • mituM
                                        mitu Global Moderator @muldjord
                                        last edited by

                                        @muldjord said in Versatile C++ game scraper: Skyscraper:

                                        Either way, the psx platform won't work with the screenscraper module, simply because the raspberry pi doesn't have enough memory to checksum the iso files.

                                        That's not necessarily true - the md5sum standard utility can operate in a constant memory space, no matter how large is the input stream/file. The checksumming does not require the whole file contents to be in memory, MD5 requires only reading sequentially 512 bits from the input.

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @mitu
                                          last edited by

                                          @mitu Thanks for pointing this out. I actually started doing sha1 and md5 summing differently in 2.0.2. Instead of feeding the functions the entire file data I feed it bits of 1 Meg until it's done. So that would actually mean that it has already been fixed. :)

                                          1 Reply Last reply Reply Quote 0
                                          • AnalogHeroA
                                            AnalogHero
                                            last edited by AnalogHero

                                            @muldjord Works with psx now. A little slow, but no problem if you dont have a big psx collection like me.

                                            Thanks for the update. :)

                                            muldjordM 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.