RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.8m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      easye9inches @muldjord
      last edited by easye9inches

      @muldjord Ahhhh. Thanks, it works flawlessly now!

      Question? I know you could have only one picture as in a screen shot, cover, etc. I found in DBS folder if i wanted to copy covers, marquees, etc over to the external as well, why are those pics named like 0b809d4d49d064bd95d84b2865cc0f9304750b9d (3d Tetris COVER)? Must you go about in renaming everyone, lol?

      muldjordM 1 Reply Last reply Reply Quote 1
      • muldjordM
        muldjord @easye9inches
        last edited by

        @easye9inches I've just released 2.7.4 earlier today. That allows you to add the 'inputFolder' variable to the [main] section in config.ini. Then it works for all platforms automatically.

        1 Reply Last reply Reply Quote 1
        • M
          Mick2K
          last edited by

          I have a problem. I can't update. Any ideas?!

          alt text

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @Mick2K
            last edited by

            @mick2k Please run the following and try again, I have been messing around with the 2.7.4 release because I found a silly bug, so I re-released.

            Run this:

            $ cd
            $ cd skysource
            $ rm VERSION
            $ ./update_skyscraper.sh
            

            That should fix it and update to 2.7.4. Let me know how it goes.

            M 1 Reply Last reply Reply Quote 0
            • M
              Mick2K @muldjord
              last edited by

              @muldjord said in Versatile C++ game scraper: Skyscraper:

              ./update_skyscraper.sh

              It worked. Thank you very much.

              1 Reply Last reply Reply Quote 1
              • E
                easye9inches
                last edited by easye9inches

                Is there a certain way to name a sub-folder so that it also could get scrapped into the gamelist.xml? For instance, I added a "#JP Games" folder to the genesis library instead of having both a megadrive and genesis system selection. Just to condense the system selection down. But it did not scrape that sub-folder. Is that possible?

                Edit: Nevermind. I see they were scraped and included in the gamelist.xml

                1 Reply Last reply Reply Quote 0
                • muldjordM
                  muldjord @AnalogHero
                  last edited by muldjord

                  @analoghero Hey man, I've implemented the checksum option you suggested some time back. It's currently on the master branch and will be in 2.7.5 to be released soon. :)

                  It works by using the command line option '--query' which basically takes either a search query for the filename based scraping modules, or either 'sha1=[checksum]', 'md5=[checksum]', or 'romnom=[filename]' (rom name in French). It also requires a single rom to be passed on command line like so:

                  $ Skyscraper -p [platform] -s screenscraper --refresh --query sha1=[checksum] /[path]/[to]/[romfile.zip]
                  

                  This will allow you to override the checksums used when searching for the game. So you can look one up at screenscraper, and just use that.

                  AnalogHeroA 1 Reply Last reply Reply Quote 1
                  • AnalogHeroA
                    AnalogHero @muldjord
                    last edited by

                    @muldjord Nice! Thanks for putting your time into this. Hipe that others could use this option, too.

                    muldjordM 1 Reply Last reply Reply Quote 1
                    • muldjordM
                      muldjord @AnalogHero
                      last edited by

                      @analoghero It's a niche feature for sure, but I got another request for it, and I figured out a way to implement it in a way I was satisfied with. :)

                      1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord
                        last edited by

                        Skyscraper version 2.7.5 released: https://github.com/muldjord/skyscraper

                        • Fixed a bug where 'brackets="false"' in config.ini would be flipped (Thanks to Vynce for reporting this)
                        • Completely refactored pass procedures for cleaner code and to enable '--query' option
                        • Added '--query' command line option. This option requires a single rom file to be passed on command line aswell, otherwise it will be ignored (Thank you to AnalogHero and Vynce for suggesting this)
                        • Added scrapers to 'psx' and 'pc' platforms when using Simple Mode

                        To elaborate on the "--query" option, this is how it works: For most modules a search query is sent to the scraping module in an URL format. That means that a filename such as "Rick Dangerous.lha" becomes "rick+dangerous". The '+' here means a space. You could probably also use the URL encoded space "rick%20dangerous" but my tests show that most modules expect spaces as '+'. And it is the "rick+dangerous" that you, as the user, can pass as the query, like so:

                        $ Skyscraper -p [platform] -s [module] --query "rick+dangerous" [filename]
                        

                        Remember to also add a filename that you wish to use the override with. Otherwise the query will be ignored.

                        But, not all of the scraping modules are search name based. For instance, the "screenscraper" module can use a variety of different search methods. So for screenscraper, you also have the option of overriding the checksums it use to search for a game. This is especially convenient in cases where a filename exists multiple times in their database and your own local file doesn't match with any of the connected checksums (maybe you've compressed the rom yourself or whatever).
                        In this case you can look up one of the working checksums on "screenscrapers" website (screenscraper.fr) and override the checksum like these examples:

                        $ Skyscraper -p [platform] -s [module] --query sha1=[checksum] [filename]
                        $ Skyscraper -p [platform] -s [module] --query md5=[checksum] [filename]
                        $ Skyscraper -p [platform] -s [module] --query sha1=[checksum]&md5=[checksum]&romnom=[exact url encoded filename] [filename]
                        

                        The last example combines two of the checksum options and even the "romnom" option which is "rom name" in French (this is a screenscraper thing, not a Skyscraper thing). You obviously only need one of the checksum options, it's just to show that you can combine them if you really need to.

                        The '--query' option is clearly an "experts only" option, but for those that like to go down the rabbit hole, I am your humble servant. Down you go... :D

                        And happy scraping! :)

                        1 Reply Last reply Reply Quote 2
                        • AnalogHeroA
                          AnalogHero
                          last edited by AnalogHero

                          @muldjord Dont know if im doing something wrong, but the new query option isnt working for me. I tried this

                          $ Skyscraper -p amiga -s screenscraper --query md5=0D5E4770B34021A666E9CADF0F39DA75 /home/pi/RetroPie/roms/amiga/BlackViper_v1.0_AGA.lha

                          and it outputs
                          1/1 (T1) Pass 1 ---- Game 'BlackViper_v1.0_AGA' not found :( ----

                          What am i doing wrong?

                          EDIT: Ok. It works. Just used a different entry from screenscraper.fr md5=7858DFE8AE9A6725B6CAD18D67FEBA61 worked.

                          muldjordM 1 Reply Last reply Reply Quote 1
                          • muldjordM
                            muldjord @AnalogHero
                            last edited by muldjord

                            @analoghero Yeah, I ran into this problem with amiga games aswell. It needs to be the same platform, otherwise it won't match them even though you put in a custom checksum. So for instance, if you copy a sha1 from a cd32 game and try to override it for a standard amiga game, it won't be found. It will be found if you use the amiga version. So this should not be an issue as I assume people will always be using sha1's from the actual platform.

                            Btw, Skyscraper differentiates the Amiga platforms from the filenames. If a filename has 'cd32' or 'cdtv' in it, it will switch to that platform for that rom.

                            1 Reply Last reply Reply Quote 1
                            • screechS
                              screech
                              last edited by

                              @muldjord Just checked for the Screenscraper DB ;) "Amiga CD32", "Amiga AGA", "Amiga CDTV",... are child of "Amiga" ;) so if you scrape a CD32 rom on "Amiga" system, it normally return the good game referenced on "Amiga CD32" ;)

                              It's the same for "MAME" for exemple, if you scrape a CPS1 game in the MAME System, it will return the good game (not needed to specify CPS1 ;) ).

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @screech
                                last edited by muldjord

                                @screech Yes, this is true, but switching the platform (in Skyscraper) is necessary for the match to occur. If you scrape the 'amiga' platform with Skyscraper, it will not accept an entry for the 'Amiga CD32' platform because some scraping modules are name search based. So it needs to change the platform in order to accept the returned entry. :)

                                With Screenscraper this is less important since it's checksum based. But for the others, it's a necessity since many games are multiplatform, and Skyscraper needs to find the right one. :)

                                1 Reply Last reply Reply Quote 0
                                • T
                                  tacodog
                                  last edited by

                                  I have a couple question about the usage of Skyscraper that I was hoping someone could help with.

                                  My first issue is that I use m3u files for my psx games because it handles multi disc games organically giving me just one entry in emulationstation. My m3u file points to ../psx-discs/...(disc 1).bin ../psx-discs/...(disc 2).bin etc. If I scrape by m3u filename, nothing is found. If I check by disc, I get duplicates for each disk, and the gamelist file doesn't point to the m3u. Is there a way to search by filename rather than checksums?

                                  Secondly, I have scraped all of my systems using screenscraper, and I am missing a handful of files.

                                  Local database cache stats:
                                  'screenscraper' module
                                    Titles       : 428
                                    Platforms    : 428
                                    Descriptions : 426
                                    Publishers   : 418
                                    Developers   : 412
                                    Ages         : 171
                                    Tags         : 414
                                    Ratings      : 279
                                    ReleaseDates : 316
                                    Covers       : 428
                                    Screenshots  : 428
                                    Wheels       : 423
                                    Marquees     : 423
                                    Videos       : 0
                                  
                                  

                                  I'd like to fill in these small gaps (just 3 wheels) from a second source, but don't necessary need to grab everything all over again. Is there a way to tell Skyscraper to only get the missing files? Or can I see which files are missing so I can try to grab them manually?

                                  Thanks

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @tacodog
                                    last edited by muldjord

                                    @tacodog You can scrape single files simply by providing the full or partial path to the rom on command line. You can also use the '--startat' and '--endat' command line options to only scrape a span of roms. As for seeing exactly which roms are missing the data, that is currently not possible.

                                    For the m3u issue, the reason it isn't working for those is that the source you are using (ScreenScraper) doesn't have those exact files in their database. You'd need to overwrite the checksum manually, which can be done with '--query'. Try checking out the release notes for 2.7.5 here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

                                    It's described in there. You can also try scraping it with another source, such as "mobygames" or "thegamesdb". They search by name instead of checksum, so if your filenames are good, they should provide good data. Not for wheels though, as those sources don't support wheels.

                                    1 Reply Last reply Reply Quote 0
                                    • H
                                      hermit
                                      last edited by

                                      would it be possible to scrape Mess roms?
                                      example for Amstrad GX4000 :
                                      barb2.zip = Barbarian 2
                                      brubber.zip = Burnin' Rubber

                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @hermit
                                        last edited by

                                        @hermit Skyscraper supports many of the MESS platforms simply by scraping them with their own names such as:

                                        $ Skyscraper -p nes -s screenscraper
                                        

                                        But GX4000 in particular isn't supported.

                                        1 Reply Last reply Reply Quote 0
                                        • A
                                          annomatik
                                          last edited by annomatik

                                          Hi! Skyscraper scraps many games for me, but some it does not scrap are quite common which I find odd. E.g. Chaos Engine and Lemmings.

                                          Here's the list of some of the currently not scraped games I'm experiencing with Skyscraper:

                                          Amiga

                                          AlienBreed1_v2.3a_0998.lha (2 + 3D scrap fine)
                                          "Bad Ninjas (Homebrew 2018).adf"
                                          Cadaver&CadaverThePayoff_v2.2_0900.lha
                                          ChaosEngine1_v1.2_AGA_1324.lha
                                          ChaosEngine2_v2.1_AGA_0173.lha
                                          CrazyCars1_v1.2.lha
                                          CrazyCars2_v1.1_0311.lha
                                          CrazyCars3_v2.1.lha
                                          DefenderOfTheCrown_v3.1_0317.lha
                                          DeluxePacMan_v1.0_AGA.lha
                                          EyeOfTheBeholder1_v1.5_0116.lha
                                          EyeOfTheBeholder2_v1.1_0834.lha
                                          GhostsNGoblins_v1.2.lha
                                          GhoulsNGhosts_v1.0_2252.lha
                                          Gobliiins1_v1.3_0065.lha
                                          Gobliins2_v1.4_0185.lha (Goblins3 scraps fine)
                                          GreatGianaSisters_v1.6_2945.lha
                                          "Inviyya (Homebrew Demo 2018).adf"
                                          ItCameFromTheDesert1_v2.0_0014.lha
                                          ItCameFromTheDesert2_v2.0.lha
                                          JamesPond1_v2.1.lha
                                          JamesPond2_v2.0_AGA_1354.lha
                                          JamesPond3_v1.0_AGA_0688.lha
                                          "Lemmings 3 AGA (1 of 4).adf"
                                          Lemmings1_v1.5_Files_2089.lha (Lemmings 2 scraps fine)
                                          Llamatron_v1.1_1MB.lha
                                          Lotus1_v1.1_0774.lha
                                          Lotus2_v1.11_0497.lha (Lotus 3 scraps fine)
                                          MegaLoMania_v1.7_0272.lha
                                          MontyPythonsFlyingCircus_v1.3_0273.lha
                                          Nebulus1_v1.3_0361.lha (Nebulus 2 scraps fine)
                                          "Operation Lemming v1.2.adf"
                                          Populous1&DataDisks_v1.1_0069&1217.lha
                                          Populous2&ChallengeGames_v1.3_0079.lha
                                          ProjectX1_v1.3_0886.lha
                                          ProjectX2_v1.3_0289.lha
                                          ProjectXSE_v1.5_0927.lha
                                          RType1_v1.4.1_0940.lha (RType 2 scraps fine)
                                          SensibleTrainSpotting_v1.0.lha
                                          ShadowOfTheBeast1_v2.2_1357.lha
                                          ShadowOfTheBeast2_v1.3a_1359.lha
                                          ShadowOfTheBeast3_v1.6_0016.lha
                                          Speedball1_v2.0_0581.lha (Speedball 2 scraps fine)
                                          SuperMethaneBros_v1.3.lha
                                          TitusTheFox_v2.0_0226.lha
                                          Turrican1_v2.0_0092.lha ( Turrican 2 + 3 scrap fine)
                                          Xenon1_v1.2_0399.lha (Xenon 2 scraps fine)
                                          "Zerosphere (Homebrew 2015).adf"
                                          NES

                                          "Balloon Fight (USA) 2P.nes"
                                          "Banana Prince (GER).nes"
                                          "Castlevania 2 (USA).nes" (Castlevania 1 scraps fine)
                                          "Donkey Kong Jr. (World).nes" (Donkey Kong scraps fine)
                                          "Galaga (USA).nes"
                                          "Ninja Gaiden 1 (USA).nes"
                                          "SCAT (USA) 2P.nes"
                                          "StarTropics 1 (USA).nes"
                                          "Tetris (Nintendo).nes" (Tetris (Tengen) scraps fine)

                                          1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord
                                            last edited by muldjord

                                            @annomatik Most of the LHA's are not yet in the screenscraper database. It will work for some of them with the openretro module, but overall they are poorly supported at this time (for all scrapers, not just Skyscraper). We are in the process of making them work with both openretro and screenscraper.

                                            As for the nes games, try looking them up on the screenscraper site and check if your checksums for your roms match those in their database. If they do please provide some documented examples of matches that don't return a result and I can look into this further. Thank you. :)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.