RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • HalvhjearneH
      Halvhjearne
      last edited by

      really nice tool
      skyscraper+screenscraper was able to scrape 98% of all my nintendo and sega roms. :D

      however i do have a problem when scraping fba and mame roms.
      as example:
      1941 when scraped with sselph scraper returns the name "1941: Counter Attack (World 900227)" from arcadeitalia and 1941u returns the name "1941: Counter Attack (USA 900227)", but when i scrape it with skyscraper, all 1941 variants have the name "1941: Counter Attack" and its impossible to tell them apart.
      is there a way to force it to return the name in the same way as sselph's scraper?
      i noticed that the nintendo and sega games show version and region in the name, but my guess is thats maybe from the filenames?
      idk

      anyway, it would also be nice to see a detailed description of the options in config.ini, its rather hard to figure out how or what some of them does.

      otherwise, keep up the good work :)

      muldjordM 1 Reply Last reply Reply Quote 0
      • muldjordM
        muldjord @Halvhjearne
        last edited by muldjord

        @Halvhjearne It already does so :) It's only in the output it doesn't show the bracket notes (if not you might have disabled it with the --nobrackets option or brackets="false" in config.ini. Please reload your ES and check it. It shows up as "1941: Counter Attack (World 900227)" here on my system.

        HalvhjearneH 1 Reply Last reply Reply Quote 0
        • HalvhjearneH
          Halvhjearne @muldjord
          last edited by

          @muldjord
          i tried it a few times and every time i scraped with skyscraper, all variants have the same name.

          its running now with brackets="true" as i thought maybe i got it wrong and it takes a while to scrape, when only using 1 thread, however its almost done now and i will see what came out and if its still the same i will try with brackets="false", but isnt that supposed to be default behaviour?

          im pretty sure i tried that too with same result or maybe im just confused by now ...

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @Halvhjearne
            last edited by muldjord

            @Halvhjearne I think you are missing out a bit. If you scrape with the same scraping module twice, it will be really fast, unless you have enabled "--refresh". If refresh is enabled it will rescrape all of the files from the source again. No need for that. Skyscraper has a cache that is much faster. With refresh disabled, it will simply use the already cached data. I recommend skimming the documentation if you haven't already done so. Understanding how the cache works is pretty important if you want to use Skyscraper to its full potential: https://github.com/muldjord/skyscraper . It's a very powerful tool beyond just scraping from a single source.

            Either way, I am not sure what you have done to your setup, but by default, if nothing in the config has been changed, it will have the USA and WORLD designations for fba and mame roms. I just tested it here and it works perfectly.

            For a quick use case example, check here: https://github.com/muldjord/skyscraper/blob/master/USECASE.md

            HalvhjearneH 1 Reply Last reply Reply Quote 0
            • HalvhjearneH
              Halvhjearne @muldjord
              last edited by

              @muldjord
              so it only cache the pictures and not the text?

              muldjordM 1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord @Halvhjearne
                last edited by

                @Halvhjearne No, it caches everything. Including the USA and WORLD designations. But it's a bit much to explain here, try reading through the documentation. I think you'll understand once you have.

                HalvhjearneH 1 Reply Last reply Reply Quote 0
                • HalvhjearneH
                  Halvhjearne @muldjord
                  last edited by

                  @muldjord
                  i think i got it now, but i am using the "interface" from retropie setup and im pretty sure there is a bug with it, if its supposed to work as you explained.
                  i just scraped with brackets="true" in config.ini and now it shows up as i wanted it.

                  in the interface it seems not to care what setting its set to, it just removes brackets with fba and mame games, no matter what, but after setting brackets in config.ini, it now works.

                  i hope it still scrapes nintendo and sega games as expected tho ... lol

                  muldjordM 1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord @Halvhjearne
                    last edited by muldjord

                    @Halvhjearne Have you updated the RetroPie script lately? There was a bug with the bracket option at one point I think. If that doesn't fix it, please report it to @mitu here (he's the one who made the RetroPie script for Skyscraper): https://retropie.org.uk/forum/topic/19588/skyscraper-now-officially-part-of-retropie-please-test

                    HalvhjearneH 1 Reply Last reply Reply Quote 0
                    • HalvhjearneH
                      Halvhjearne @muldjord
                      last edited by Halvhjearne

                      @muldjord
                      it was updated within the last 3 days

                      edit:
                      i like that it does not sort "the" under t, although i almost got used to that by now, so probably wont be able to find the games i play that starts with "the" :D

                      muldjordM 1 Reply Last reply Reply Quote 1
                      • muldjordM
                        muldjord @Halvhjearne
                        last edited by

                        @Halvhjearne Yeah, that's one of the earliest features I added back a year ago or so. :)

                        1 Reply Last reply Reply Quote 0
                        • HalvhjearneH
                          Halvhjearne
                          last edited by

                          i got an api key from igdb and i added userCreds="APIKEY" in config.ini, but for some reason it keeps saying the key dosnt work even if i run it at command line with -u APIKEY, am i doing something wrong?

                          also i noticed that when i set maxFails in config.ini over 200 it will always end when it reaches 42 and that it for some reason always refreshes when scraping mame-mame4all ... is this intended behavior?

                          S 1 Reply Last reply Reply Quote 0
                          • S
                            sglavach @Halvhjearne
                            last edited by

                            @Halvhjearne There are responses to this issue below but in a nutshell IGDB scraping won't work until the next release and you won't need your personal api key as it will use an api key assigned to skyscraper. There will also be a limit to the number of skyscraper api calls per day/month I believe.

                            1 Reply Last reply Reply Quote 1
                            • BazB
                              Baz
                              last edited by

                              I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @Baz
                                last edited by

                                @Baz said in Versatile C++ game scraper: Skyscraper:

                                I noticed Donkey Kong Country 3: Dixie Kong's Double Trouble! scrapes as Donkey Kong Country 3: Dixie Kong's Double Troubl (no E and no !). Is that down to the scraper source (screenscraper.fr) or is it a problem with Skyscraper dealing with exclamation points?

                                That would indicate an error at screenscraper, yes. :)

                                1 Reply Last reply Reply Quote 0
                                • S
                                  Slow1
                                  last edited by Slow1

                                  Firstly thanks for such an awesome scraper!

                                  Got this running from a VM processing and storing all the data on my NAS and it's working great!! :)

                                  Secondly, I have a question about matching ROMs that can't be found. Is there a nice easy way to do this via Skyscraper?

                                  An example would be Bust-A-Move '99.cue with a MD5 of cd1fac631f6a45ae4ae2b28738ba4f6a which doesn't match anything on screenscraper.

                                  However I expected it to match on name? I can see on the info pane on the left it's listed by name as Bust-a-move 3 Dx. Would my ROM need to be named to same (or rather added to the alias csv) in order to match? Or am I totally misunderstanding here :D

                                  Edited: Added more info

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @Slow1
                                    last edited by

                                    @Slow1 It needs to match exactly one of the filenames at screenscraper and there needs to exist only one in total in the entirety of screenscraper before it works. If two platforms has a file with the same name, it won't return anything on filename search.

                                    Instead I recommend checking the --query option as described here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

                                    It allows you to overwrite the md5 sum for instance.

                                    S BazB 2 Replies Last reply Reply Quote 1
                                    • S
                                      Slow1 @muldjord
                                      last edited by

                                      @muldjord fantastic! Thank you - down I go..! :D

                                      1 Reply Last reply Reply Quote 1
                                      • BazB
                                        Baz @muldjord
                                        last edited by Baz

                                        @muldjord I could do with using this actually because I can't get Pokemon Blaze Black (a romhack version of Pokémon: Black Version) to scrape for love nor money.

                                        I'm a complete linux noob but would it be $ Skyscraper -p nds -s screenscraper --query md5=37BFF1431EDA9B3A525737C7F59A432D PokemonBlazeBlack.nds , presuming I rename my rom to PokemonBlazeBlack.nds?

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @Baz
                                          last edited by

                                          @Baz Looks correct yes :)

                                          1 Reply Last reply Reply Quote 1
                                          • megaJekanM
                                            megaJekan
                                            last edited by

                                            Hi there. I am used to emulationstation internal scrapper and just switched to skyscraper few days ago. In general, I like the way it works, also it could easily scrap my TurboGrafx-CD images. But there are few things I wanted to specify.

                                            It seems like after scrapping (using retropie menu), the games are removed from the favourites list. Could it be fixed somehow or is there a parameter in config.ini to keep games in the list?

                                            Maybe you can also advice how to search only for newly added roms/images without updating and overwriting the whole database. I have some manual changes in Mega CD names (like disc 1, disc 2), which I would like to keep.

                                            mituM megaJekanM 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.