RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.5m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • muldjordM
      muldjord @AnalogHero
      last edited by muldjord

      @analoghero You need to look at "~/.skyscraper/dbs/[platform]/priorities.xml". That file defines source priorities for the different resources. Currently you probably have openretro above thegamesdb for a bunch of the resource types which makes it prioritize the faulty entries from Openretro. So just move the "<source>thegamesdb</source> up above the "<source>openretro</source>" node for all resources and it should work as you expect.

      EDIT: And yes, the gamelist.xml is NOT being written when scraping filenames on command line. It merely adds the data to the local cache so you can use it with localdb as you did.

      EDIT2: I will advice against deleting the entire dbs folder. Not that it breaks anything, but if you have scraped all platforms, most of the dbs inside the folder are absolutely fine and will be useful for rescrapings later. Instead, consider deleting just the one folder inside dbs matching the platform you are having trouble with. But, best of all, consider using the prioritize feature as described in the documentation.

      1 Reply Last reply Reply Quote 0
      • AnalogHeroA
        AnalogHero
        last edited by AnalogHero

        @muldjord I just deleted the amiga folder insided the dbs folder. Its not that big loss.

        I was thinking about priorities.xml but its a bit hard to decide, as sometimes it isnt clear which source is better or its depends on the game.

        Another thing bugs me for long time: Can the result in number of players somehow streamlined? Most Emulationstation themes cant handle any text. 1 - 2 (2 simultaneous) would only display garbage.

        1 Reply Last reply Reply Quote 0
        • hi_im_daftH
          hi_im_daft
          last edited by

          Question: I know there's a way to do it, and it's probably posted somewhere but I haven't found it yet.

          Can you change the priority for databases when the scraper runs? I don't like the current "primary" database, it throws in a lot of incorrect game names (or maybe just defaults to EU region titles instead of US), so I want to change it to a different one. I'm sure it's in a configuration somewhere to do it.

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @hi_im_daft
            last edited by

            @hi_im_daft Please read the documentation: https://github.com/muldjord/skyscraper
            You set scraper with '-s'. Check all options with '--help'. Please keep in mind that if you only scrape a platform with one scraper you are not using Skyscraper to the best of it's ability. Check here for more information on the local database cache: https://github.com/muldjord/skyscraper#local-database-features

            B 1 Reply Last reply Reply Quote 0
            • B
              bleuge @muldjord
              last edited by

              @muldjord I have some little question. I run Skyscraper in all my folders, some roms are not identified and so no images or metadata.
              But, then, if i run sselph scraper, and end populating the lists, because it can find some things that Skycraper can't.

              If i run again Skyscraper, the local db is updated with the new content downloaded with sselph? So it can be more complete and useful in the future?

              Thanks!

              AnalogHeroA 1 Reply Last reply Reply Quote 0
              • AnalogHeroA
                AnalogHero @bleuge
                last edited by

                @bleuge Try running skyscraper more than one time per system with different scrapers each time.

                You can specify the scraper with -s So if you try to scrape snes do your second run like this : Skyscraper -p snes -s screenscraper . Maybe after this you can do: Skyscraper -p snes -s openretro

                After it finished do as final step Skyscraper -p snes -s localdb so that all scraped resources are used.

                B 1 Reply Last reply Reply Quote 1
                • B
                  bleuge @AnalogHero
                  last edited by

                  @analoghero I am using skyscript.sh generated script, for every platform. So i guess this is right.

                  1 Reply Last reply Reply Quote 0
                  • hi_im_daftH
                    hi_im_daft
                    last edited by

                    If it uses an amalgam of information from different databases though, how does it know what info is "correct"? Or, at least, more accurate to use? That's the problem I fear I'm going to wind up with.

                    muldjordM 1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @hi_im_daft
                      last edited by

                      @hi_im_daft You can prioritize sources for each type of information (check the documentation). But you will undoubtedly end up with some faulty entries, that can't be avoided with an automated scraper like this. If you only want correct results, just only ever scrape with 'screenscraper' and 'arcadedb'. Those are based on mame names and file checksums. But then you'll give up the option of scraping platforms that have scripts to start the games. Such as svm for scummvm and cue files for psx.

                      Bottom line, you can't avoid faulty results with sources that are based on file name searches.

                      1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord
                        last edited by

                        In release related news: I think I tracked down the segfault that made Skyscraper crash sporadically. And I might even have fixed the rare problem some have been reporting where image and or resources would have been mixed up.

                        So, once again, I will be doing testing over the next days. And if I don't run into more trouble, I can start documenting the new compositor and, finally, release 2.3.0... Phew. I have literally been coding on this for 6-10 hours a day for the past week. I hope you guys appreciate this! It's starting to take it's toll here :D

                        1 Reply Last reply Reply Quote 3
                        • hi_im_daftH
                          hi_im_daft
                          last edited by hi_im_daft

                          @muldjord: I'm sure I can say for many of us here that we definitely appreciate the effort. The system itself is pretty great.

                          One last silly question, is it possible to set a different UI theme to use different Metadata? Some themes seems to look a lot nicer with the layout you use (screenshot with boxart in the corner), while others have a smaller area for images and squishing yours looks worse.

                          For example, I think the Skyscraper images look great with Retrorama, but for the NES Mini theme the boxart by itself would look better.

                          I don't think there's a way to do this one, but I am curious.

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • AnalogHeroA
                            AnalogHero
                            last edited by

                            @bleuge Ok if youre using simplemode the script is correct. Sometimes roms doesnt get scraped because the match wasnt close enough. What im doing then is lowering the -m value. For example Skyscraper -p snes -m 50. Answer Y to overwrite and Y to skip existing entries . You can lower the value if you like in a second run.

                            @hi_im_daft If the information online isnt correct, your scraped data will be so. If you have like 700 snes games for example, i would never know if a few games have incorrect descriptions.

                            1 Reply Last reply Reply Quote 1
                            • AnalogHeroA
                              AnalogHero
                              last edited by

                              @hi_im_daft No, you have to decide. Emulationstation always uses the same gamelist.xml, no matter what theme youre using. This would ve a nice addition to ES imho. You should ask the dev of ES!

                              1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @hi_im_daft
                                last edited by muldjord

                                @hi_im_daft If you want to change the look of the artwork exported by Skyscraper you can do that by editing "~/.skyscraper/artwork.xml". This is also well documented, so please check it out. Then, when you're done editing that file, rescrape with '-s localdb' and it will create new artwork from the recipe in artwork.xml. As AnalogHero notes, you can't have different screenshot artwork per ES theme, but if the theme uses the cover/boxart artwork specifically in one theme, and the screenshot in another, then those two can be exported from Skyscraper, and it will use one with one theme, and the other with the other. It's all up to how the ES theme is composed, and not something Skyscraper has control over.

                                So bottom line, if you have one ES theme only using cover, and another theme only usinge screenshot, then edit artwork.xml so it exports both to your liking. Then it will work as you expect.

                                With that said, please wait a few days. The new 2.3.0 release has A LOT of eye candy compared to the current release of Skyscraper. It can even render 3D boxart just by adding a <gamebox/> node to an artwork.xml <layer> node. And it will be well documented and easy to use.

                                1 Reply Last reply Reply Quote 0
                                • hi_im_daftH
                                  hi_im_daft
                                  last edited by

                                  I'll definitely be looking forward to the new release. Thanks for all the help guys. :)

                                  1 Reply Last reply Reply Quote 1
                                  • B
                                    bleuge
                                    last edited by

                                    Is it possible to add Bandai Wonderswan and Color platforms?
                                    I can't see it in the platforms source code.
                                    Thanks!

                                    muldjordM 1 Reply Last reply Reply Quote 0
                                    • muldjordM
                                      muldjord @bleuge
                                      last edited by

                                      @bleuge Sure, it'll be in 2.3.0 when it's done (which is really soon now!)

                                      1 Reply Last reply Reply Quote 0
                                      • AnalogHeroA
                                        AnalogHero
                                        last edited by

                                        @muldjord This is awesome, cant wait. Did you had the time to look into the number of players issue? Its not that important really, just add it to your todo list :)

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @AnalogHero
                                          last edited by

                                          @analoghero I'm sorry, I forgot what that was... Can you refresh my memory?

                                          Aaaaaaaaaaadn, release is imminent now. I've created a quick video showing off the new compositor features and I've completely updated the artwork documentation. Pretty much all I need to do now is go through the last couple of sanity checks and 2.3.0 will be released. Which means it'll be out within an hour probably. :)

                                          B 1 Reply Last reply Reply Quote 1
                                          • B
                                            bleuge @muldjord
                                            last edited by

                                            @muldjord profanitywordhere man, you work pretty fast. Thanks for doing the best scraper around.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.