RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      annomatik
      last edited by

      Wow, that's a ton of games there... :-)

      I just checked the first couple of games. Balloon Fight, Banana Prince, Castlevania 2, Donkey Kong Jr. : many versions are there, but not with the checksums I have.

      So... I can think of at least four solutions:

      • Do I really have to keep looking for the right roms ? That would be probably a lot of effort.
      • I could try adding my own versions to the lists, but why, the meta-data is already there, the roms are working fine, there's already many different versions of the same game (why not just have a list of different checksums for the same game?)...
      • Could Skyscrapper just match filenames, not CRC sums? How would I enable that? and, if not
      • Can I maybe give Skyscrapper a pointer to the right meta-data?

      In Kodi, it is possible to add a .nfo file, which can contain the link to the details-page of a movie / tv show episode. It is in the exact same solution like the video, just instead of mp4/mkv/whatever, it has the extension nfo. In its simplest form, it's just a simple link, but it can also be used for storing all meta-data, if wanted. And if combined with a link, it first takes the provided meta-data from the nfo and takes the rest via scrapping from the nfo.

      Maybe, if that's not yet possible, it would be possible to provide direct links somehow?

      Thanks!

      muldjordM 1 Reply Last reply Reply Quote 0
      • T
        tacodog
        last edited by

        @muldjord Thanks for the replay about the psx games. I was able to write a script that would read the m3u file to get the cue file, and then grab the checksum of the bin/cue files and grab the correct images using --query. It works great.

        I have another question/issue. I just scraped my nes roms and Super Mario 2 seems to grab the japanese wheel no matter what I do. I've tried --region 'us', 'eu', and 'wor', but I always get the "Super Mario USA" wheel.

        Is this an issue with the way I'm running Skyscraper? an issue with the program itself? or an issue with the screenscraper database? Can you repeat this?

        The wheel I want is located here:
        https://screenscraper.fr/image.php?gameid=1248&media=wheel&hd=1&region=wor&num=&version=&maxwidth=600&maxheight=300

        Skyscraper -p nes --refresh --region 'wor' --unpack -s screenscraper -i . -o . -g . --pretend /mnt/exhd0/roms/nes/Favorites/Super\ Mario\ Bros.\ 2\ \(USA\)\ \(Rev\ A\).zip
        

        The md5 of my (unzipped) rom is d769d73689ea4684dbb0bd7cf3a24b2f.

        Thanks

        muldjordM 1 Reply Last reply Reply Quote 0
        • muldjordM
          muldjord @annomatik
          last edited by

          @annomatik Skyscraper already matches just filenames when it looks for games at screenscraper. But it only works if the filename is unique in their database. So if they have 2 roms with the same filename, it won't work. This is not a Skyscraper thing, but a screenscraper thing, so not something I can change. You can try scraping the games with some of the other sources. They are filename search based.

          You can, however, overwrite the checksums Skyscraper uses when looking for the games with the '--query' option. Read more about that here: https://github.com/muldjord/skyscraper/releases/tag/2.7.5

          It works from 2.7.5 and newer. :)

          1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @tacodog
            last edited by

            @tacodog I'll need to look into that when I get the time. It could be a bug of some sort. If the '--region us' doesn't pull it in, there might be a problem. If the wheel exists in their database, it should use it. If it didn't it will prioritize other regions until it finds one that does exist. So either there's a problem with the us wheel in their database, or there's an issue in Skyscraper. Should be easy to reproduce though.

            1 Reply Last reply Reply Quote 0
            • T
              tacodog
              last edited by

              @muldjord One more request/suggestion. Would it make sense to divide the import directory by system? My understanding is that all images now go in import/screenshots. So if I wanted to use custom images for Mortal Kombat 2, I would have to remember to change the images in the import directory when building snes or megadrive databases. Not really a big problem, just an idea.

              muldjordM 2 Replies Last reply Reply Quote 1
              • A
                annomatik
                last edited by

                Cool, will try the query option. Thanks!

                1 Reply Last reply Reply Quote 0
                • A
                  annomatik
                  last edited by

                  Didn't work :-( I used the following script, it said it scraped something, but when I started Emulationstation again, there still was no meta-data for the games I specified:

                  #!/bin/bash
                  ./Skyscraper -p nes -s screenscraper --query sha1=F5E4CD9F287A8233EEE35AD1881B2770D9F2C453 "/home/pi/RetroPie/roms/nes/Balloon Fight (USA) 2P.nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=74357C80B61FFD25AC2A802391068600F2646D98 "/home/pi/RetroPie/roms/nes/Banana Prince (GER).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=85793B0F013EC0F24F36FED0A2859B3B74131510 "/home/pi/RetroPie/roms/nes/Castlevania 2 (USA).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=02633E208732B598E3A8EB80B6E0E09926F25E83 "/home/pi/RetroPie/roms/nes/Donkey Kong Jr. (World).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=DA54C223D79FA59EB95437854B677CF69B5CAC8A "/home/pi/RetroPie/roms/nes/Galaga (USA).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=CA513F841D75EFEB33BB8099FB02BEEB39F6BB9C "/home/pi/RetroPie/roms/nes/Ninja Gaiden 1 (USA).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=74A15693EEBC2C01C7982C0C9A3F294BC26F53A7 "/home/pi/RetroPie/roms/nes/SCAT (USA) 2P.nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=FDD3D68C7B5B2F9DC7EC50D9888B0A8EDBAAEC28 "/home/pi/RetroPie/roms/nes/StarTropics 1 (USA).nes"
                  ./Skyscraper -p nes -s screenscraper --query sha1=77747840541BFC62A28A5957692A98C550BD6B2B "/home/pi/RetroPie/roms/nes/Tetris (Nintendo).nes"

                  1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord
                    last edited by

                    @annomatik You need to always rescrape with:

                    $ Skyscraper -p [platform]
                    

                    when you've added new data to the cache as you just did. That will regenerate the gamelist.xml using all of your cached data. Please read the github documentation thoroughly. It's all very well-documented including these features. :)

                    1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @tacodog
                      last edited by

                      @tacodog That's a good idea, I will add it to the roadmap. :)

                      1 Reply Last reply Reply Quote 0
                      • A
                        annomatik
                        last edited by annomatik

                        Thanks, that worked. To be honest, I'm a little overwhelmed by the doc, so if it's ok for you, I'll just ask my question here.

                        Next topic. After scraping, I now have a "Adventure Island 3", a "Adventure Island Classic" and a "Adventure Island II".

                        What I'd like to have instead: "Adventure Island 1", "Adventure Island 2" and "Adventure Island 3".

                        Where would I specify that so it does not get overwritten with the next db update? Thanks

                        alt text

                        muldjordM 1 Reply Last reply Reply Quote 0
                        • A
                          annomatik
                          last edited by annomatik

                          Also, is it possible to configure the console-colors used? The yellow is very hard to read on a white background. Thanks!

                          alt text

                          1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord @annomatik
                            last edited by

                            @annomatik If you wish to use your own titles (and keep them) for any game, I suggest using the 'import' feature. Please read the documentation on how to use that. I am not gonna go over it here as it is already well-documented.

                            I will answer questions that aren't covered in the documentation, but I expect you to take your time and read it, overwhelming as it might be. Just read the sections you need to learn about. It'll be worth it in the end.

                            As for terminal colors, I don't know if you can change them in Putty. I am not a Windows user myself, so you will have to look into Putty's documentation. There is not a way to change them in Skyscraper.

                            1 Reply Last reply Reply Quote 0
                            • A
                              annomatik
                              last edited by

                              Ok, thanks.

                              1 Reply Last reply Reply Quote 0
                              • M
                                martin_f
                                last edited by

                                Just started using Skyscraper, excellent little tool muldjord ! Appreciate your time in putting this together. I had been using UXML for a little bit, but much prefer Skyscraper.

                                One question I have. I was reading dorkvader's comments regarding the "spine". Am I correct in assuming that at current time Skyscraper doesn't scrape "spine" art, such as that on https://www.screenscraper.fr/ ? If not, is there any plan to add this feature?

                                dorkvader, were you manually downloading the spine art, or are there packs floating about?

                                Anyway, keep up the good work, I am sure its appreciated by many.

                                muldjordM 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @martin_f
                                  last edited by

                                  @martin_f I don't really plan to expand the types of resources that can be fetched currently. But you can sortof cheat and use the 'marquee' as the 'spine' art if you don't use it for anything already. I did this by downloading some of the spine art packs from emumovies, then importing them as 'marquee' and then added them on the side of the boxart effect. That turned out perfect.

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    martin_f @muldjord
                                    last edited by martin_f

                                    Nice one muldjord, I will give that a go.

                                    And thanks again for such an awesome tool !

                                    1 Reply Last reply Reply Quote 1
                                    • muldjordM
                                      muldjord @tacodog
                                      last edited by muldjord

                                      @tacodog I have just implemented the feature to have subfolders in the import folder. So if you have 'import/snes/screenshots' it will look for specifically snes screenshots in there and so on.

                                      It's not officially released yet, just wanted to let you know that the feature is done and will be in the next release. You can already try it out now if you'd like to, it's on the master branch on git.

                                      EDIT: It is now released in 2.7.8

                                      1 Reply Last reply Reply Quote 3
                                      • muldjordM
                                        muldjord
                                        last edited by

                                        Skyscraper version 2.7.8 released: https://github.com/muldjord/skyscraper

                                        • Added CRC32 checksum attribute to 'screenscraper' pass
                                        • Changed the 'import' folder to also look into 'import/[platform]' if it exists so users can create imports per platform. Old method also works (Thanks to tacodog for suggesting this)
                                        • Added 'mpixels' attribute to compositor layers that allows users to scale images to a desired area size instead of using width and height (Thank you to Vynce for suggesting this)

                                        Next up is 2.8.0 which will implement some interesting preserve features for the current gamelists. For instance, 'hidden' and 'favorite' aren't preserved for the EmulationStation gamelist.xml currently. They will be from 2.8.0 and forward.

                                        1 Reply Last reply Reply Quote 4
                                        • T
                                          tacodog
                                          last edited by

                                          Thanks for adding the import subdirectories. I just tested it and it's working well. Just a note for anyone trying this, 'definitions.dat' needs to go in the system subdirectory as well if you want to use textual data. This took me a little while to figure out.

                                          muldjordM 1 Reply Last reply Reply Quote 1
                                          • muldjordM
                                            muldjord @tacodog
                                            last edited by muldjord

                                            @tacodog I was actually wondering if people would have problems figuring that out as it isn't all that intuitive to also have to put it in the platform subfolder. So thank you for reporting this, I will consider making it work by only having it in the import base folder.

                                            EDIT: This is fixed in the upcoming 2.8.0 release (not released yet). It will first look for 'definitions.dat' in the platform subfolder and then in the base import folder.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.