RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.7m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      movisman @muldjord
      last edited by movisman

      @muldjord

      Hi,

      I think I solved the problem with Amiga. My CD32 files run from *.cue files (which is the most common for this system I think), and I had a suspicion that maybe the scraper wasn't supporting them.

      So I added *.cue to an Amiga specific config.ini, and then scraped again from the local db.

      It pulled the file straight in!

      As a request, can you add *.cue support to Amiga possibly? Those who run CD32 will pretty much only use .cue files to launch I reckon.

      Cheers!

      1 Reply Last reply Reply Quote 0
      • M
        movisman
        last edited by movisman

        @muldjord

        One other question from me, do you think you might ever incorporate a function into the scraper where it is able to 'detect' the region?

        I used a beta version of "Skraper" on my previous installation, which uses screenscraper.fr only, and this has a tick option in the GUI called "Use ROM region first (when available)", and then it has a free text priority list underneath, where you can put US, EU, etc - which I guess it uses if it cannot work out the ROM region itself.

        When this option is ticked, any US roms I have get US artwork and naming, and EU ones get EU artwork and naming, etc, etc. It happens automatically, and seems generally very accurate.

        I don't know how it works it out, maybe it uses information from the existing filename or something, not sure. But either way, is this something you could potentially incorporate as an additional switch, rather than having a fixed priority list?

        I could use Skraper again, but I like the compositor and artwork style on yours better and prefer scraping from the command line as Skraper doesn't really work on Linux yet. And the fact I can use an SHA1 value as a one time override is excellent.

        Cheers!

        mituM 1 Reply Last reply Reply Quote 0
        • mituM
          mitu Global Moderator @movisman
          last edited by

          @movisman Skyscraper already has a switch to choose the region --region and even the language (--lang) for the sources that supports it.

          M 1 Reply Last reply Reply Quote 0
          • M
            movisman @mitu
            last edited by

            @mitu

            Hi there,
            No worries, but that wasn't really what I was referring to - I know about the switch to set a region and have used it to set to US first, but Skraper has an option to set the region by rom somehow - before using a priority. It means my mixture of EU and US games get picked up correctly on a single scrape, whereas with skyscraper you have to explicitly set a region for the scrape or leave as the default priority (EU first). So it is an all or nothing approach, which doesn't work so well if you have say, a megadrive folder with both EU and US ROMs.

            Does that make sense?

            mituM 1 Reply Last reply Reply Quote 0
            • mituM
              mitu Global Moderator @movisman
              last edited by

              @movisman said in Versatile C++ game scraper: Skyscraper:

              Does that make sense?

              I understood your request in the first place, I didn't know if you were aware of the command line switches to choose region/language.

              M 1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord
                last edited by muldjord

                @movisman I don't currently plan to change the existing region and language functionality. Currently it has a prioritized list it tries. If you set one manually it will move that region og language to the top of that list.

                I have added .cue to the list of default supported files for Amiga.

                M 1 Reply Last reply Reply Quote 0
                • M
                  movisman @mitu
                  last edited by

                  @mitu

                  No worries, I thought there was some confusion, but thanks for the heads up, and as always, thanks for helping.

                  1 Reply Last reply Reply Quote 0
                  • M
                    movisman @muldjord
                    last edited by

                    @muldjord

                    Perfect, thanks for adding .cue to the list of files.

                    No worries about the region functionality, if that won't change in the foreseeable future, I could always pop the EU games in another directory, scrape as US, move the EU games back, then scrape again as EU but use the switch to skip the previously scraped files. That should work...

                    I wonder how Skraper does it, perhaps it looks at the filename and makes a judgement. I have no idea but it does work quite well.

                    Cheers for all of the help so far.

                    muldjordM 1 Reply Last reply Reply Quote 0
                    • muldjordM
                      muldjord @movisman
                      last edited by muldjord

                      @movisman Can you give me examples of your filenames? I've decided to implement the region override (after a bit of thinking I realized this was a pretty easy thing to implement). But I need filename examples of how they are designated. For instance, are they called "Gamename (Europe).zip"? or maybe "Gamename (eu).zip"?

                      EDIT: The implementation is done now. I've implemented it so it looks for "europe", "usa", "japan", and "world" in the filename, but only if it comes after the first "(" and the user hasn't manually set the "--region" flag. It will then add that region to the top of the priority list automatically.

                      EDIT2: @movisman Skyscraper 2.8.2 now tested and released, please update and test out the region detection. You should get the artwork from the region that's in the filename if they are from the above mentioned regions.

                      1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord
                        last edited by muldjord

                        Skyscraper version 2.8.2 released: https://github.com/muldjord/skyscraper

                        • Now autodetects region from filename and prioritizes it when scraping with "screenscraper" unless user has forced region with "--region" option (thank you to "movisman" for suggesting this)
                        • Fixed issue where "wheel" artwork would not be fetched for specified region even though it exists
                        • Added ".cue" file extension to Amiga
                        M 2 Replies Last reply Reply Quote 1
                        • M
                          movisman @muldjord
                          last edited by

                          @muldjord

                          Amazing!! Thank you for implementing this!
                          I have been out this evening, but will give it a go shortly and let you know.

                          You are correct, most of the files are (Europe), (USA), (Japan), and (World). Some are a combination, eg. (US, Europe) - I guess in this instance it should scrape the US one.

                          Will let you know shortly how it goes. I'll test on one of the folders which has these brackets on every file (megadrive + snes).

                          Cheers

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord @movisman
                            last edited by

                            @movisman For the ones with (Europe, USA) or similar, it will look for the first one first, then the second one and so forth. So in this examples it would look for 'eu' first. And if that isn't found, it will look for 'us'. And if that isn't found, it will go back to it's original prioritization.

                            1 Reply Last reply Reply Quote 0
                            • M
                              movisman @muldjord
                              last edited by

                              @muldjord

                              Just tried Megadrive, and the good news is it works way better, it is picking up the region as you specify above.

                              However, something I noticed previously which seems to be the same here, is some (Europe) games scraped as EU have the US marquee for some reason. Three examples:

                              Another World (Europe) - shows the 'Out Of This World' logo, which is US
                              Thunder Force IV (Europe) - shows the 'Lightening Force' logo, which is US
                              Chaos Engine, The (Europe) - shows the 'Soldiers of Fortune' logo, which is US

                              The box is the European type, so that's good - it's just the marquee which displays incorrectly.

                              Does this need the same treatment with the region autodetect you implemented?

                              I don't believe this is a screenscraper.fr issue, because if I scrape these with skraper, the marquees are all detected as the correct EU ones. Also if I do a manual lookup, say on Another World, you can see the 'wheel' lists the Europe and US logo correctly, as does marquees etc.

                              Thanks!

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @movisman
                                last edited by

                                @movisman Probably because they are cached and therefore not rescraped. Try scraping with '--refresh'.

                                M 2 Replies Last reply Reply Quote 0
                                • M
                                  movisman @muldjord
                                  last edited by

                                  @muldjord

                                  Yep I was just editing my post to say I have already refreshed with --refresh.

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @movisman
                                    last edited by

                                    @movisman The method I've implemented concerns all data that has a region, so it should work for all types.

                                    I'll need to look into this. "Another World" is probably a good examples to test it with, so I'll try that.

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      movisman @muldjord
                                      last edited by

                                      @muldjord

                                      Command used for testing:

                                      /opt/retropie/supplementary/skyscraper/Skyscraper --unattendskip --refresh -g /home/pi/RetroPie/roms/megadrive -o /home/pi/RetroPie/roms/megadrive/media --relative -p megadrive -s screenscraper -u username:password --videos

                                      username:password was filled out correctly of course.

                                      Cheers

                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @movisman
                                        last edited by muldjord

                                        @movisman Just tested with "Another World (Europe)" for snes. It works as expected. I don't get the "Out of this world" wheel. And if I force us region it also correctly shows the "Out of this world" resources. I'm not sure what could be causing your issue. Have you perhaps scraped the platform with other sources aswell? It might use the artwork from cache from other sources.

                                        EDIT: It seems like SS is down at the moment? I can't scrape anything from it right now. It worked just a few minutes ago.

                                        EDIT2: It's working again it seems. Odd.

                                        M 2 Replies Last reply Reply Quote 0
                                        • M
                                          movisman @muldjord
                                          last edited by

                                          @muldjord

                                          Hi,

                                          No I have only scraped against screenscraper.fr so far, as it has pretty much everything I need, and if it's missing a video I obtain one and submit it. Maybe I could clear the db folder out for megadrive.

                                          I was working with the Megadrive version of Another World, perhaps that version is worth a try?

                                          Or any of the above three examples maybe?

                                          Cheers

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            movisman @muldjord
                                            last edited by movisman

                                            @muldjord

                                            Just tried to purge the DB for megadrive within the GUI, and I ran the command above again, but the same occurs. Correct boxart but the US marquee/wheel on those three files I mentioned, but they have an (Europe) filename and european boxart.

                                            This was happening before btw when I was playing around with the region switch. Not sure this happens for all roms, but I am not sure what the differentiator is.

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.