RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      movisman @muldjord
      last edited by

      @muldjord

      Hi,

      Thanks for the help. I have read all the options on the command line, I always try to check stuff out myself first - but am still slightly confused by the usage of the --skipped option and when it's required? I must be missing something obvious, but I am not sure why it would be used when you run either interactively or with the --unattend option.

      Are you able to elaborate a bit for me? Apologies if I am missing something here.

      Understood about the config.ini file, I think once I have a command working the way i'd like, i'll transition those into a .ini file for ease of use.

      Thanks

      mituM 1 Reply Last reply Reply Quote 0
      • mituM
        mitu Global Moderator @movisman
        last edited by

        @movisman I think --unattend also adds --skipped, I just left it in the scriptmodule to be set anyways. The --unattendskip was removed in the last version of the scriptmodules.

        M 1 Reply Last reply Reply Quote 0
        • M
          movisman @mitu
          last edited by movisman

          @mitu

          Cheers. I need to use --unattendskip so it doesn't overwrite my renamed files, so I can leave --skipped off then I guess?

          Does --skipped have any use on it's own? Considering interactive asks whether you wish to skip anyway - and --unattend also adds it? Does it have any use when combined with --unattendskip? From what I can tell, it appears not?

          Thanks!

          muldjordM 1 Reply Last reply Reply Quote 0
          • M
            movisman
            last edited by

            Sorry, corrected my command above as it was wrong!
            Cheers

            1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @movisman
              last edited by muldjord

              @movisman You seem to misunderstanding what --skipped does. The skipping prompt is not related to --skipped.

              Answering yes to skipping at the beginning will skip all entries you already have in your gamelist, and therefore not try to rescrape them (and also not recreate the media for them). I never personally answer Y here as it makes more sense to let them be rescraped and regenerated from the cache. Answering Y will save you some time, but that's all. Basically it only makes sense to answer Y if you have manually changed some of the publishers, developers or data for some games in the gamelist.

              Adding --skipped will make sure games that returned "not found" are still added as empty entries to the gamelist when it's generated at the end of a scraping run. That's all this option does.

              I never personally use the --skipped option, but some people like to have those entries in the gamelist.

              M 1 Reply Last reply Reply Quote 0
              • M
                movisman @muldjord
                last edited by

                @muldjord said in Versatile C++ game scraper: Skyscraper:

                Adding --skipped will make sure games that returned "not found" are still added as empty entries to the gamelist when it's generated at the end of a scraping run. That's all this option does.

                This is exactly what I was looking for. That's what it does! Thank you! Maybe the help could be slightly tweaked to mention the behaviour in slightly more detail, it might just be me though!

                I actually don't want that for my use case, so I can safely not use that option. I also have to answer Y to skip existing entries for certain systems (or use --unattendskip) rather than have them be rescraped. The reason for this, is after scraping successfully, I have to rename the files back to a .chd extension for some systems (and edit gamelist.xml to suit), and answering N means they get rescraped and not found due to unique checksums etc, thus overwriting their entry in the gamelist with nothing.

                Unless you have any ideas on a better way for me to manage this, I think I am ok to just use --unattendskip, because if I add the odd game later, I can use this command along with a query (if needed) to append new entries to the gamelist, and it won't overwrite my CHD entries mentioned previously.

                Some systems which use .zip, are of course a lot easier. Does that make sense?

                Thanks a lot for your help as always, this scraper is really very flexible and powerful.

                1 Reply Last reply Reply Quote 1
                • M
                  movisman
                  last edited by

                  Me again :)

                  I see there is Amiga support, but should it work with CD32 games (this is a separate section on screenscraper)? Running a basic command doesn't seem to yield any results, however, when I run a query against a valid SHA1 checksum, it finds and scrapes the information to the DB.

                  However when running the command again but with -s localdb, it doesn't write anything at all to the gamelist.xml or copy the media across. I am not skipping any files or anything like that.

                  Using the same command with other system (Atari 2600) is fine.

                  Before I spend more time troubleshooting, are you able to confirm if the system is even supported?

                  Thanks!

                  muldjordM 1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord @movisman
                    last edited by muldjord

                    @movisman Yes, the system is supported, but the Amiga platform is a bit difficult to work with as it is basically more systems inside one system. So Skyscraper looks for the "aga", "cd32" and "cdtv" in the filenames to switch the platform when scraping them. Please try scraping them using the openretro source. It will only work with games that have 2 or less words in them due to a limitation on their website, but openretro is the best source for amiga games.
                    If it scrapes the data when using a custom sha1 and the data is entered into the local cache, it will use the data when scraping with '-s localdb' afterwards. If it doesn't something is wrong with your command or your config.ini.

                    M 2 Replies Last reply Reply Quote 0
                    • M
                      movisman @muldjord
                      last edited by

                      @muldjord

                      Hi there,

                      Thanks for the info. I stayed up late last night to try and get my systems scraped from the command line, and generally I had very good success.

                      Using a platform specific variant of the same command line for each system, I managed to scrape all of the obvious ones that have common .zip filenames with no issue at all. Others systems were a bit more picky, especially with me using .chd on all supported emulators, but I was able to sort with manual lookups and some file renaming to fool the scraper into obtaining the correct data.

                      A few of them incorrectly picked the wrong region for the game, for example Contra Hard Corps (USA) on Megadrive was picked up as Probotector, which is the European name. I haven't played around with the region forcing flags yet though. This is the next thing for me to tackle/address. Luckily there are only a few which have completely different names.

                      The only platform I had major issues with was the Amiga.

                      I was happy to do a manual lookup for every game (or any which do not pick up automatically). As there are about 20 in total so it won't take forever, and once it's done it's done. The plan was to do a manual lookup on a valid SHA1 and have it write to the localdb. I did a couple of tests and got a match on screenscraper. I was then going to scrape the localdb so it could write the changes to the gamelist.xml, but for Amiga only, this was just not happening. Using the same command (but changing platform and directory to suit) on another system, it was writing just fine. So I have no idea what is going on, i'll find the commands I used and paste them here.

                      Thanks for the heads up about having 'cd32' in the filename, some of the titles do not have this. I will also look into trying openretro.

                      1 Reply Last reply Reply Quote 0
                      • M
                        movisman @muldjord
                        last edited by movisman

                        @muldjord

                        Hi,

                        I think I solved the problem with Amiga. My CD32 files run from *.cue files (which is the most common for this system I think), and I had a suspicion that maybe the scraper wasn't supporting them.

                        So I added *.cue to an Amiga specific config.ini, and then scraped again from the local db.

                        It pulled the file straight in!

                        As a request, can you add *.cue support to Amiga possibly? Those who run CD32 will pretty much only use .cue files to launch I reckon.

                        Cheers!

                        1 Reply Last reply Reply Quote 0
                        • M
                          movisman
                          last edited by movisman

                          @muldjord

                          One other question from me, do you think you might ever incorporate a function into the scraper where it is able to 'detect' the region?

                          I used a beta version of "Skraper" on my previous installation, which uses screenscraper.fr only, and this has a tick option in the GUI called "Use ROM region first (when available)", and then it has a free text priority list underneath, where you can put US, EU, etc - which I guess it uses if it cannot work out the ROM region itself.

                          When this option is ticked, any US roms I have get US artwork and naming, and EU ones get EU artwork and naming, etc, etc. It happens automatically, and seems generally very accurate.

                          I don't know how it works it out, maybe it uses information from the existing filename or something, not sure. But either way, is this something you could potentially incorporate as an additional switch, rather than having a fixed priority list?

                          I could use Skraper again, but I like the compositor and artwork style on yours better and prefer scraping from the command line as Skraper doesn't really work on Linux yet. And the fact I can use an SHA1 value as a one time override is excellent.

                          Cheers!

                          mituM 1 Reply Last reply Reply Quote 0
                          • mituM
                            mitu Global Moderator @movisman
                            last edited by

                            @movisman Skyscraper already has a switch to choose the region --region and even the language (--lang) for the sources that supports it.

                            M 1 Reply Last reply Reply Quote 0
                            • M
                              movisman @mitu
                              last edited by

                              @mitu

                              Hi there,
                              No worries, but that wasn't really what I was referring to - I know about the switch to set a region and have used it to set to US first, but Skraper has an option to set the region by rom somehow - before using a priority. It means my mixture of EU and US games get picked up correctly on a single scrape, whereas with skyscraper you have to explicitly set a region for the scrape or leave as the default priority (EU first). So it is an all or nothing approach, which doesn't work so well if you have say, a megadrive folder with both EU and US ROMs.

                              Does that make sense?

                              mituM 1 Reply Last reply Reply Quote 0
                              • mituM
                                mitu Global Moderator @movisman
                                last edited by

                                @movisman said in Versatile C++ game scraper: Skyscraper:

                                Does that make sense?

                                I understood your request in the first place, I didn't know if you were aware of the command line switches to choose region/language.

                                M 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord
                                  last edited by muldjord

                                  @movisman I don't currently plan to change the existing region and language functionality. Currently it has a prioritized list it tries. If you set one manually it will move that region og language to the top of that list.

                                  I have added .cue to the list of default supported files for Amiga.

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    movisman @mitu
                                    last edited by

                                    @mitu

                                    No worries, I thought there was some confusion, but thanks for the heads up, and as always, thanks for helping.

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      movisman @muldjord
                                      last edited by

                                      @muldjord

                                      Perfect, thanks for adding .cue to the list of files.

                                      No worries about the region functionality, if that won't change in the foreseeable future, I could always pop the EU games in another directory, scrape as US, move the EU games back, then scrape again as EU but use the switch to skip the previously scraped files. That should work...

                                      I wonder how Skraper does it, perhaps it looks at the filename and makes a judgement. I have no idea but it does work quite well.

                                      Cheers for all of the help so far.

                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @movisman
                                        last edited by muldjord

                                        @movisman Can you give me examples of your filenames? I've decided to implement the region override (after a bit of thinking I realized this was a pretty easy thing to implement). But I need filename examples of how they are designated. For instance, are they called "Gamename (Europe).zip"? or maybe "Gamename (eu).zip"?

                                        EDIT: The implementation is done now. I've implemented it so it looks for "europe", "usa", "japan", and "world" in the filename, but only if it comes after the first "(" and the user hasn't manually set the "--region" flag. It will then add that region to the top of the priority list automatically.

                                        EDIT2: @movisman Skyscraper 2.8.2 now tested and released, please update and test out the region detection. You should get the artwork from the region that's in the filename if they are from the above mentioned regions.

                                        1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord
                                          last edited by muldjord

                                          Skyscraper version 2.8.2 released: https://github.com/muldjord/skyscraper

                                          • Now autodetects region from filename and prioritizes it when scraping with "screenscraper" unless user has forced region with "--region" option (thank you to "movisman" for suggesting this)
                                          • Fixed issue where "wheel" artwork would not be fetched for specified region even though it exists
                                          • Added ".cue" file extension to Amiga
                                          M 2 Replies Last reply Reply Quote 1
                                          • M
                                            movisman @muldjord
                                            last edited by

                                            @muldjord

                                            Amazing!! Thank you for implementing this!
                                            I have been out this evening, but will give it a go shortly and let you know.

                                            You are correct, most of the files are (Europe), (USA), (Japan), and (World). Some are a combination, eg. (US, Europe) - I guess in this instance it should scrape the US one.

                                            Will let you know shortly how it goes. I'll test on one of the folders which has these brackets on every file (megadrive + snes).

                                            Cheers

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.