• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.6m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mo418 @mitu
    last edited by mo418 4 Jun 2019, 18:42 6 Apr 2019, 16:39

    @mitu

    Yes I will later today, but as you can see in the picture it says that my user is found and that I’m allowed 1 thread.

    Edit : Hum, where’s the config.ini file on a Lubuntu install? Can’t seem to find it.

    Thanks

    M 1 Reply Last reply 6 Apr 2019, 17:45 Reply Quote 0
    • M
      mitu Global Moderator @mo418
      last edited by 6 Apr 2019, 17:45

      @mo418 said in Versatile C++ game scraper: Skyscraper:

      Edit : Hum, where’s the config.ini file on a Lubuntu install? Can’t seem to find it.

      The same place where is on every install - $HOME/.skyscraper. How did you add your user/pass to the config.ini if you didn't edit it ?

      M 1 Reply Last reply 6 Apr 2019, 21:29 Reply Quote 0
      • M
        mo418 @mitu
        last edited by mo418 4 Jul 2019, 00:59 6 Apr 2019, 21:29

        @mitu
        Using the skyscraper advanced option, from the Retropie setup script in terminal. No edit in Lubuntu directly.

        Edit: Hidden folders were not showing. Now I see everything

        Screenshot.png

        M 1 Reply Last reply 7 Apr 2019, 03:05 Reply Quote 0
        • M
          mitu Global Moderator @mo418
          last edited by 7 Apr 2019, 03:05

          @mo418 Run this command from a command line terminal:

          sudo chown -R js $HOME/.skyscraper/*
          

          then re-try the scraping, I suspect there's a permissions problem.

          M 1 Reply Last reply 7 Apr 2019, 11:55 Reply Quote 0
          • M
            mo418 @mitu
            last edited by 7 Apr 2019, 11:55

            @mitu

            Yup, success for creating folders! Scraping at the moment.

            Thanks a lot.

            1 Reply Last reply Reply Quote 0
            • V
              victordg
              last edited by 8 Apr 2019, 08:20

              First I'd like to thank you @muldjord for this amazing tool, I love the way collections look tidy and neat with the same artwork styling, and how customizable is everything. I take it must be hard to deal with improvements and bugs, so my sincere admiration to you.

              I'm having problems scraping certain SNES roms from screenscraper.fr because they have added those as a different system (Nintendo Power). Same occurs with Satellaview for example, if one decides to have their collection as normal SNES games they won't be scraped. I've tried adding the --query "md5=..." or "sha1=..." but it doesn't work for those (NP or Satellaview), although it was useful for other difficult scrapes.
              After reading the documentation I don't see a way to scrape those roms from Skyscraper (using the latest as of now, 3.1.4). Any workarounds or hacks to have those parsed?

              Thanks very much in advance!

              M 1 Reply Last reply 8 Apr 2019, 15:14 Reply Quote 0
              • M
                muldjord @victordg
                last edited by muldjord 4 Aug 2019, 16:18 8 Apr 2019, 15:14

                @victordg As it turns out, there are actually quite a few of these aliases. I've added the following:

                    aliases.append("nintendo power");
                    aliases.append("satellaview");
                    aliases.append("sufami turbo");
                    aliases.append("snes - super mario world hacks");
                    aliases.append("super nintendo msu-1");
                

                which will be available in 3.1.5 when it's released (no eta currently).

                If you want to add it in your current version you can just edit the platforms.cpp source code file and add the above lines to the aliases function. Just remember to compile it and install it afterwards, otherwise it'll just keep using your current version.

                EDIT: Actually I've decided to remove the platform check altogether when scraping from ScreenScraper. There's no real need for it as ScreenScraper always returns correct results since it checksum based.

                Q V 2 Replies Last reply 8 Apr 2019, 15:44 Reply Quote 0
                • Q
                  quicksilver @muldjord
                  last edited by 8 Apr 2019, 15:44

                  @muldjord how will removing the platform check affect games that are found through name match and have multiple platform entries? Games launched by .sh or .conf files for example.

                  M 1 Reply Last reply 8 Apr 2019, 20:50 Reply Quote 0
                  • V
                    victordg @muldjord
                    last edited by victordg 4 Aug 2019, 21:26 8 Apr 2019, 18:07

                    @muldjord Nice to know! Both ideas are perfect to me, actually I was also kind of expecting that the platform would be ignored when expliciting the rom hash with --query, so I totally agree to your approach, at least using screenscraper.fr there is no real need to match the platform since hashes are included for every game :) I'm too impatient though so I'll recompile with those aliases to see the NPs and Satellaviews right now :D

                    Thumbs up for you @muldjord, Skyscraper is getting better and better.

                    EDIT: Just to confirm that now it works as expected. Thanks again @muldjord!

                    1 Reply Last reply Reply Quote 1
                    • M
                      muldjord @quicksilver
                      last edited by muldjord 4 Aug 2019, 21:54 8 Apr 2019, 20:50

                      @quicksilver said in Versatile C++ game scraper: Skyscraper:

                      @muldjord how will removing the platform check affect games that are found through name match and have multiple platform entries? Games launched by .sh or .conf files for example.

                      There is a potential chance that a search by romnom (file name) will return an entry for a different platform. And if this entry is "faulty" it will no longer be filtered out by the platform check. But I don't think it will anymore as I am complying exactly with the demands ScreenScraper have for their API calls these days. And that requires a parent system id to be part of the query. This is always prepended to the query, even when using --query. So any false positives should be avoided that way. Even with the platform check disabled.

                      1 Reply Last reply Reply Quote 1
                      • M
                        muldjord
                        last edited by 5 Jul 2019, 20:58

                        Skyscraper 3.1.5 coming up in a few days. I had some leftover changes that are actually pretty cool, some minor additions as requested by users and some platform additions. And I've also added a --cache report:missing=<RESOURCE> option.

                        I don't have much time for Skyscraper these days, so I want this stuff out there. I just need to do some testing over the coming days to make sure everything is still a-okay.

                        R 1 Reply Last reply 5 Jul 2019, 22:25 Reply Quote 3
                        • R
                          rkoster @muldjord
                          last edited by 5 Jul 2019, 22:25

                          @muldjord Thanks for adding those two I submitted the ticket about. :)

                          Github/helper scripts * Creativision/Arcadia/Astrocade guide * Amiga guide * Atari 8bit guide

                          M 1 Reply Last reply 6 Jul 2019, 07:50 Reply Quote 1
                          • M
                            muldjord @rkoster
                            last edited by 6 Jul 2019, 07:50

                            @rkoster said in Versatile C++ game scraper: Skyscraper:

                            @muldjord Thanks for adding those two I submitted the ticket about. :)

                            You're welcome :)

                            1 Reply Last reply Reply Quote 0
                            • M
                              muldjord
                              last edited by 8 Jul 2019, 12:46

                              Skyscraper 3.2.0 released: https://github.com/muldjord/skyscraper

                              • Improved output of edit mode to show queue size and initial prioritized resources
                              • Now clearly shows missing resources when adding resources in editing mode
                              • Made '--cache vacuum' more verbose at verbosity level 2 or higher
                              • Removed platform check for screenscraper module as it wasn't really necessary
                              • Added 'videoSizeLimit' config variable (Thank you to 'silverfr0st' for suggesting this)
                              • Now also processes brackets for skipped games (Thanks to 'hexive' for pointing this out)
                              • Added 'arcadia' and 'astrocde' MESS platforms (Thank you to 'raphkoster' for suggesting these)
                              • Added '--cache report:missing=<GROUP or RESOURCE1,2,...>' option (Thank you to JaymesRS for suggesting this)
                              • Added '--fromfile' option allowing you to work on a subset of files taken from a file. Compatible with files created using the '--cache report' option
                              • MobyGames module now includes 'platform' id query when relevant. Should improve result accuracy
                              • Added region support for MobyGames cover artwork
                              • Added A LOT of regions to region autodetection in general
                              • Now no longer creates media folders when in resource gathering mode
                              • Moved help texts for '--cache' options to '--cache help'

                              Lots of interesting stuff in this release. The --cache edit function has been improved a bit to better show what resources are missing on each rom. I've also added a nifty --fromfile <FILENAME> option which allows you to add filenames, one per line, into a file and use it with this option. Skyscraper will then work on those files only. This brings me to the new --cache report:missing option which allows you to create file reports containing filenames of all files missing certain resources. When those are created you can then combine the edit option with the fromfile option and add the missing resources like so --cache edit --fromfile /home/pi/.skyscraper/reports/reportfile.txt.

                              I've also improved region autodetection quite a bit. It will now detect pretty much all relevant regions and / or countries from the rom filename. Also added region support to mobygames covers!

                              As for minor stuff, some new platforms were added and some requested features aswell. Check the list above for all of it.

                              Let me know if you run into problems.

                              Have fun!

                              A Bored NinjaA 1 Reply Last reply 8 Jul 2019, 14:31 Reply Quote 4
                              • A Bored NinjaA
                                A Bored Ninja @muldjord
                                last edited by 8 Jul 2019, 14:31

                                @muldjord said in Versatile C++ game scraper: Skyscraper:

                                new platforms were a

                                @muldjord Thank you!

                                I'm excited to now be able to scrape my PC Engine CD roms, as I was getting a "Platform not set" error before.

                                Keep up the good work!

                                M 1 Reply Last reply 8 Jul 2019, 15:05 Reply Quote 0
                                • M
                                  muldjord @A Bored Ninja
                                  last edited by muldjord 7 Aug 2019, 16:05 8 Jul 2019, 15:05

                                  @A-Bored-Ninja This has not been addressed in this release. Otherwise it would have been part of the release notes, sorry. Or have you found some other way of doing it?

                                  A Bored NinjaA 1 Reply Last reply 8 Jul 2019, 16:32 Reply Quote 0
                                  • A Bored NinjaA
                                    A Bored Ninja @muldjord
                                    last edited by 8 Jul 2019, 16:32

                                    @muldjord I assumed that when you took the platform option out of screenscraper it would sort of resolve itself for me.

                                    Since I have a screenscraper account and get six threads that's always what I use.

                                    What I not benefit from the screenscraper platform removal?

                                    M 1 Reply Last reply 8 Jul 2019, 16:57 Reply Quote 0
                                    • M
                                      muldjord @A Bored Ninja
                                      last edited by muldjord 7 Aug 2019, 17:58 8 Jul 2019, 16:57

                                      @A-Bored-Ninja Ah, yes, that is correct. I thought you were talking about the "platform alias" option I have under consideration which is not yet implemented (and might not be as it a pretty big change for little gain).

                                      1 Reply Last reply Reply Quote 1
                                      • O
                                        OldSchool
                                        last edited by 8 Jul 2019, 17:14

                                        Hi @muldjord, is it possible to "clean up" the media? Let's say I had a bunch of games that were scraped and looking good, but some time later they were deleted for being dupes, or I just didn't want them anymore. Is there an easy way to delete the media associated with those now deleted games without clearing all of the media folders and cache, and scraping everything again? And can the cache be emptied if the games already have their media in the media folders after generating the gsmelist? Because with the cache and media folders combined I believe there's a duplicate of every media file, correct? Thanks for the help.

                                        M 1 Reply Last reply 8 Jul 2019, 17:22 Reply Quote 0
                                        • M
                                          muldjord @OldSchool
                                          last edited by muldjord 7 Aug 2019, 18:53 8 Jul 2019, 17:22

                                          @OldSchool Yes, but only from the cache. Please read the documentation for the --cache command here

                                          Particularly the --cache vacuum option.

                                          Concerning the ES media folder Skyscraper can't and won't keep track of what is in it. Remember that Skyscraper supports multiple frontends. Keeping track of all that is mindbogglingly difficult. But as long as you haven't made manual changes and kept your Skyscraper cache, you can just delete the media folder and regenerate the gamelist and media files. Then it will be "clean" again.

                                          O 1 Reply Last reply 8 Jul 2019, 18:05 Reply Quote 1
                                          • First post
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            [[user:consent.lead]]
                                            [[user:consent.not_received]]