• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.6m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    steptoe @muldjord
    last edited by 13 Jan 2020, 19:55

    @muldjord Thanks for the extra efforts for us windows users

    1 Reply Last reply Reply Quote 1
    • Q
      quicksilver
      last edited by 22 Jan 2020, 22:00

      Is it possible to adjust settings in the artwork.xml for certain systems only? For example if I wanted only screenshots displayed, but just for arcade?

      I searched the documentation but either couldn't find anything or missed it.

      M 1 Reply Last reply 22 Jan 2020, 22:19 Reply Quote 0
      • M
        muldjord @quicksilver
        last edited by muldjord 22 Jan 2020, 22:19

        @quicksilver Yes, just create individual artwork.xml files for the platforms and enter them into the config.ini's platform sections. For instance like so:

        [amiga]
        artworkXml="artwork-amiga.xml"
        

        You can also do it similarly for different frontends like so:

        [pegasus]
        artworkXml="artwork-pegasus.xml"
        

        It's documented here

        Q 2 Replies Last reply 22 Jan 2020, 22:39 Reply Quote 1
        • Q
          quicksilver @muldjord
          last edited by 22 Jan 2020, 22:39

          @muldjord thank you so much, knew it had to be there somewhere. Skyscraper can do it all!

          1 Reply Last reply Reply Quote 1
          • Q
            quicksilver @muldjord
            last edited by quicksilver 23 Jan 2020, 00:14

            @muldjord Sorry to bug you again, but what I am trying to accomplish is to have the marquees show up in a specific area in the theme I am using but not composited with the rest of the artwork. I can get the marquees to show up in the theme but it also shows up in the composite image too. How can get the marquee to show up in the correct place in the theme but not also display in the artwork composite?

            EDIT: I figured it out. I added <output type="marquee" resource="marquee"/> to the .xml

            sorry for pinging you!

            1 Reply Last reply Reply Quote 1
            • M
              muldjord
              last edited by 4 Feb 2020, 18:37

              Skyscraper 3.4.0 released: https://github.com/muldjord/skyscraper

              • Disabled config.ini migration as most people should be migrated now
              • Added 'launcher' option to 'simple mode' when using 'pegasus' frontend
              • Added 'excludeFiles' config option that allows excluding certain files when scraping (Thank you to 'timothybrown' for suggesting this)
              • Added 'includeFiles' config option that allows only including certain files when scraping
              • Added '--excludefiles' cli option that allows excluding certain files when scraping
              • Added '--includefiles' cli option that allows only including certain files when scraping
              • If 'noresize' is set all images are now saved to cache in their original format and size instead of always converting them to PNG's (Thank you to 'krkroft' for requesting this)
              • Added 'jpgQuality' config option that sets the default JPG quality (0-100) when '--noresize' is NOT set. Screenshots and images with transparency are still saved as PNG's.
              • Upped 'screenscraper' request limiter to 1.2 seconds per request to avoid 'maximum threads per minute reached' error message

              Quite a few new things made it into this release. The most prominent one is that Skyscraper now handles the --noresize option a bit differently. The option makes Skyscraper save all media "as is" from the sources, without resizing them. But in older versions I always converted all artwork resources to PNG when saving them, which caused the 4000x3000 cover artwork for some games to take up huge amounts of space. So when user 'krcroft' requested that the artwork would use the format that the sources served to me instead, I thought about that for half a year... hrm... Anyways, I completely rewrote the artwork acquiring code and internal structures so it will now, in fact, save the artwork exactly as is when using the --noresize or cacheResize="false" options. While doing this I also optimized how Skyscraper works by default, so the sizes of any artwork will now be a bit bigger and take up less space, pretty much without any loss of quality. So it's a win-win on all fronts! You can even save more space by setting and lowering the jpgQuality="95"config.ini option.

              Another quite interesting feature is the --excludefiles / excludeFiles= option and companion --includefiles/ includeFiles= option. These will allow you to mask out any files within the scraping scope. So if you have a bunch of files with [BIOS] in their name, you can now mask these out but entering excludeFiles="*[BIOS]*" in config.ini or similar for the CLI option. This can be set on several levels, check the documentation for details.

              Lastly, I rarely scrape a huge number of files these days as I basically have all of my data in my cache now. So it was only today, when testing this release, I stumbled upon an error when scraping with the screenscraper module. Turns out my 60 request per minut limiter was too close to the edge and sometimes ScreenScraper would actually give back a "too many requests" error. So I've upped the limiter a bit to 1.2 seconds per request per thread. This fixed the issue. So if you've been having json errors from screenscraper, please try it again.

              As I have rewritten quite a lot of code for this release I've also tested it quite thoroughly. I have not found any errors. But software has bugs, so please let me know if this version gives you any problems.

              1 Reply Last reply Reply Quote 4
              • C
                Clyde
                last edited by 5 Feb 2020, 17:22

                Thank you so much for your ongoing effort! 👍

                1 Reply Last reply Reply Quote 3
                • M
                  muldjord
                  last edited by 8 Feb 2020, 11:12

                  Is anyone else having trouble scraping through the screenscraper module right now? I just keep getting a maximum threads allowed already used : 5/5 and I keep getting libpng errors... Something is going on. Please let me know if you are also seeing these issues.

                  Q 1 Reply Last reply 8 Feb 2020, 12:44 Reply Quote 0
                  • Q
                    quicksilver @muldjord
                    last edited by 8 Feb 2020, 12:44

                    @muldjord I scraped some PSP ROMs about 6 hours ago and didn't have any issues. So if there is a problem it just started recently.

                    1 Reply Last reply Reply Quote 1
                    • M
                      muldjord
                      last edited by muldjord 2 Nov 2020, 22:05 8 Feb 2020, 13:05

                      Thanks, yeah, I just tried it again and now it seems to be working... Oh well. :)

                      EDIT: Asked on their Discord and turned out it was a bug in the API that they've now fixed.

                      1 Reply Last reply Reply Quote 0
                      • M
                        muldjord
                        last edited by 11 Feb 2020, 17:44

                        Skyscraper 3.4.1 released: https://github.com/muldjord/skyscraper

                        • Further optimized artwork space requirements. Now checks if original takes up less space than resized artwork, then forces use of original for those cases
                        • The 'thegamesdb' module now also supports wheel and marquee for the games that have them (Thank you to 'tv21' for pointing this out)
                        • Updated developer and publisher json list for 'thegamesdb'

                        thegamesdb now supports retrieving wheel and marquee artwork resource types. And I optimized the artwork resource export pipeline a bit further. If you scraped with 3.4.0 there's no need to redo it, it's only a small difference.

                        1 Reply Last reply Reply Quote 3
                        • ?
                          A Former User
                          last edited by 1 Mar 2020, 20:46

                          Any chance you can updated the Windows version to 3.4.1? Unless you have an update in the pipeline.

                          M 1 Reply Last reply 2 Mar 2020, 18:44 Reply Quote 0
                          • M
                            muldjord @A Former User
                            last edited by 2 Mar 2020, 18:44

                            @LiveFastCyYoung Sure, I've updated it to the latest 3.4.2 now. Get it here: http://www.muldjord.com/downloads/Skyscraper_3.4.2_unsupported_win_version.zip

                            ? 1 Reply Last reply 3 Mar 2020, 01:05 Reply Quote 0
                            • ?
                              A Former User @muldjord
                              last edited by 3 Mar 2020, 01:05

                              @muldjord Much appreciated!

                              1 Reply Last reply Reply Quote 1
                              • M
                                muldjord
                                last edited by 14 Mar 2020, 18:29

                                Skyscraper 3.4.3 released: https://github.com/muldjord/skyscraper

                                • Implemented workaround to incorrectly formatted JSON returned from 'screenscraper' when checking user credentials
                                • All arcade platforms now use 'flyer' from 'screenscraper' for cover artwork instead of 'box-2D'

                                There have been issues logging in to screenscraper lately. This stems from a bug in their code which formats the returned JSON incorrectly after checking the credentials. This leads to a parse error in the Skyscraper JSON parser which then lead to a failed login. I've implemented a workaround that fixes the JSON before parsing it. I have reported the bug to them, I hope they fix it soon so I can remove this workaround again.

                                User @aidy80-s suggested I use the flyer artwork from screenscraperfor the arcade platforms. This was a brilliant idea, so this has now been implemented as well.

                                A 1 Reply Last reply 3 Apr 2020, 16:10 Reply Quote 4
                                • ?
                                  A Former User
                                  last edited by 16 Mar 2020, 01:36

                                  Have you ever looked into adding the LaunchBox DB as a scraping source?

                                  https://gamesdb.launchbox-app.com/

                                  M 1 Reply Last reply 16 Mar 2020, 10:22 Reply Quote 0
                                  • M
                                    muldjord @A Former User
                                    last edited by 16 Mar 2020, 10:22

                                    @LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

                                    Have you ever looked into adding the LaunchBox DB as a scraping source?

                                    https://gamesdb.launchbox-app.com/

                                    Yes, several times actually. I can't remember exactly why, but as I recall it is not an open API like other sources use.

                                    1 Reply Last reply Reply Quote 0
                                    • Q
                                      quicksilver
                                      last edited by quicksilver 24 Mar 2020, 14:46

                                      Certain files get scraped as ZZZnotagame instead of just ignored (especially +StartDOSbox.sh) Its really annoying to have to manually rename these after scraping. Is there anyway to avoid this?

                                      M 1 Reply Last reply 24 Mar 2020, 15:27 Reply Quote 0
                                      • M
                                        muldjord @quicksilver
                                        last edited by 24 Mar 2020, 15:27

                                        @quicksilver Hmm, I already have a filter to avoid these, but maybe some of them goes under the radar. Can you provide a specific filename for an entry that returns this?

                                        Q 1 Reply Last reply 24 Mar 2020, 15:39 Reply Quote 0
                                        • Q
                                          quicksilver @muldjord
                                          last edited by quicksilver 24 Mar 2020, 15:39

                                          @muldjord Ok cool, if that is the case let me try doing a cache refresh and then rescrape first to see if its fixed now and I'll report back.

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            [[user:consent.lead]]
                                            [[user:consent.not_received]]