• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.6m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Q
    quicksilver @muldjord
    last edited by 4 Mar 2019, 04:09

    @muldjord the refresh option took care of my issue. Once again thank you for your help!

    1 Reply Last reply Reply Quote 0
    • Z
      ZXDunny
      last edited by ZXDunny 3 Apr 2019, 15:59 4 Mar 2019, 15:56

      @muldjord - a quick question; can the "dbs" folder be relocated anywhere else? My uSD card is full and nothing can be scraped any further despite all my ROMs and skyscraper cache being located on the external 2TB HDD. The dbs folder contains 2.3GB of data, by all accounts.

      Can it be relocated? I'd empty it but I'm not sure what effect that would have, and it will be a PITA to have to do that every time I want to scrape a collection.

      RetroPie is running from a 32GB uSD card that I cannot access physically, hence I've attached the HDD to the board so that ~/pi/RetroPie is now accessed from there. I edited config to move the cache there also, but the dbs folder doesn't seem to be configurable?

      Edit: Even though my config has been edited to move the cache to ~/pi/RetroPie/skyscraper/cache, Skyscraper is still using the original location on my uSD card :(

      M 1 Reply Last reply 4 Mar 2019, 16:55 Reply Quote 0
      • M
        mitu Global Moderator @ZXDunny
        last edited by 4 Mar 2019, 16:55

        @ZXDunny Can you post your config.ini file and your Skyscraper version ? Since 3.x the dbs folder is named cache, so you might be using an older version.

        1 Reply Last reply Reply Quote 0
        • Q
          quicksilver
          last edited by 4 Mar 2019, 19:30

          How does Skyscraper handle/display arcade roms? Obviously there is no "box art". Anyone willing to post a sample? Curious before I start scraping 1500 roms.

          M 1 Reply Last reply 4 Mar 2019, 19:41 Reply Quote 0
          • M
            mitu Global Moderator @quicksilver
            last edited by 4 Mar 2019, 19:41

            @quicksilver Here are some samples:

            ef7312aa-168f-42a8-95d2-436cc4fcb8b3-image.png

            ac81a905-80ee-4fc9-bff8-611939d6b88e-image.png

            dc662fd7-ffcd-4efb-bc0a-236537d4d875-image.png

            Note that I only used ScreenScraper for scraping - they do have boxart for some of the Arcade titles.

            Q 1 Reply Last reply 4 Mar 2019, 19:52 Reply Quote 1
            • Q
              quicksilver @mitu
              last edited by 4 Mar 2019, 19:52

              @mitu Thank you! This is exactly what I was looking for. I have had good results with screenscraper for console scraping, so will probably stick with it for arcade then. I havent gotten very accurate results with the gamesdb (probably because it searches based on rom name).

              1 Reply Last reply Reply Quote 0
              • H
                Halvhjearne @muldjord
                last edited by 4 Mar 2019, 20:13

                @muldjord

                i was just scraping my gba games and i noticed i got a lot of errors like this:

                ScreenScraper APIv2 returned invalid XML for the following query:
                romnom=Dragon%20Ball%20Z%20-%20Buu_s%20Fury%20%28USA%29.gba&crc=1C1707F&md5=3A74FCE97F1EA2B28C2A50EC3DF0ACEE&sha1=F1C4B07554D2A3B1AD2F325307051E775CE68087&romtaille=8388608
                

                it used to scrape all my gba games perfectly, so i tried to update to latest version, but it still persists resulting in a fail rate of about 6-7% of the games, where it used to find basicly every game in earlier versions of skyscraper

                M 1 Reply Last reply 4 Mar 2019, 20:15 Reply Quote 0
                • M
                  mitu Global Moderator @Halvhjearne
                  last edited by mitu 3 Apr 2019, 20:15 4 Mar 2019, 20:15

                  @Halvhjearne You may be running into https://retropie.org.uk/forum/topic/11826/versatile-c-game-scraper-skyscraper/1123. It's not a Skyscraper problem.

                  H 1 Reply Last reply 4 Mar 2019, 20:18 Reply Quote 0
                  • H
                    Halvhjearne @mitu
                    last edited by 4 Mar 2019, 20:18

                    @mitu said in Versatile C++ game scraper: Skyscraper:

                    @Halvhjearne You may be running into https://retropie.org.uk/forum/topic/11826/versatile-c-game-scraper-skyscraper/1123. It's not a Skyscraper problem.

                    im a registered donor and skyscraper recognizes i have 6 threads available ...

                    M 1 Reply Last reply 4 Mar 2019, 20:21 Reply Quote 0
                    • M
                      mitu Global Moderator @Halvhjearne
                      last edited by 4 Mar 2019, 20:21

                      @Halvhjearne The screenscraper site dashboard shows their CPU at 294% right now, so I wouldn't be surprised if their system overload would result in the errors you're getting.

                      H 1 Reply Last reply 4 Mar 2019, 20:22 Reply Quote 0
                      • H
                        Halvhjearne @mitu
                        last edited by 4 Mar 2019, 20:22

                        @mitu
                        scraping again without the --refresh seems to work fine ...

                        M 1 Reply Last reply 4 Mar 2019, 20:26 Reply Quote 0
                        • M
                          mitu Global Moderator @Halvhjearne
                          last edited by 4 Mar 2019, 20:26

                          @Halvhjearne Why are you using --refresh by default ? It forces Skyscraper to re-fetch the data when you already have it in cache.

                          H 1 Reply Last reply 4 Mar 2019, 20:36 Reply Quote 0
                          • H
                            Halvhjearne @mitu
                            last edited by 4 Mar 2019, 20:36

                            @mitu
                            im not using it by default, but i do run skyscraper sometimes with the --refresh to see if there is any missing artwork available ...

                            1 Reply Last reply Reply Quote 0
                            • M
                              mitu Global Moderator
                              last edited by mitu 3 Apr 2019, 21:10 4 Mar 2019, 21:09

                              That's not what --refresh should be used for. If you don't have the corresponding artwork in the cache (marquee, cover, screenshot, video), Skyscraper would try and get it, so you don't have to add --refresh to get missing artwork. @muldjord can correct me if I'm wrong, but forcing --refresh just negates any advantage of having a cache.

                              M 1 Reply Last reply 4 Mar 2019, 21:26 Reply Quote 0
                              • M
                                muldjord @mitu
                                last edited by muldjord 3 Apr 2019, 21:27 4 Mar 2019, 21:26

                                @mitu said in Versatile C++ game scraper: Skyscraper:

                                That's not what --refresh should be used for. If you don't have the corresponding artwork in the cache (marquee, cover, screenshot, video), Skyscraper would try and get it, so you don't have to add --refresh to get missing artwork. @muldjord can correct me if I'm wrong, but forcing --refresh just negates any advantage of having a cache.

                                This is true, but there is one issue where it helps. When the screenscraper servers are overloaded, some of the media might not be scraped for a certain game, even though the textual data is (each artwork resource is a separate request that might fail due to the high load). So in those (hopefully rare cases, especially for registered users) they will need to use --refresh to grab them.

                                With that said, I would always recommend only scraping a few roms with --refreshenabled. Instead, just add those few files to command line to grab the missing data. That is much better than scraping everything again.

                                Q 1 Reply Last reply 4 Mar 2019, 21:42 Reply Quote 2
                                • Q
                                  quicksilver @muldjord
                                  last edited by 4 Mar 2019, 21:42

                                  @muldjord Is there a way to use the --refresh function only on roms that are detected as missing some media? I.e. screenshot, box art or description are missing? Im not sure if what I am asking makes sense. Unfortunately for me I scraped hundreds of roms before realizing that credentials for screenscraper made such a difference. Now there are tons of roms that are missing some media. Going through and checking them one by one will be a huge task.

                                  M 1 Reply Last reply 4 Mar 2019, 21:56 Reply Quote 0
                                  • M
                                    muldjord @quicksilver
                                    last edited by muldjord 3 Apr 2019, 21:57 4 Mar 2019, 21:56

                                    @quicksilver said in Versatile C++ game scraper: Skyscraper:

                                    @muldjord Is there a way to use the --refresh function only on roms that are detected as missing some media? I.e. screenshot, box art or description are missing? Im not sure if what I am asking makes sense. Unfortunately for me I scraped hundreds of roms before realizing that credentials for screenscraper made such a difference. Now there are tons of roms that are missing some media. Going through and checking them one by one will be a huge task.

                                    What you are asking makes perfect sense, but it is not possible to do this at the moment I'm afraid. I had not anticipated so many requests to be rejected, so this is actually quite a big problem at the moment. And it's a bit of a bad circle to get into. People will notice media being missing, and then they will start using refresh, which again puts even more load on the source servers and so on...

                                    I am wondering if screenscraper has any sort of server caching installed on their service. This could potentially alleviate the problem. I've asked on their forum.

                                    Q 1 Reply Last reply 6 Mar 2019, 20:07 Reply Quote 1
                                    • Q
                                      quicksilver @muldjord
                                      last edited by 6 Mar 2019, 20:07

                                      @muldjord So I'm slowly but surely getting all of ROMs scraped using skyscraper and the results are simply amazing. I should have done this long ago!

                                      I just wanted to thank you for such a great scraper! I have read back through a lot of the posts on this thread and your commitment to creating a great program clearly shows. I hope you take great pride in this! The next time I'm at my computer I will be sure to send a tip your way. Thanks again for all your hard work!

                                      M 1 Reply Last reply 6 Mar 2019, 20:33 Reply Quote 1
                                      • M
                                        muldjord @quicksilver
                                        last edited by 6 Mar 2019, 20:33

                                        @quicksilver said in Versatile C++ game scraper: Skyscraper:

                                        @muldjord So I'm slowly but surely getting all of ROMs scraped using skyscraper and the results are simply amazing. I should have done this long ago!

                                        I just wanted to thank you for such a great scraper! I have read back through a lot of the posts on this thread and your commitment to creating a great program clearly shows. I hope you take great pride in this! The next time I'm at my computer I will be sure to send a tip your way. Thanks again for all your hard work!

                                        You are very welcome and thanks! Skyscraper is sort of a passion project of mine and I probably spend more time on it than I should. :D

                                        Q 1 Reply Last reply 6 Mar 2019, 20:47 Reply Quote 1
                                        • Q
                                          quicksilver @muldjord
                                          last edited by quicksilver 3 Jun 2019, 21:10 6 Mar 2019, 20:47

                                          @muldjord sorry to be a nuisance again so soon, but I'm having issues generating Gameslist for "ports". Using the GUI I was unable to scrape any games because the platform is set as "ports" which the sources don't recognize as a valid platform. So I scraped them all from the commandline using pc as the platform. However, when I go to generate the Gameslist it just says the games cannot be found.

                                          Edit: I see scummvm set to look for PC and Amiga matches. Perhaps ports can be set to look at pc?

                                          M 1 Reply Last reply 6 Mar 2019, 21:14 Reply Quote 0
                                          1149 out of 1594
                                          • First post
                                            1149/1594
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received