• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.6m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    muldjord @quicksilver
    last edited by 9 Mar 2019, 21:15

    @quicksilver said in Versatile C++ game scraper: Skyscraper:

    When scraping a game that won't be able to be found by it's checksum, Screenscraper will attempt to match by exact rom name from what I understand. However if there is more than one game with that exact title it will fail to scrape. I have a number of .sh and .conf files that cannot be found by checksum but fail based on ROM name because there are many versions of the game in screenscraper's database (quake II for example). Is there anyway to work around this?

    Not that I know of. This is a small limitation in how their database handles things. I do believe it only searches on the platform in question though, so this shouldn't happen too often as the platform id is also send in the requests.

    Q 1 Reply Last reply 9 Mar 2019, 23:56 Reply Quote 0
    • M
      muldjord @Silent
      last edited by muldjord 3 Sept 2019, 21:39 9 Mar 2019, 21:15

      @Silent Awesome, I'll look into it and include it in the next release. Thanks.

      EDIT:

      • Only igdb has a platform for Steam related platform called steamos. I have added that to pc and steam but it is only relevant when scraping with igdb
      • steamplatform added with all relevant aliases including steamos
      • ps2 platform added with all relevant aliases
      • *.ml added for all platforms

      I'll leave out ps3 and x360 for now. I don't want to spam platforms all over the place just for the sake of it.

      1 Reply Last reply Reply Quote 1
      • Q
        quicksilver @muldjord
        last edited by 9 Mar 2019, 23:56

        @muldjord strangely the built in scraper in emulation station will find the same titles using screenscraper as the source, unless of course it searches for ROM name differently. Either way, I was able to scrape those problematic games using different sources. Again, thank you for your help. I have been promoting skyscraper every chance I get!

        M 1 Reply Last reply 10 Mar 2019, 00:18 Reply Quote 0
        • M
          muldjord @quicksilver
          last edited by 10 Mar 2019, 00:18

          @quicksilver said in Versatile C++ game scraper: Skyscraper:

          @muldjord strangely the built in scraper in emulation station will find the same titles using screenscraper as the source, unless of course it searches for ROM name differently. Either way, I was able to scrape those problematic games using different sources. Again, thank you for your help. I have been promoting skyscraper every chance I get!

          Can you give me an exact file name and platform for a game it doesn't find that the built-in scraper does find when using screenscraper as source? I would like to test this.

          Q 1 Reply Last reply 10 Mar 2019, 00:52 Reply Quote 0
          • Q
            quicksilver @muldjord
            last edited by quicksilver 3 Oct 2019, 05:01 10 Mar 2019, 00:52

            @muldjord "Oregon Trail II". It's a .conf file in my PC folder. Built in scraper will find it with screenscraper as the source but skyscraper will not. There were a couple others, but Im away from my pi and can't remember off the top of my head. But I'll check when next I'm home.

            Edit: "Myst" is another one.

            1 Reply Last reply Reply Quote 0
            • Q
              quicksilver @muldjord
              last edited by 10 Mar 2019, 05:03

              @muldjord said in Versatile C++ game scraper: Skyscraper:

              pc and ports have the exact same aliases

              I wondering if the built in scraper doesn't have multiple aliases for PC and that's why it's able to find them.

              M 1 Reply Last reply 10 Mar 2019, 05:13 Reply Quote 0
              • M
                mitu Global Moderator @quicksilver
                last edited by mitu 3 Oct 2019, 05:17 10 Mar 2019, 05:13

                @quicksilver said in Versatile C++ game scraper: Skyscraper:

                I wondering if the built in scraper doesn't have multiple aliases for PC and that's why it's able to find them.

                Nope - the internal scraper (when using ScreenScraper) only knows about pc, for ports it ignores the platform and queries by name only.

                EDIT: either that or the Ports system is considered as pc by Emulationstation.

                Q M 2 Replies Last reply 10 Mar 2019, 05:19 Reply Quote 0
                • Q
                  quicksilver @mitu
                  last edited by 10 Mar 2019, 05:19

                  @mitu thanks for the confirmation. I would assume that's where the difference is then.

                  1 Reply Last reply Reply Quote 0
                  • M
                    muldjord @mitu
                    last edited by 10 Mar 2019, 07:40

                    @mitu said in Versatile C++ game scraper: Skyscraper:

                    Nope - the internal scraper (when using ScreenScraper) only knows about pc, for ports it ignores the platform and queries by name only.

                    Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

                    M 1 Reply Last reply 10 Mar 2019, 08:23 Reply Quote 0
                    • M
                      mitu Global Moderator @muldjord
                      last edited by 10 Mar 2019, 08:23

                      @muldjord said in Versatile C++ game scraper: Skyscraper:

                      Oh, that's right. It uses the new search query instead of the jeuInfos, right? Then that's the difference. I might look into using the search query one at one point.

                      No, it's still using the game info endpoint (v2 WebApi). I've looked over using the search endpoint - see my post on their forum about the search returning to many unrelated results. However, the search is slower then the game info endpoint and I think it's not worth (at the moment) the effort to switch to it - considering their server(s) are already busy. Improving the search over the regular game info endpoint is a better investment at this point (checksum, search-by-name, search-by-id).

                      1 Reply Last reply Reply Quote 1
                      • M
                        muldjord
                        last edited by 10 Mar 2019, 10:09

                        Skyscraper 3.1.2 released: https://github.com/muldjord/skyscraper

                        • Added support for Moonlight '*.ml' extension for all platforms
                        • Added 'ps2' platform for use with Moonlight and all other supported Ps2 extensions
                        • Added 'steam' platform for use with Moonlight '*.ml' files
                        • Fixed minor bug in CRC checksum calculator for 'screenscraper'
                        • Improved workarounds for ScreenScraper XML errors

                        Yet again I fixed some errors in the screenscraper module. My CRC checksum calculator sometimes returned 7 char strings. This caused issues with the replies from screenscraper, so I've made sure I check the length and prepend a "0" when this happens. I also improved some of the workaround for the XML errors screenscraper sometimes send out.
                        Lastly you should now be able to scrape moonlight files with all platforms as requested by @Silent. Pretty untested though, so let me know how this works for you.

                        S 1 Reply Last reply 10 Mar 2019, 11:18 Reply Quote 1
                        • S
                          Silent @muldjord
                          last edited by Silent 3 Oct 2019, 11:25 10 Mar 2019, 11:18

                          @muldjord I quickly tested scraping a few "roms" from screenscraper/thegamesdb/igdb and I can confirm it seems to work fine :) Thanks a lot!

                          EDIT:
                          Quick observation: I tried to scrape this game off MobyGames:
                          https://www.mobygames.com/game/spintires-mudrunner

                          MobyGames name is "Spintires: MudRunner", whereas my .ml file is named "Spintires MudRunner". If you try to search for it like this on the website, you get normal results:
                          https://www.mobygames.com/search/quick?q=Spintires+Mudrunner

                          Yet, Skyscraper couldn't find it unless I leveraged aliasMap.csv to add : to the name. API's fault or Skyscraper not filtering results well enough?

                          M 2 Replies Last reply 10 Mar 2019, 12:34 Reply Quote 1
                          • S
                            Silent
                            last edited by Silent 3 Oct 2019, 11:54 10 Mar 2019, 11:51

                            Another note: I can't find absolutely nothing for steam platform from thegamesdb scraper. My cache has entries for those, which means I used to be able to find things smoothly, but now I can't.

                            EDIT:
                            Also, looks like NVidia Gamestream added support for Forza Horizon 4:
                            https://www.mobygames.com/game/forza-horizon-4

                            Platform there shows as "Windows Apps", and looks like Skyscraper can't find it because of that either.

                            M 1 Reply Last reply 10 Mar 2019, 12:30 Reply Quote 0
                            • M
                              muldjord @Silent
                              last edited by muldjord 3 Oct 2019, 12:50 10 Mar 2019, 12:30

                              @Silent said in Versatile C++ game scraper: Skyscraper:

                              Another note: I can't find absolutely nothing for steam platform from thegamesdb scraper. My cache has entries for those, which means I used to be able to find things smoothly, but now I can't.

                              Double negative, but I assume you meant "I can't find anything". I have added the *.ml extension to be id'et by filename since they are scripts. This might have fixed it but you will have to scrape all of your ml files all over again. I just re-released 3.1.2 for this. Maybe I should start doing betas instead.

                              I've also added the "windows apps" alias to steam and pc. So please re-update to 3.1.2.

                              EDIT: Figured out why you get no results with thegamesdb. They've apparently added new platforms and I haven't updated my internal list in a while since I mostly focus on retro platforms. So I'll need to figure that out... Let me look into this.

                              1 Reply Last reply Reply Quote 0
                              • M
                                muldjord @Silent
                                last edited by 10 Mar 2019, 12:34

                                This post is deleted!
                                1 Reply Last reply Reply Quote 0
                                • M
                                  muldjord @Silent
                                  last edited by 10 Mar 2019, 12:40

                                  @Silent said in Versatile C++ game scraper: Skyscraper:

                                  @muldjord I quickly tested scraping a few "roms" from screenscraper/thegamesdb/igdb and I can confirm it seems to work fine :) Thanks a lot!

                                  EDIT:
                                  Quick observation: I tried to scrape this game off MobyGames:
                                  https://www.mobygames.com/game/spintires-mudrunner

                                  MobyGames name is "Spintires: MudRunner", whereas my .ml file is named "Spintires MudRunner". If you try to search for it like this on the website, you get normal results:
                                  https://www.mobygames.com/search/quick?q=Spintires+Mudrunner

                                  Yet, Skyscraper couldn't find it unless I leveraged aliasMap.csv to add : to the name. API's fault or Skyscraper not filtering results well enough?

                                  Seems like this is due to their API. The query Skyscraper sends for that file name is "spintires+mudrunner". This gives a result when using the website search, but not when using the API.

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    muldjord
                                    last edited by 10 Mar 2019, 13:17

                                    Ok, thegamesdb should provide more results for the steamplatform now. But I had to add aliases for the xbone and ps4 for it. It rarely returns PC results it seems, which is really odd. But not something I can change. Re-released 3.1.2 again... Please re-update.

                                    1 Reply Last reply Reply Quote 0
                                    • SgtJimmyRustlesS
                                      SgtJimmyRustles
                                      last edited by 10 Mar 2019, 14:55

                                      I love this tool, thanks for making it. Really does wonders for scraping roms.

                                      Check out /r/SBCGaming for Retropie, Recalbox, OGST (Odroid) talk and more!
                                      http://www.reddit.com/r/sbcgaming

                                      Hop on the Dicscord as well and chat about all things Retropie (or recalbox, lakka, etc)
                                      https://discord.gg/JdXc6nx

                                      Check out our Podcast!

                                      1 Reply Last reply Reply Quote 1
                                      • RanmaR
                                        Ranma
                                        last edited by Ranma 23 Mar 2019, 16:55

                                        Would appreciate a tiny bit of help. I've read the documentation and various threads but yet to find an answer. So far doing really well with Skyscraper. It's amazing! But I'm having an issue with one or two games that share a name with each other and it's driving me crazy!

                                        Particularly, there's a MAME title called Badlands. Then there's a MAME title called Bad Lands. It's always selecting the default incorrect one for my needs and I don't know how to get it to offer me a choice between them. I thought the --interactive option would help but it didn't seem to make any difference. How can I select the correct one? For info I am using Skyscraper on my PC running Manjaro Linux. It's otherwise perfect.

                                        Any help with this very much appreciated. :-)

                                        Q 1 Reply Last reply 23 Mar 2019, 17:13 Reply Quote 0
                                        • Q
                                          quicksilver @Ranma
                                          last edited by 23 Mar 2019, 17:13

                                          @Ranma which source is returning the wrong result? And have you tried a different source?

                                          RanmaR 1 Reply Last reply 23 Mar 2019, 18:13 Reply Quote 0
                                          • First post
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            [[user:consent.lead]]
                                            [[user:consent.not_received]]