RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mituM
      mitu Global Moderator @quicksilver
      last edited by mitu

      @quicksilver Ah, ok, then it's normal that generating the list won't work, since the cache entry is associated with the pc platform.

      quicksilverQ 1 Reply Last reply Reply Quote 0
      • quicksilverQ
        quicksilver @mitu
        last edited by

        @mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

        muldjordM 1 Reply Last reply Reply Quote 0
        • muldjordM
          muldjord @quicksilver
          last edited by

          @quicksilver said in Versatile C++ game scraper: Skyscraper:

          @mitu I figured it had something to do with that. But according to @muldjord, ports should already be looking for PC matches but unless I tell it to look for them by setting pc as the platform, it can't find them.

          Quick note: You should never give it a file name on command line that is not related to the current input folder (which in this case is roms/pc). If you do, it won't be included when you generate the game list afterwards since it is no longer in the input directory. You are basically caching data for a file that effectively doesn't exist when you generate the platform later on (since it will look for files in roms/ports at that point). :)

          Ports currently looks for both Windows, Dos and PC as platforms when using the file name based modules (all modules except screenscraper). So that should work. If you use the fix I gave you earlier it might also work with screenscraper.

          quicksilverQ 1 Reply Last reply Reply Quote 0
          • quicksilverQ
            quicksilver @muldjord
            last edited by quicksilver

            @muldjord said in Versatile C++ game scraper: Skyscraper:

            Ports currently looks for both Windows, Dos and PC as platforms when using the file name based modules (all modules except screenscraper). So that should work. If you use the fix I gave you earlier it might also work with screenscraper.

            Except that it isn't working properly. The gamesdb will give a "file not found" for every one of my ports games. If I manually switch the platform to pc it will find the games no problem (I am aware this will not work properly for generating the Gameslist). This says to me that the platform "ports" is not properly searching "pc" like it's supposed to.

            Also do you have a PayPal account? Or only patreon? I would like to tip you for all your hard work on this but would like to give a one-time payment. Wasn't sure if cancelling a patreon after a one month paemt was considered bad form.

            muldjordM 1 Reply Last reply Reply Quote 0
            • ?
              A Former User
              last edited by

              I really wish I could use this on my Windows machine. :(

              Maybe I'll set up a Linux partition just to run this.

              muldjordM 1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord @quicksilver
                last edited by muldjord

                @quicksilver said in Versatile C++ game scraper: Skyscraper:

                Except that it isn't working properly. The gamesdb will give a "file not found" for every one of my ports games. If I manually switch the platform to pc it will find the games no problem (I am aware this will not work properly for generating the Gameslist). This says to me that the platform "ports" is not properly searching "pc" like it's supposed to.

                Also do you have a PayPal account? Or only patreon? I would like to tip you for all your hard work on this but would like to give a one-time payment. Wasn't sure if cancelling a patreon after a one month paemt was considered bad form.

                pc and ports have the exact same aliases, so this is caused by something else. The only difference between the platforms is that pc also looks for *.com *.bat and *.exe in addition to *.sh where ports only look for *.sh. So that might be the issue instead. The aliases are the same. But once again, you need to do the code change I mentioned earlier if you want it to work for screenscraper as well (I know you are using thegamesdb). Just to be clear. :)

                You can donate directly to paypal by using muldjordlars@gmail.com, but I'd actually suggest you donate to the RetroPie project instead. :) That's up to you of course.

                quicksilverQ 2 Replies Last reply Reply Quote 0
                • muldjordM
                  muldjord @A Former User
                  last edited by

                  @LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

                  I really wish I could use this on my Windows machine. :(

                  Maybe I'll set up a Linux partition just to run this.

                  I'm not sure how much work it would take to get it on Windows (probably not that much). But if I do port it to Windows, it will be command line only. :)

                  ? 1 Reply Last reply Reply Quote 0
                  • ?
                    A Former User @muldjord
                    last edited by

                    @muldjord I'd be fine with command line only. :D

                    mituM muldjordM 2 Replies Last reply Reply Quote 0
                    • quicksilverQ
                      quicksilver @muldjord
                      last edited by quicksilver

                      @muldjord thanks for the reply. All of my ports games are launched via .sh files so it shouldn't be a problem. But obviously it's not working so I'm not entirely sure what to do then.

                      I do donate already to RetroPie from time to time but I do also like to reward others as well!

                      Edit: looks like it's just thegamesdb that has an issue with ports. I scraped using openretro and mobygames and it picked up a lot of the games without an issue.

                      1 Reply Last reply Reply Quote 0
                      • mituM
                        mitu Global Moderator @A Former User
                        last edited by mitu

                        @LiveFastCyYoung Skyscraper is command line only. If you feel adventurous, install Cygwin and try to compile it on Windows.
                        Have you tried Skraper ?

                        ? 1 Reply Last reply Reply Quote 0
                        • muldjordM
                          muldjord @A Former User
                          last edited by muldjord

                          @LiveFastCyYoung said in Versatile C++ game scraper: Skyscraper:

                          @muldjord I'd be fine with command line only. :D

                          I've compiled a Windows version that seemingly works. Please follow the instructions in the INSTALL_README.txt after unzipping it. This is important, otherwise it won't run!

                          Also note that this version is 100% unsupported, so you are on your own in using it. :) The only thing I am interested in is to know whether it runs with the dll's I've included. If not I need to include them. So please let me know if it complains about missing dll's when running it. Other than that, it should work just fine. I did a really quick test with it and it scraped nes roms just fine and also generated a gamelist without problems.

                          Here ya go: http://www.muldjord.com/downloads/Skyscraper_3.1.2_unsupported_win_version.zip

                          EDIT: Another quick note. Windows probably has the worst command prompt of any OS and so it does not support the color codings I use. So the output will look pretty weird.

                          EDIT2: I just tried running it from the Powershell. This won't work. You apparently need to use the older Windows command prompt that you can run with "cmd".

                          EDIT3: Updated it to 3.1.1

                          EDIT4: Updated it to 3.1.2

                          ByteThisB 1 Reply Last reply Reply Quote 1
                          • ?
                            A Former User @mitu
                            last edited by A Former User

                            @mitu I currently use Skraper. Since Skyscraper can use more sources than just ScreenScraper, I'd like to give this a shot.

                            @muldjord Excellent! I will run it from cmd prompt when I get a chance tonight and let you know how it goes. Thanks! :)

                            1 Reply Last reply Reply Quote 0
                            • muldjordM
                              muldjord
                              last edited by muldjord

                              Skyscraper 3.1.1 released: https://github.com/muldjord/skyscraper

                              • Worked around a bug in the 'screenscraper' xml output improving stability
                              • Added limiter to media getter functions for 'screenscraper' which should improve stability

                              I've been working to improve the stability of the screenscrapermodule. The fixes included in this release seem to have made quite a difference, so please update.

                              1 Reply Last reply Reply Quote 2
                              • chubstaC
                                chubsta
                                last edited by

                                Quick question guys - if I am trying to update a set of roms, do I have to delete the game list, rescrape the roms, and generate the new game list, or just rescrape and generate?

                                Raspberry Pi 400 - overclocked to 2.3GHz
                                Theme - Comic book
                                Emulators: All of them...
                                Roms: Pretty much all of them...
                                Favourite games: Circus Charlie, Gorf, Gauntlet Legends
                                Controllers: Mayflash F101

                                muldjordM 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @chubsta
                                  last edited by muldjord

                                  @chubsta said in Versatile C++ game scraper: Skyscraper:

                                  Quick question guys - if I am trying to update a set of roms, do I have to delete the game list, rescrape the roms, and generate the new game list, or just rescrape and generate?

                                  Depends on what you mean by "update a set of roms". If you mean you want to update the data from the sources, then you need to regather the data first using the "--refresh" option and then regenerate the game list afterwards. But if you mean "update a set or roms" after you've just changed some artwork configuration or other local Skyscraper setting, then you just need to regenerate the game list. No need to regather the data as it is already cached and ready for use.

                                  EDIT: This page explains it: https://github.com/muldjord/skyscraper/blob/master/docs/USECASE.md

                                  chubstaC 1 Reply Last reply Reply Quote 0
                                  • chubstaC
                                    chubsta @muldjord
                                    last edited by

                                    @muldjord thanks for the fast reply - I have removed a load of roms from one of my Mame sets, and added some more. I don't have any cached info as I clear the cache once I have scraped, my concern was that when writing a new game list it would add the new info to the old and I would end up with both duplicate games and also games in the list that are no longer in the folder.

                                    I guess in this case it is simple enough to just delete the old one and rescrape to give myself a fresh game list.

                                    Great scraper BTW, works so much better than anything else I have tried!

                                    Raspberry Pi 400 - overclocked to 2.3GHz
                                    Theme - Comic book
                                    Emulators: All of them...
                                    Roms: Pretty much all of them...
                                    Favourite games: Circus Charlie, Gorf, Gauntlet Legends
                                    Controllers: Mayflash F101

                                    1 Reply Last reply Reply Quote 0
                                    • ByteThisB
                                      ByteThis @muldjord
                                      last edited by

                                      @muldjord said in Versatile C++ game scraper: Skyscraper:

                                      Windows probably has the worst command prompt

                                      And OS ever. I'll leave this here..lol.

                                      1 Reply Last reply Reply Quote 1
                                      • S
                                        Silent
                                        last edited by

                                        Hey,

                                        Moonlight scriptmodule has now been merged into RetroPie-Setup. Hence, this is the correct time for me to summarize a request to add support for streaming media, like I did in my PR a while ago.

                                        What I propose is:

                                        1. Make .ml a valid extension for all platforms, like .zip .7z is at the moment. Technically you can stream anything from your PC, so there is no reason not to allow it everywhere.
                                        2. Add steam platform support. Should have same aliases as PC platforms, maybe with PC platform also getting a Steam alias.
                                        3. Add ps2 platform support. Even though there is no PS2 emulator on Pi, you can obviously stream it and technically RetroPie supports it. Most EmulationStation themes already support this platform, too.

                                        Those are the changes I made in my local copy, and I think they should be sufficient. Technically, you could argue that ps3 (because of RPCS3) and x360 (because of Xenia) platforms also make sense, but I don't know if there are themes supporting that.

                                        Thanks in advance!

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • quicksilverQ
                                          quicksilver
                                          last edited by

                                          When scraping a game that won't be able to be found by it's checksum, Screenscraper will attempt to match by exact rom name from what I understand. However if there is more than one game with that exact title it will fail to scrape. I have a number of .sh and .conf files that cannot be found by checksum but fail based on ROM name because there are many versions of the game in screenscraper's database (quake II for example). Is there anyway to work around this?

                                          muldjordM 1 Reply Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord @quicksilver
                                            last edited by

                                            @quicksilver said in Versatile C++ game scraper: Skyscraper:

                                            When scraping a game that won't be able to be found by it's checksum, Screenscraper will attempt to match by exact rom name from what I understand. However if there is more than one game with that exact title it will fail to scrape. I have a number of .sh and .conf files that cannot be found by checksum but fail based on ROM name because there are many versions of the game in screenscraper's database (quake II for example). Is there anyway to work around this?

                                            Not that I know of. This is a small limitation in how their database handles things. I do believe it only searches on the platform in question though, so this shouldn't happen too often as the platform id is also send in the requests.

                                            quicksilverQ 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.