RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • muldjordM
      muldjord @theshadowzero
      last edited by muldjord

      @theshadowzero said in Versatile C++ game scraper: Skyscraper:

      is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

      No, it will always scrape the metadata since it's "free". When a game is looked up, the metadata is already there in the data received from the server. So it would be pretty silly to not use it.
      It's the media that might take some time to grab - especially the videos of course.

      I would simply rescrape all of the games with videos enabled. That's the easiest for this use case. You will need --cache refresh for that to work.

      i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless

      Correct, you cannot use --cache edit for this use case. It can only edit metadata and add metadata. Not media.

      1 Reply Last reply Reply Quote 0
      • muldjordM
        muldjord
        last edited by muldjord

        Skyscraper 3.5.9 released: https://github.com/muldjord/skyscraper

        • Implemented the new IGDB v4 authentication method. IGDB will now work again, and requires free credentials. Read more about that here
        • Improved memory consumption when handing entries back to main thread
        • Added '.au3' file extension to id script exception list (Thank you to 'o0alucard0o' for reporting this)

        Most prominent feature is the updated IGDB v4 API authentication method. This was broken due to IGDB moving away from their old v3 API. This is now fixed and requires free personal credentials. Read more about that here.

        Let me know if you run into issues.

        ClydeC 1 Reply Last reply Reply Quote 2
        • ClydeC
          Clyde @muldjord
          last edited by Clyde

          @muldjord Perfect timing as ScreenScraper is currently down! šŸ˜‰

          Seriously, thanks for still working on this awsome tool. šŸ‘

          muldjordM 1 Reply Last reply Reply Quote 1
          • muldjordM
            muldjord @Clyde
            last edited by muldjord

            @Clyde Yeah, they seem to be hit hard this time. I hope they aren't about to shut down... :S

            EDIT: Ok, just visited your link. So they are working on it. Good to know. :) Thanks.

            1 Reply Last reply Reply Quote 0
            • ClydeC
              Clyde
              last edited by Clyde

              Hi,
              I just noticed that Skyscraper doesn't seem to generate a valid gamelist.xml for Daphne, at least not from the file structure for Daphne that's described in the Docs.

              Since that structure requires directories instead of archive files, Skyscraper writes a <folder> element instead of <game>:

              <?xml version="1.0"?>
              <gameList>
                <folder>
                  <path>./ace.daphne</path>
              ...
              

              This makes Emulation Station ignore the game's entry:

              Nov 19 23:26:24 lvl2: 	Parsing XML file "/home/pi/RetroPie/roms/daphne/gamelist.xml"...
              Nov 19 23:26:24 lvl1: 	gameList: folder doesn't already exist, won't create
              Nov 19 23:26:24 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/daphne/ace.daphne", skipping.
              

              Can you confirm this? Should I open an issue on Github?

              Thanks
              Clyde

              1 Reply Last reply Reply Quote 0
              • ClydeC
                Clyde
                last edited by

                And another question about Daphne:

                SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

                Thanks again
                Clyde

                muldjordM 2 Replies Last reply Reply Quote 0
                • muldjordM
                  muldjord @Clyde
                  last edited by muldjord

                  @clyde I spend quite a lot of time getting Daphne to work some time ago I think, but I can't remember how it works. I'll have to spend some time looking into that when I get the time.

                  EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

                  ClydeC 2 Replies Last reply Reply Quote 0
                  • ClydeC
                    Clyde @muldjord
                    last edited by

                    @muldjord Of course, take your time. And please tell me if I can help you with that in any way (other than coding, which I am not proficient in).

                    1 Reply Last reply Reply Quote 0
                    • ClydeC
                      Clyde @muldjord
                      last edited by

                      @muldjord said in Versatile C++ game scraper: Skyscraper:

                      EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

                      Alas, I'm using version 3.6.1 already, and I have four roms in daphne/roms (ace.zip, dle21.zip, lair.zip, lair2.zip).

                      1 Reply Last reply Reply Quote 0
                      • muldjordM
                        muldjord @Clyde
                        last edited by

                        @clyde said in Versatile C++ game scraper: Skyscraper:

                        And another question about Daphne:

                        SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

                        Thanks again
                        Clyde

                        By SS do you mean Skyscraper or Screenscraper in this case? :D

                        muldjordM ClydeC 2 Replies Last reply Reply Quote 0
                        • muldjordM
                          muldjord @muldjord
                          last edited by

                          @muldjord Another thing, do you adhere to the folder layout as specified here?

                          1 Reply Last reply Reply Quote 0
                          • ClydeC
                            Clyde
                            last edited by

                            @muldjord Yes, I do:

                            $ tree /home/pi/RetroPie/roms/daphne/
                            /home/pi/RetroPie/roms/daphne/
                            ā”œā”€ā”€ ace.daphne
                            │   ā”œā”€ā”€ ace.commands
                            │   ā”œā”€ā”€ ace.dat
                            │   ā”œā”€ā”€ ace.m2v
                            │   ā”œā”€ā”€ ace.ogg
                            │   └── ace.txt
                            ā”œā”€ā”€ daphne_log.txt
                            ā”œā”€ā”€ dapinput.ini
                            ā”œā”€ā”€ dapinput.ini.backup
                            ā”œā”€ā”€ dle21.daphne
                            │   ā”œā”€ā”€ dle21.commands
                            │   ā”œā”€ā”€ dle21.dat
                            │   ā”œā”€ā”€ dle21.ogg
                            │   └── dle21.txt
                            ā”œā”€ā”€ gamelist.xml
                            ā”œā”€ā”€ lair2.daphne
                            │   ā”œā”€ā”€ dl2-00001.dat
                            │   ā”œā”€ā”€ dl2-00001.m2v
                            │   ā”œā”€ā”€ dl2-00001.ogg
                            [... many more dat/m2v/ogg files ...]
                            │   ā”œā”€ā”€ dl2-44550.m2v
                            │   ā”œā”€ā”€ dl2-44550.ogg
                            │   ā”œā”€ā”€ dl2-framefile.txt
                            │   └── lair2.txt
                            ā”œā”€ā”€ lair.daphne
                            │   ā”œā”€ā”€ lair.commands
                            │   ā”œā”€ā”€ lair.dat
                            │   ā”œā”€ā”€ lair.m2v
                            │   ā”œā”€ā”€ lair.ogg
                            │   └── lair.txt
                            ā”œā”€ā”€ media
                            │   ā”œā”€ā”€ covers
                            │   │   ā”œā”€ā”€ ace.png
                            │   │   ā”œā”€ā”€ dle21.png
                            │   │   ā”œā”€ā”€ lair2.png
                            │   │   └── lair.png
                            │   ā”œā”€ā”€ marquees
                            │   │   ā”œā”€ā”€ ace.png
                            │   │   ā”œā”€ā”€ dle21.png
                            │   │   ā”œā”€ā”€ lair2.png
                            │   │   └── lair.png
                            │   ā”œā”€ā”€ screenshots
                            │   │   ā”œā”€ā”€ ace.png
                            │   │   ā”œā”€ā”€ dle21.png
                            │   │   ā”œā”€ā”€ lair2.png
                            │   │   └── lair.png
                            │   ā”œā”€ā”€ videos
                            │   │   ā”œā”€ā”€ ace.mp4
                            │   │   ā”œā”€ā”€ dle21.mp4
                            │   │   ā”œā”€ā”€ lair2.mp4
                            │   │   └── lair.mp4
                            │   └── wheels
                            ā”œā”€ā”€ README.txt
                            └── roms
                                ā”œā”€ā”€ ace.zip
                                ā”œā”€ā”€ dle21.zip
                                ā”œā”€ā”€ lair2.zip
                                └── lair.zip
                            
                            ClydeC 1 Reply Last reply Reply Quote 0
                            • ClydeC
                              Clyde @muldjord
                              last edited by

                              @muldjord said in Versatile C++ game scraper: Skyscraper:

                              By SS do you mean Skyscraper or Screenscraper in this case? :D

                              Skyscraper :)

                              1 Reply Last reply Reply Quote 0
                              • ClydeC
                                Clyde @Clyde
                                last edited by Clyde

                                @clyde said in Versatile C++ game scraper: Skyscraper:

                                ā”œā”€ā”€ dle21.daphne
                                │   ā”œā”€ā”€ dle21.commands
                                │   ā”œā”€ā”€ dle21.dat
                                │   ā”œā”€ā”€ dle21.ogg
                                │   └── dle21.txt
                                

                                There is a m2v missing that I plan to hard link to because my dle21.txt points to ../lair.daphne/lair.m2v (edit: as I just remembered :*) ), but that shouldn't concern my problem, since none of the four games are put correctly in the gamelist.xml by Skyscraper. All have the <folder> element instead of <game>.

                                muldjordM 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @Clyde
                                  last edited by

                                  @clyde Ok, I think that's enough for me to go on, thanks. I might look into it the coming days.

                                  muldjordM 1 Reply Last reply Reply Quote 2
                                  • muldjordM
                                    muldjord @muldjord
                                    last edited by muldjord

                                    @Clyde I've tested it today with lair and lair2 and it works just fine for me, showing the screenshot and data when moving the selection over lair.daphne folder in ES and starting the game when selecting the folder. It should be a <path> in the gamelist.xml since that is what initializes the game. At least on my default daphne installation.

                                    Perhaps something is messed up with your ace files, I can't really know since I don't have that game. But I don't get any ES errors for lair and lair2.

                                    ClydeC 1 Reply Last reply Reply Quote 0
                                    • ClydeC
                                      Clyde @muldjord
                                      last edited by Clyde

                                      @muldjord Here's the gamelist my Skyscraper 3.6.1 is generating for the daphne system:

                                      <?xml version="1.0"?>
                                      <gameList>
                                        <folder>
                                          <path>./ace.daphne</path>
                                          <name>Ace</name>
                                          <thumbnail>./media/covers/ace.png</thumbnail>
                                          <image>./media/screenshots/ace.png</image>
                                          <marquee>./media/marquees/ace.png</marquee>
                                          <video>./media/videos/ace.mp4</video>
                                          <rating>0.1</rating>
                                          <desc>An old air battle game that features dogfights between 2 biplanes.</desc>
                                          <releasedate>19760101T000000</releasedate>
                                          <developer />
                                          <publisher>Allied Leisure</publisher>
                                          <genre />
                                          <players>2</players>
                                        </folder>
                                        <folder>
                                          <path>./lair.daphne</path>
                                          <name>Dragon&apos;s Lair</name>
                                          <thumbnail>./media/covers/lair.png</thumbnail>
                                          <image>./media/screenshots/lair.png</image>
                                          <marquee>./media/marquees/lair.png</marquee>
                                          <video>./media/videos/lair.mp4</video>
                                          <rating>0.8</rating>
                                          <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
                                          <releasedate>19830601T000000</releasedate>
                                          <developer>Sullivan Bluth Interactive</developer>
                                          <publisher>Cinematronics</publisher>
                                          <genre>Platform, Plateform / Run Jump</genre>
                                          <players>2</players>
                                        </folder>
                                        <folder>
                                          <path>./dle21.daphne</path>
                                          <name>Dragon&apos;s Lair</name>
                                          <thumbnail>./media/covers/dle21.png</thumbnail>
                                          <image>./media/screenshots/dle21.png</image>
                                          <marquee>./media/marquees/dle21.png</marquee>
                                          <video>./media/videos/dle21.mp4</video>
                                          <rating>0.8</rating>
                                          <desc>Originally released in the arcades as a laserdisc game, Dragon&apos;s Lair is an interactive cartoon movie. Players control Dirk the Daring as he struggles his way through a dungeon to fight Singe, the Dragon, and rescue the beautiful Princess Daphne. The game consists of animated scenes, during which the player has to press direction buttons or the sword button in the right moment to trigger the next segment of the movie.</desc>
                                          <releasedate>19830601T000000</releasedate>
                                          <developer>Sullivan Bluth Interactive</developer>
                                          <publisher>Cinematronics</publisher>
                                          <genre>Platform, Plateform / Run Jump</genre>
                                          <players>2</players>
                                        </folder>
                                        <folder>
                                          <path>./lair2.daphne</path>
                                          <name>Dragon&apos;s Lair 2 - Time Warp</name>
                                          <thumbnail>./media/covers/lair2.png</thumbnail>
                                          <image>./media/screenshots/lair2.png</image>
                                          <marquee>./media/marquees/lair2.png</marquee>
                                          <video>./media/videos/lair2.mp4</video>
                                          <rating>0.9</rating>
                                          <desc>Princess Daphne has been spirited away to a wrinkle in time by the Evil Wizard Mordroc who plans to force her into marriage. Only you, Dirk the Daring, can save her. Transported by a bumbling old time machine, you begin the rescue mission. But you must hurry, for once the Casket of Doom has opened, Mordroc will place the Death Ring upon Daphne&apos;s finger in marriage and she will be lost forever in the Time Warp!</desc>
                                          <releasedate>19910616T000000</releasedate>
                                          <developer>Sullivan Bluth Interactive</developer>
                                          <publisher>Leland Corporation</publisher>
                                          <genre>Platform, Plateform / Run Jump</genre>
                                          <players>1</players>
                                        </folder>
                                      </gameList>
                                      

                                      The <path> element is not the problem, but the <folder> where there should be <game>, and it occurs for all of my four daphne games ace, dle21, lair and lair2.

                                      Would the complete output of Skyscraper help you? If so, could you please tell me the best way to generate it?

                                      As for ace, could you maybe copy your lair directory and then rename it and everything in it to mirror an ace installation to see if SS would scrape the MAME game Ace instead of Space Ace? (As long as SS won't recognise the files' checksums and ignore their naming.)

                                      Just some suggestions to dive deeper into those two issues that you may sacrifice your time for or not at your discretion.

                                      muldjordM 2 Replies Last reply Reply Quote 0
                                      • muldjordM
                                        muldjord @Clyde
                                        last edited by muldjord

                                        @clyde Sorry, I misspoke, it should be <folder> since it is a folder. Why do you want it to be a <game>? I don't quite understand (maybe you know something I don't).

                                        It works well on my system, simply showing the game name while hovering over the folder and starting the game when selecting it.

                                        EDIT: Your gamelist.xml looks fine to me. I can't really figure out what might cause your issue or what the issue is, sorry.

                                        EDIT2: I'll try and test it with Space Ace. Stay tuned.

                                        1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @Clyde
                                          last edited by muldjord

                                          @clyde Got my hands on Space Ace and tested it. Scraped it with Skyscraper using the -s screenscraper source. Found it perfectly with title and artwork. Shows up perfectly in ES and launches and plays just fine. Here's my gamelist entry for Space Ace:

                                            <folder>
                                              <path>./ace.daphne</path>
                                              <name>Space Ace</name>
                                              <thumbnail />
                                              <image>./media/screenshots/ace.png</image>
                                              <marquee>./media/marquees/ace.png</marquee>
                                              <video />
                                              <rating>0.9</rating>
                                              <desc>Space Ace is a laserdisc video game produced by Don Bluth Studios, Cinematronics and Advanced Microcomputer Systems (later renamed RDI Video Systems). It was unveiled in October 1983, just four months after the Dragon&apos;s Lair game, then released in Spring 1984, and like its predecessor featured film-quality animation played back from a laserdisc.
                                          
                                          The gameplay is also similar, requiring the player </desc>
                                              <releasedate>19840424T000000</releasedate>
                                              <developer>Advanced Microcomputer Systems</developer>
                                              <publisher>Cinematronics</publisher>
                                              <genre>Plateform / Run Jump, Platform, Action</genre>
                                              <players>1</players>
                                              <kidgame>true</kidgame>
                                            </folder>
                                          

                                          This works as expected on my setup, so I can't reproduce your error it seems.

                                          EDIT: I do notice some odd behaviour from ES though. If they are present as <folder> in gamelist.xml ES suddenly adds it as a <game> as well when it writes the gamelist.xml back to disk when quitting ES. So it suddenly has both a <folder> entry for it and a <game> entry.

                                          I think this is an ES bug. It shouldn't add that by itself. Perhaps it gets confused since the folder has a . in it (eg ace.daphne).

                                          ES doesn't do this for psx games that are also folders (eg. 3D Lemmings (USA)).

                                          EDIT2: Yeah, I think we've found some sort of bug in ES. When I add a . to one of the folders in one of my psx games (eg. 3d.lemmings) ES suddenly also adds that game as a <game> when it quits. This should probably not happen.

                                          EDIT3: Maybe you're right, maybe it simply should be a <game> entry. Maybe I am misunderstanding the use of it. I juse assumed, since it is a folder, that obviously it should be a <folder> entry.

                                          EDIT4: The behaviour seems to be described here and might be intentional. I can't figure out if I should change it to <game> though... Probably should. Comments?

                                          ClydeC 1 Reply Last reply Reply Quote 1
                                          • ClydeC
                                            Clyde @muldjord
                                            last edited by Clyde

                                            @muldjord Wow, thanks for all the diagnostic work. What's strange is that my ES doesn't recognise <folder> entries at all and throws the errors I quoted earlier, while it does work for you. I have to rename them to <game> to see them in ES.

                                            So, subject to the different behaviour of your ES and mine regarding <folder> elements, I would propose to write <game> entries in every case for maximum compatibility as long as there are no discernable drawbacks to is.

                                            edit: Also strange is that SS scrapes your Space Ace correctly. Can you tell if it recognises the checksum or does it just use the name? Either way, I'll test this a little more on my side.

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.