RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      AlCzervik
      last edited by

      @muldjord Any reason when I rescraped my NES many of them now are showing "load screens" (e.g., Select 1 or 2 player screen) vs. gameplay screens?

      muldjordM 1 Reply Last reply Reply Quote 0
      • muldjordM
        muldjord @AlCzervik
        last edited by

        @AlCzervik What source is it using for the screenshots? (you can check it with Skyscraper -p nes --cache edit FILENAME. The reason is that sometimes the source has the title screens in their databases as screenshots. It's quite normal, but is a bit of a challenge when scraping from them, since I don't know one screenshot from another.

        A 1 Reply Last reply Reply Quote 0
        • A
          AlCzervik @muldjord
          last edited by

          @muldjord I'm using screenscraper. Is it random? As in, if I scrape again would it pull different images they've stored? Some examples I know changed were Super Mario 3, Metroid, Zelda 2.

          muldjordM 1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @AlCzervik
            last edited by

            @AlCzervik When scraping with -s screenscraper it will first look for a screenshot of the type ss which is usually a normal screenshot. If it doesn't find that, it will look for one called sstitlewhich, obviously, is the game title screen screenshot. So in this case, my guess is that it doesn't find any of type ss and therefore uses the sstitle screenshots.

            A 1 Reply Last reply Reply Quote 0
            • A
              AlCzervik @muldjord
              last edited by

              @muldjord That must be it for those titles but it is strange that they changed recently. These are older NES titles so perhaps screenscraper changed for some of those titles. Not a big deal, just wanted to point it out. Thanks!

              1 Reply Last reply Reply Quote 0
              • T
                theshadowzero
                last edited by theshadowzero

                @muldjord i've got a quick question about skyscraper, i've tried messing with the --cache settings but haven't been able to quite hit my use case. i scraped all my roms over a couple days with skyscraper from screenscraper, but i had videos disabled. now, i'd like to add videos, but i haven't been able to figure out how to get skyscraper to just scrape videos. because the roms already have metadata in the cache, skyscraper seems to want to either skip them entirely, or with a --cache refresh completely scrape them fresh. is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

                ive tried running it with just --flags videos and it asks me if i want to skip already existing game list entries. saying yes obviously skips all the roms, and saying no will show that it finds the rom, show all the metadata and media i have (with a big red NO() for video), and then do nothing and proceed to the next rom. if i scrape with --cache refresh it finds the rom and the video and will download and include it. i was under the impression skyscraper would automatically add media that is missing but that doesn't seem to be the case, or more likely i am doing something wrong. guidance would be appreciated! i don't want to spend another 2-3 days hammering screenscraper.fr to get videos when i already have 90% of the scrapes i need

                edit: i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless

                muldjordM 1 Reply Last reply Reply Quote 0
                • muldjordM
                  muldjord @theshadowzero
                  last edited by muldjord

                  @theshadowzero said in Versatile C++ game scraper: Skyscraper:

                  is there a way to tell skyscraper "the metadata is fine, i just want you to scrape the videos that are missing and not rescrape all the other fields?"

                  No, it will always scrape the metadata since it's "free". When a game is looked up, the metadata is already there in the data received from the server. So it would be pretty silly to not use it.
                  It's the media that might take some time to grab - especially the videos of course.

                  I would simply rescrape all of the games with videos enabled. That's the easiest for this use case. You will need --cache refresh for that to work.

                  i also tried with --cache edit:new=videos but it errors saying it only supports certain resources. not sure that would have solved it for me anyway since i fear it would have expected manual input rather than scraping but i tried it regardless

                  Correct, you cannot use --cache edit for this use case. It can only edit metadata and add metadata. Not media.

                  1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord
                    last edited by muldjord

                    Skyscraper 3.5.9 released: https://github.com/muldjord/skyscraper

                    • Implemented the new IGDB v4 authentication method. IGDB will now work again, and requires free credentials. Read more about that here
                    • Improved memory consumption when handing entries back to main thread
                    • Added '.au3' file extension to id script exception list (Thank you to 'o0alucard0o' for reporting this)

                    Most prominent feature is the updated IGDB v4 API authentication method. This was broken due to IGDB moving away from their old v3 API. This is now fixed and requires free personal credentials. Read more about that here.

                    Let me know if you run into issues.

                    ClydeC 1 Reply Last reply Reply Quote 2
                    • ClydeC
                      Clyde @muldjord
                      last edited by Clyde

                      @muldjord Perfect timing as ScreenScraper is currently down! šŸ˜‰

                      Seriously, thanks for still working on this awsome tool. šŸ‘

                      muldjordM 1 Reply Last reply Reply Quote 1
                      • muldjordM
                        muldjord @Clyde
                        last edited by muldjord

                        @Clyde Yeah, they seem to be hit hard this time. I hope they aren't about to shut down... :S

                        EDIT: Ok, just visited your link. So they are working on it. Good to know. :) Thanks.

                        1 Reply Last reply Reply Quote 0
                        • ClydeC
                          Clyde
                          last edited by Clyde

                          Hi,
                          I just noticed that Skyscraper doesn't seem to generate a valid gamelist.xml for Daphne, at least not from the file structure for Daphne that's described in the Docs.

                          Since that structure requires directories instead of archive files, Skyscraper writes a <folder> element instead of <game>:

                          <?xml version="1.0"?>
                          <gameList>
                            <folder>
                              <path>./ace.daphne</path>
                          ...
                          

                          This makes Emulation Station ignore the game's entry:

                          Nov 19 23:26:24 lvl2: 	Parsing XML file "/home/pi/RetroPie/roms/daphne/gamelist.xml"...
                          Nov 19 23:26:24 lvl1: 	gameList: folder doesn't already exist, won't create
                          Nov 19 23:26:24 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/daphne/ace.daphne", skipping.
                          

                          Can you confirm this? Should I open an issue on Github?

                          Thanks
                          Clyde

                          1 Reply Last reply Reply Quote 0
                          • ClydeC
                            Clyde
                            last edited by

                            And another question about Daphne:

                            SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

                            Thanks again
                            Clyde

                            muldjordM 2 Replies Last reply Reply Quote 0
                            • muldjordM
                              muldjord @Clyde
                              last edited by muldjord

                              @clyde I spend quite a lot of time getting Daphne to work some time ago I think, but I can't remember how it works. I'll have to spend some time looking into that when I get the time.

                              EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

                              ClydeC 2 Replies Last reply Reply Quote 0
                              • ClydeC
                                Clyde @muldjord
                                last edited by

                                @muldjord Of course, take your time. And please tell me if I can help you with that in any way (other than coding, which I am not proficient in).

                                1 Reply Last reply Reply Quote 0
                                • ClydeC
                                  Clyde @muldjord
                                  last edited by

                                  @muldjord said in Versatile C++ game scraper: Skyscraper:

                                  EDIT: @Clyde, please check the release info here. That version (3.5.0 and 3.5.1) is where I implemented daphne properly.

                                  Alas, I'm using version 3.6.1 already, and I have four roms in daphne/roms (ace.zip, dle21.zip, lair.zip, lair2.zip).

                                  1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @Clyde
                                    last edited by

                                    @clyde said in Versatile C++ game scraper: Skyscraper:

                                    And another question about Daphne:

                                    SS scrapes Space Ace wrongly as the MAME aircraft game Ace, probable because of SA's rom directory ace.daphne. How can I fix that?

                                    Thanks again
                                    Clyde

                                    By SS do you mean Skyscraper or Screenscraper in this case? :D

                                    muldjordM ClydeC 2 Replies Last reply Reply Quote 0
                                    • muldjordM
                                      muldjord @muldjord
                                      last edited by

                                      @muldjord Another thing, do you adhere to the folder layout as specified here?

                                      1 Reply Last reply Reply Quote 0
                                      • ClydeC
                                        Clyde
                                        last edited by

                                        @muldjord Yes, I do:

                                        $ tree /home/pi/RetroPie/roms/daphne/
                                        /home/pi/RetroPie/roms/daphne/
                                        ā”œā”€ā”€ ace.daphne
                                        │   ā”œā”€ā”€ ace.commands
                                        │   ā”œā”€ā”€ ace.dat
                                        │   ā”œā”€ā”€ ace.m2v
                                        │   ā”œā”€ā”€ ace.ogg
                                        │   └── ace.txt
                                        ā”œā”€ā”€ daphne_log.txt
                                        ā”œā”€ā”€ dapinput.ini
                                        ā”œā”€ā”€ dapinput.ini.backup
                                        ā”œā”€ā”€ dle21.daphne
                                        │   ā”œā”€ā”€ dle21.commands
                                        │   ā”œā”€ā”€ dle21.dat
                                        │   ā”œā”€ā”€ dle21.ogg
                                        │   └── dle21.txt
                                        ā”œā”€ā”€ gamelist.xml
                                        ā”œā”€ā”€ lair2.daphne
                                        │   ā”œā”€ā”€ dl2-00001.dat
                                        │   ā”œā”€ā”€ dl2-00001.m2v
                                        │   ā”œā”€ā”€ dl2-00001.ogg
                                        [... many more dat/m2v/ogg files ...]
                                        │   ā”œā”€ā”€ dl2-44550.m2v
                                        │   ā”œā”€ā”€ dl2-44550.ogg
                                        │   ā”œā”€ā”€ dl2-framefile.txt
                                        │   └── lair2.txt
                                        ā”œā”€ā”€ lair.daphne
                                        │   ā”œā”€ā”€ lair.commands
                                        │   ā”œā”€ā”€ lair.dat
                                        │   ā”œā”€ā”€ lair.m2v
                                        │   ā”œā”€ā”€ lair.ogg
                                        │   └── lair.txt
                                        ā”œā”€ā”€ media
                                        │   ā”œā”€ā”€ covers
                                        │   │   ā”œā”€ā”€ ace.png
                                        │   │   ā”œā”€ā”€ dle21.png
                                        │   │   ā”œā”€ā”€ lair2.png
                                        │   │   └── lair.png
                                        │   ā”œā”€ā”€ marquees
                                        │   │   ā”œā”€ā”€ ace.png
                                        │   │   ā”œā”€ā”€ dle21.png
                                        │   │   ā”œā”€ā”€ lair2.png
                                        │   │   └── lair.png
                                        │   ā”œā”€ā”€ screenshots
                                        │   │   ā”œā”€ā”€ ace.png
                                        │   │   ā”œā”€ā”€ dle21.png
                                        │   │   ā”œā”€ā”€ lair2.png
                                        │   │   └── lair.png
                                        │   ā”œā”€ā”€ videos
                                        │   │   ā”œā”€ā”€ ace.mp4
                                        │   │   ā”œā”€ā”€ dle21.mp4
                                        │   │   ā”œā”€ā”€ lair2.mp4
                                        │   │   └── lair.mp4
                                        │   └── wheels
                                        ā”œā”€ā”€ README.txt
                                        └── roms
                                            ā”œā”€ā”€ ace.zip
                                            ā”œā”€ā”€ dle21.zip
                                            ā”œā”€ā”€ lair2.zip
                                            └── lair.zip
                                        
                                        ClydeC 1 Reply Last reply Reply Quote 0
                                        • ClydeC
                                          Clyde @muldjord
                                          last edited by

                                          @muldjord said in Versatile C++ game scraper: Skyscraper:

                                          By SS do you mean Skyscraper or Screenscraper in this case? :D

                                          Skyscraper :)

                                          1 Reply Last reply Reply Quote 0
                                          • ClydeC
                                            Clyde @Clyde
                                            last edited by Clyde

                                            @clyde said in Versatile C++ game scraper: Skyscraper:

                                            ā”œā”€ā”€ dle21.daphne
                                            │   ā”œā”€ā”€ dle21.commands
                                            │   ā”œā”€ā”€ dle21.dat
                                            │   ā”œā”€ā”€ dle21.ogg
                                            │   └── dle21.txt
                                            

                                            There is a m2v missing that I plan to hard link to because my dle21.txt points to ../lair.daphne/lair.m2v (edit: as I just remembered :*) ), but that shouldn't concern my problem, since none of the four games are put correctly in the gamelist.xml by Skyscraper. All have the <folder> element instead of <game>.

                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.