RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Versatile C++ game scraper: Skyscraper

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscrapergamelist.xmlscrapinggithub
    1.6k Posts 113 Posters 1.6m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I
      incunabula
      last edited by

      This tool is awesome! Thanks loads for all of the hard work and sharing it with the community!

      muldjordM 1 Reply Last reply Reply Quote 0
      • UDb23U
        UDb23
        last edited by

        @muldjord said in Versatile C++ game scraper: Skyscraper:

        I will look into vic20 later today. :)
        EDIT: I've just had a look at the RetroPie platforms wiki page. I don't see vic20 in there. I know it's a Commodore branded machine, but how does the platform work in RetroPie? Where do the roms reside?

        Commodore Vic20 is emulated by Vice; same as Commodore64.
        So the System still is Commodore64 in Retropie ,but it can emulate also Vic20 (just different runcommand option)
        Vic20 roms & images are therefore located inside the Commodore64 system folder.

        Thanks!!

        muldjordM 1 Reply Last reply Reply Quote 0
        • muldjordM
          muldjord @incunabula
          last edited by

          @incunabula Thank you! Enjoy! :)

          1 Reply Last reply Reply Quote 0
          • muldjordM
            muldjord @UDb23
            last edited by

            @udb23 In that case, you should just be able to use the 'c64' platform which also looks for vic20 files. Just run it with "./Skyscraper -p c64". Have you tried that or am I missing something?

            UDb23U 1 Reply Last reply Reply Quote 0
            • UDb23U
              UDb23 @muldjord
              last edited by

              @muldjord ok, will try it out over the weekend and let you know.

              muldjordM 1 Reply Last reply Reply Quote 0
              • muldjordM
                muldjord @UDb23
                last edited by

                @udb23
                Awesome!

                Used2BeRXU 1 Reply Last reply Reply Quote 0
                • Used2BeRXU
                  Used2BeRX @muldjord
                  last edited by

                  @muldjord I still have some more NES games to add and some more synopsis tweaks, but I'm going to be away from this work for a few weeks unfortunately. I won't forget about this though and you'll get the pack of updated NES synopsis.txt files when they're done. I've got this page bookmarked so I can catch up when I'm back to it.

                  muldjordM 1 Reply Last reply Reply Quote 0
                  • muldjordM
                    muldjord @Used2BeRX
                    last edited by

                    @Used2BeRX No hurries, I'll look into it when you have things ready. :)

                    1 Reply Last reply Reply Quote 0
                    • I
                      incunabula
                      last edited by

                      Is there a way to change the output of the <name> tag that is written to the gamelist.xml file such that the country information is not shown in ES? It looks like it is currently using the file name of the rom. For example:

                      Super Fun Game Deluxe (USA, Europe, Japan).zip

                      should be displayed in the ES game list as

                      Super Fun Game Deluxe

                      Is this already possible? I looked throught the readme and the command line switches but didn't see anything relevant. Thank you!

                      muldjordM 2 Replies Last reply Reply Quote 0
                      • muldjordM
                        muldjord @incunabula
                        last edited by muldjord

                        @incunabula No, this is not possible at the moment. It is not actually using the file name. What it does is register any round and squarebracket "notes" as I call them. It then uses the web result title, and adds the notes back in.

                        I'll add a '--nobrackets' option in the next release that will disable the bracket tags. :) You will also be able to set 'brackets="false"' in the config.ini file under both [main] and [platform]. That should give you plenty of options for disabling it. :)

                        Also, stay tuned for attractmode support, also coming in the next release.

                        Used2BeRXU I 2 Replies Last reply Reply Quote 1
                        • Used2BeRXU
                          Used2BeRX @muldjord
                          last edited by

                          @muldjord Hey guys. I won't be around for a few weeks, but I'll get right back to work on the NES synopsis stuff when I do.

                          I will only have the NES/Famicom/FDS games done at that point, but the problem you mentioned above won't be an issue if you scrape from the top line of the synopsis. I have a spreadsheet that displays all of this, but I wasn't able to get it ready for a public release before it was time to wrap it up unfortunately.

                          The spreadsheet shows the file name for the roms, synopsis and all associated media. It shows the top line of the synopsis which is the name displayed in the romlist using meleu's script, as well as using the XBox emulators. It also shows which games have a manual, which ones have videos, and the exact dimensions of the raw artwork files for Box Front and Cart images. Every single game has these two images. Hundreds of them were made by me personally for the more obscure games, hundreds more have been touched up to varying degrees, and anywhere from 1,000-1500 of them were cropped slightly to have a uniform look and to get rid of any beat up edges. (Most US games had great restorations done by other people, but a lot of foreign, pirate and other games had some pretty shoddy boxes even if they were HD images).

                          Anyways.... gotta go for now, but I'll be back soon. Good luck on your project muldjord.

                          muldjordM 1 Reply Last reply Reply Quote 1
                          • muldjordM
                            muldjord @Used2BeRX
                            last edited by

                            @Used2BeRX Have fun dude! I'll be here when you get back. :)

                            1 Reply Last reply Reply Quote 0
                            • I
                              incunabula @muldjord
                              last edited by

                              @muldjord That would be great, thank you! If you need someone to test out new builds or whatever, i'm happy to help.

                              muldjordM 1 Reply Last reply Reply Quote 0
                              • muldjordM
                                muldjord @incunabula
                                last edited by muldjord

                                @incunabula If I can get you to test the current built, that would be great. The tag option I mentioned is in there and I've fixed a bunch of other stuff aswell. Any feedback would be very welcome before I release it officially:
                                https://github.com/muldjord/skyscraper/archive/b031b26889c827f2559995fc6592f827e4b00a4b.zip

                                1 Reply Last reply Reply Quote 0
                                • I
                                  incunabula
                                  last edited by

                                  OK, i'll give it a go. Something i noticed last night was that my MSX and Game Gear roms had to be unzipped before they could be scraped. All other platforms that i've scraped so far worked fine as zipped.

                                  muldjordM 1 Reply Last reply Reply Quote 0
                                  • muldjordM
                                    muldjord @incunabula
                                    last edited by muldjord

                                    @incunabula Yes, according to the RetroPie wiki, the GameGear and MSX emulators don't support .zip files. So that's why it isn't included. If you can confirm that it works with those filenames without unzipping them, I can easily add it. Using zipped roms does have a few disadvantages though, so I don't recommend ever using zipped roms.

                                    EDIT: Let me elaborate a bit on that. I use the sha1 checksum for storage of local resources as a means of having a unique key per rom. For best results the actual rom data is preferred.
                                    Another disadvantage is that the 'screenscraper' module uses the sha1 checksum of rom data for identifying them. If they are zipped, it can't do that. And unzipping them internally makes no sense, since zips often contain more than 1 rom.

                                    The only reason I can see to actually zip roms, is that it makes it easier to pack together different roms for the same game. It doesn't save much space because of the type of data anyways, and if you use a zip with multiple roms, it actually costs you space, since you have a bunch of roms inside the zip you won't ever use.

                                    So, that's my thoughts and concerns on the subject. :) Not trying to tell you what to do, just thought I'd give a bit of background for why I think zipped roms is a problem.

                                    1 Reply Last reply Reply Quote 0
                                    • I
                                      incunabula
                                      last edited by incunabula

                                      That makes sense - the checksum could be anything depending on what application was used to compress the file, what level of compression was used, etc. Ok, i'm fine with unzipping these roms (rather small file sizes already) but i can confirm that MSX (using lr-bluemsx) and GG (using lr-genesis-plus-gx) both do in fact work with zipped roms.

                                      muldjordM 1 Reply Last reply Reply Quote 0
                                      • I
                                        incunabula
                                        last edited by

                                        Is there a way you can add local folders as a scraping option? For instance if i already had a folder of boxart and it has some images that the scraper modules are not returning could it be possible to "scrape" images from a default path like %roms%/boxart ?

                                        muldjordM Used2BeRXU 2 Replies Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @incunabula
                                          last edited by muldjord

                                          @incunabula I'm considering options for this, same thing I am working on with Used2BeRX. The problem here of course being that people tend to have varying ways of saving these files so I would need to handle "all of them" so to speak.

                                          If would be really cool if you could sortof "import" a snap folder into the localdb and then just scrape using the 'localdb' module and it would use those snaps. But all local db resources are identified by sha1 checksum, and the snap images that are not easily identifiable other than from their filenames. And that poses a big problem.

                                          I think a first implementation will be to define an xml format where your snaps would need to be in. Then you can scrape from that. That would then automatically add it to the localdb. So when you scrape from 'localdb' afterwards, it would use those snaps aswell as any other snaps you might have acquired.

                                          EDIT: I realize what I just wrote might seem a bit fuzzy :D Bottom line is that I am considering it. And if I find "the right way"(tm) I will implement it.

                                          I UDb23U 2 Replies Last reply Reply Quote 0
                                          • muldjordM
                                            muldjord @incunabula
                                            last edited by muldjord

                                            @incunabula said in Versatile C++ game scraper: Skyscraper:

                                            That makes sense - the checksum could be anything depending on what application was used to compress the file, what level of compression was used, etc. Ok, i'm fine with unzipping these roms (rather small file sizes already) but i can confirm that MSX (using lr-bluemsx) and GG (using lr-genesis-plus-gx) both do in fact work with zipped roms.

                                            Exactly. But I will add .zip to both MSX and GameGear, no problem.

                                            EDIT: zip support is implemented in this release, feel free to test it: https://github.com/muldjord/skyscraper/archive/a820fbd8368372c09c5f3a00338dbf040bc23e43.zip

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.