• Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login
RetroPie forum home
  • Recent
  • Tags
  • Popular
  • Home
  • Docs
  • Register
  • Login

Versatile C++ game scraper: Skyscraper

Scheduled Pinned Locked Moved Ideas and Development
skyscraperscrapergamelist.xmlscrapinggithub
1.6k Posts 113 Posters 1.5m Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    muldjord @megaJekan
    last edited by 17 Jan 2019, 09:20

    @megaJekan You can use the --query option to force md5 sums for any rom: https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md#--query-string

    1 Reply Last reply Reply Quote 0
    • S
      Silent @muldjord
      last edited by 17 Jan 2019, 10:53

      @muldjord said in Versatile C++ game scraper: Skyscraper:

      And yes, there will be incompatibility, so we'd need to coordinate the release precisely somehow.

      Why not leave those as undocumented aliases for compatibility? Yes, it sucks, but sometimes you gotta do what you gotta do.

      Either that or the RetroPie-Setup script should detect the exact Skyscraper version and adapt. Is this possible?

      M 1 Reply Last reply 17 Jan 2019, 11:11 Reply Quote 0
      • M
        muldjord @Silent
        last edited by muldjord 17 Jan 2019, 11:11

        @Silent I think you answered that yourself. Because it sucks (and I can't "ghost" them, they will be visible and cause confusion). ;) And I do migrate old config.ini and folders. And it's to be expected that there are changes for major versions. Users should know that. I do keep the --refreshfor a while though, so for those where it makes sense, I do actually keep them a couple months. And the change from localdb to cache is compatible with both versions, as it should be simply left out for both which gives the same result. I think, in the end, this will turn out to be a non-issue when it comes to the script as we can work around it.

        1 Reply Last reply Reply Quote 0
        • M
          muldjord
          last edited by muldjord 23 Jan 2019, 18:21

          Skyscraper 3.0.0 released: https://github.com/muldjord/skyscraper

          • Game list creation and media compositing now only processed when leaving out the -s option
          • Now converts textual "\n" in descriptions to newlines. Useful when importing textual data (Thank you to "scthirouin" for requesting it)
          • Added disk space check. Now aborts if disk has less than 200 megs left
          • Updated "igdb" API til APIv3000. Key now global for all users
          • Fixed ES game list release dates by adding "T000000" (Thank you to "michaelbaudino" for noticing)
          • Added "subdirs" to config.ini (Thank you to "otherguy" for noticing it was missing)
          • "--startat" and "--endat" now supports current workdir file paths
          • Changed "allowExtension" config variable to "addExtensions"
          • Changed "noHints" config variable to "hints"
          • Changed "noBrackets" config variable to "brackets"
          • Changed "noResize" config variable to "resize"
          • Removed "--updatedb" cli option as it was deprecated (use "--cache refresh" instead)
          • Removed "--nolocaldb" cli option as it is irrelevant after 3.0.0 usage changes
          • Changed "--allowext" cli option to "--addext"
          • Removed "localDb"/"cache" config sections and moved options to "main"
          • Removed "userCreds" option from main and platform sections in config.ini. They should be set in scraping module section instead
          • Removed "scraper" config.ini variable as it made no sense to ever set it like that
          • Changed "startat" config.ini option to "startAt" to conform with style of other variables
          • Changed "endat" config.ini option to "endAt" to conform with style of other variables
          • Fixed "--relative" option when combined with roms in subdirs
          • "--unattend" options now answers "y" to all "--cache purge:" prompts automatically
          • Added "regionPrios" and "langPrios" variables to config.ini for main and platform sections
          • Added auto-migration from old config.ini format to v3.0.0 format
          • Added auto-migration from "dbs" folder to "cache" folder
          • Optimized "--cache validate" A LOT
          • Now converts ES game list entries to <folder> entries if a .cue or .m3u file is found in a subfolder. This causes the information and artwork to be shown when selecting the folder in ES rather than the .cue or .m3u file inside the subfolder. This is very useful for psx scraping

          This release is a major release. It contains a couple of name convention changes that users will need to adjust to. The old localdb module is now simply called the game list generation mode and it enabled by leaving out the -s <MODULE> option entirely. The -s option is used for gathering data into the resource cache. When you've gathered the resources you need, you simply run it again for the platform without the -s option to generate the game list.

          Some command line options have changed a bit. Most importantly, all things related to the resource cache (previously known as the localdb cache) have been converged into the new --cache option. Please check out the full documentation for all available options here: https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md

          Some config.ini variables have changes names. And the dbs folder is now called cache. All of this will be auto-migrated when you start Skyscraper 3.0.0 for the first time.

          Some fixes also made it in. Most importantly the --relative option have been fixed for users who have roms in subfolders. And for users who scrape .cue or .m3u files in subfolders, Skyscraper will now autodetect this and convert the game list entry to a <folder> entry instead of a <game> entry. This means that the game data and artwork will show when selecting the folder in EmulationStation.

          The igdb module has been updated to APIv3000. As the old module violated the terms of use for the database, it now uses a global key and therefore no longer requires user credentials to be used. There is a big caveat to this though, as the key provided with Skyscraper has a limit of 10000 requests per month for the entire Skyscraper user base. For this reason the module has been limited to 5 rom scrapins per run. Supply file names on command line to use it or use the --startat and --endat options to narrow it down to 5 roms. The module also doesn't support artwork (neither did it before), as it doesn't distinguish between platform versions for each screenshot. So these had to be left out. But it is still suitable for grabbing textual data if you have a few hard-to-scrape roms.

          Lastly, the docs have been completely overhauled. I've documented all of the command line options and all of the available config.ini options. The scraping modules now have their own documentation and the same goes for language and region support. Please check all of the new and updated docs in the new /docs folder here: https://github.com/muldjord/skyscraper/tree/master/docs

          Thank you to everyone who helped out with suggestions and well-documented bug reports. This new release also coincides with the newly updated RetroPie-Setup script (so be sure to update your RetroPie script prior to updating Skyscraper if you've installed Skyscraper from there) made by @mitu . Thanks for your work on this!

          S 1 Reply Last reply 23 Jan 2019, 19:43 Reply Quote 5
          • M
            mitu Global Moderator
            last edited by 23 Jan 2019, 19:22

            Congratulations on the new release. Thank you for the continuous work on Skyscraper ! Docs should be updated soon, following the updates in the wiki.

            B 1 Reply Last reply 23 Jan 2019, 19:38 Reply Quote 4
            • B
              BuZz administrators @mitu
              last edited by 23 Jan 2019, 19:38

              @mitu @muldjord thanks!

              To help us help you - please make sure you read the sticky topics before posting - https://retropie.org.uk/forum/topic/3/read-this-first

              1 Reply Last reply Reply Quote 1
              • S
                Silent @muldjord
                last edited by 23 Jan 2019, 19:43

                @muldjord said in Versatile C++ game scraper: Skyscraper:

                Now converts ES game list entries to <folder> entries if a .cue or .m3u file is found in a subfolder. This causes the information and artwork to be shown when selecting the folder in ES rather than the .cue or .m3u file inside the subfolder. This is very useful for psx scraping

                Can you give an example? This is the only changelog entry which I don't find too clear.

                Congratulations on the release, too! Looking forward to scraping some PC titles with IGDB =)

                M 1 Reply Last reply 23 Jan 2019, 20:23 Reply Quote 0
                • M
                  muldjord @Silent
                  last edited by muldjord 23 Jan 2019, 20:23

                  @Silent said in Versatile C++ game scraper: Skyscraper:

                  Can you give an example? This is the only changelog entry which I don't find too clear.

                  Check further down :) I elaborate on it there.

                  1 Reply Last reply Reply Quote 0
                  • L
                    LineOf7s
                    last edited by 24 Jan 2019, 00:46

                    I've never had much luck scraping my Amiga games despite all the improvements made recently. With the release of v3.0.0, what's the best way for me to remove anything I've scraped in the past and start again from scratch? FS-UAE Arcade on Windows also scrapes from OpenRetro.org and I'm hoping for something approaching that level of success (which I'm sure SkyScraper can do).

                    M 1 Reply Last reply 24 Jan 2019, 08:15 Reply Quote 0
                    • M
                      muldjord @LineOf7s
                      last edited by 24 Jan 2019, 08:15

                      @LineOf7s Best approach is simply to run the following commands:

                      $ Skyscraper -p amiga -s openretro --refresh
                      $ Skyscraper -p amiga
                      

                      The openretro module has been improved a bit, so you might have luck with that. With that said, I would strongly recommend using the RetroPlay lha files with Amiberry. It turns the Amiga into almost-a-console in that it uses the WHDLoad files. And Skyscraper scrapes these really well with OpenRetro since I've worked behind-the-scenes with some really cool guys to make use of OpenRetro's uuid's for these games.

                      M L 2 Replies Last reply 24 Jan 2019, 10:08 Reply Quote 2
                      • M
                        megaJekan @muldjord
                        last edited by 24 Jan 2019, 10:08

                        Finally figured out how to use skyscrapper in command line mode. Unfortunately, when I scrap single roms, it always writes in log that "Pretend is set" (event if I set it to false in config.ini). So my custom query works, the game is found, but gamelist.xml is not updated. When I don't specify a filename, it goes through all roms per platform and works fine.

                        M M M 3 Replies Last reply 24 Jan 2019, 10:29 Reply Quote 0
                        • M
                          mitu Global Moderator @megaJekan
                          last edited by 24 Jan 2019, 10:29

                          @megaJekan The new 3.0.0 version no longer updates your gamelist.xml after gathering the info, you need to run the scraper again to generate the gamelist(s) - as indicated by @muldjord above - adding just the platform parameter.

                          1 Reply Last reply Reply Quote 1
                          • M
                            muldjord @megaJekan
                            last edited by 24 Jan 2019, 11:04

                            @megaJekan And you can just remove the pretend from your config again. It isn't relevant.

                            H 1 Reply Last reply 24 Jan 2019, 22:17 Reply Quote 0
                            • M
                              megaJekan @megaJekan
                              last edited by 24 Jan 2019, 12:01

                              @mitu said in Versatile C++ game scraper: Skyscraper:

                              @megaJekan The new 3.0.0 version no longer updates your gamelist.xml after gathering the info, you need to run the scraper again to generate the gamelist(s) - as indicated by @muldjord above - adding just the platform parameter.

                              Thanks. I am still using the old version. The problem was that I haven't read the documentation for scraping single roms carefully enough: "It DOES NOT update it in your game list. To do so you need to regenerate the game list by simply leaving out the -s option entirely like so Skyscraper -p [platform]."

                              1 Reply Last reply Reply Quote 0
                              • H
                                Halvhjearne @muldjord
                                last edited by 24 Jan 2019, 22:17

                                @muldjord

                                i just updated and the first thing i noticed is that the menu in retropie setup is now missing the refresh and purge options and it seems it is just set to always refresh the db.

                                also my config.ini did not migrate propper and i had to manually edit it after first run.

                                other than that, it seems to work as expected so far, alltho i would love to see a skipped file for each console, so it does not overwrite and only shows last run, if i choose to use the menu or automated bach script.

                                M M 2 Replies Last reply 24 Jan 2019, 22:19 Reply Quote 0
                                • M
                                  mitu Global Moderator @Halvhjearne
                                  last edited by 24 Jan 2019, 22:19

                                  @Halvhjearne said in Versatile C++ game scraper: Skyscraper:

                                  i just updated and the first thing i noticed is that the menu in retropie setup is now missing the refresh and purge options and it seems it is just set to always refresh the db.

                                  They're not missing, they have their sub-menu now and the purge options have been moved in the Cache options sub-menu.
                                  The refresh part depends on your settings - it has also been moved in the Cache options sub-menu, but if you set it to 'always refresh', then the configuration is kept.

                                  H 1 Reply Last reply 24 Jan 2019, 22:24 Reply Quote 0
                                  • H
                                    Halvhjearne @mitu
                                    last edited by 24 Jan 2019, 22:24

                                    @mitu

                                    ahh, i see ... it just almost crashed, when i went there first time, so i killed it and thought it was a menu error or something.

                                    i guess its processing the db, when you enter that menu.

                                    M 1 Reply Last reply 24 Jan 2019, 22:25 Reply Quote 0
                                    • M
                                      mitu Global Moderator @Halvhjearne
                                      last edited by 24 Jan 2019, 22:25

                                      @Halvhjearne said in Versatile C++ game scraper: Skyscraper:

                                      i guess its processing the db, when you enter that menu.

                                      It's just calculating the space occupied by the cache - how large is your cache folder ?

                                      H 1 Reply Last reply 24 Jan 2019, 22:27 Reply Quote 0
                                      • H
                                        Halvhjearne @mitu
                                        last edited by 24 Jan 2019, 22:27

                                        @mitu

                                        its about 52gb. :)

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          muldjord @Halvhjearne
                                          last edited by 24 Jan 2019, 23:12

                                          @Halvhjearne said in Versatile C++ game scraper: Skyscraper:

                                          also my config.ini did not migrate propper and i had to manually edit it after first run.

                                          What exactly didn't migrate properly? This would be pretty useful to me.

                                          H 1 Reply Last reply 24 Jan 2019, 23:43 Reply Quote 0
                                          1011 out of 1594
                                          • First post
                                            1011/1594
                                            Last post

                                          Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                          Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.

                                            This community forum collects and processes your personal information.
                                            consent.not_received