RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Skyscraper now officially part of RetroPie, please test

    Scheduled Pinned Locked Moved Ideas and Development
    skyscraperscraperretropie-setupoptionalexperimental
    275 Posts 28 Posters 109.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • mituM
      mitu Global Moderator @yyandros
      last edited by

      @yyandros You can try asking for the US region when scraping from the command lina, with the --region us parameter.

      1 Reply Last reply Reply Quote 3
      • muldjordM
        muldjord @yyandros
        last edited by muldjord

        @yyandros I just updated it yesterday to support auto-region detection, so if the filenames have (USA) in them, it will automatically try to grab the Genesis versions. Otherwise, if you wish to force a certain region for it (megadrive and genesis are usually seen as the same platform) you can use the "--region us" or set it in config.ini with "region=us" under the "[main]" section. Remember to uncomment the lines by removing the "#" in front of them, otherwise they won't apply.

        1 Reply Last reply Reply Quote 3
        • muldjordM
          muldjord
          last edited by

          @mitu It has been asked that the "mobygames" scraping module is removed from the RetroPie script. This is a fair point, as the module is limited to 25 files at a time and only meant to be used for single file command line provided scrapings. Thus it is both pretty useless to have in the script and also not obvious to the user why it won't work for most scrapings.

          I would therefore ask for it to be removed, unless you have an idea for how we can handle this and force the use of it to be restricted to just 1 file at a time.

          The reason for the limit on the "mobygames" module is that the key I have for it is limiting the ENTIRE Skyscraper user base. So if one user is using it when you want to use it, it will run into issues. It is therefore imperative that it be used only for single game scrapings. It's basically meant to be used only for those few singular roms you have left that the other sources won't provide data for. :)

          mituM 1 Reply Last reply Reply Quote 0
          • mituM
            mitu Global Moderator @muldjord
            last edited by

            @muldjord Sure, I'll prepare a modification this week-end and send a PR. But, as I'm sure you know, this will not make it inaccessible to the CLI.

            muldjordM 1 Reply Last reply Reply Quote 0
            • muldjordM
              muldjord @mitu
              last edited by muldjord

              @mitu Great. And yes, that's fine. It's supposed to be used from CLI to the degree I mentioned. But this way at least users won't get frustrated that it seemingly just quits every time they try to use it. When they use it from the CLI they will be able to ready the yellow "quit text" that tells why it's quitting.

              mituM 1 Reply Last reply Reply Quote 0
              • mituM
                mitu Global Moderator @muldjord
                last edited by mitu

                @muldjord said in Skyscraper now officially part of RetroPie, please test:

                When they use it from the CLI they will be able to ready the yellow "quit text" that tells why it's quitting.

                Yes, I was thinking about that. Although I've put a small sleep delay at the end of the scraping command (when run from the GUI), I don't think that would be enough to spot the message on a first run.

                muldjordM 1 Reply Last reply Reply Quote 0
                • muldjordM
                  muldjord @mitu
                  last edited by muldjord

                  @mitu Yes, there's not much point in having it in the GUI as users can't restrict how many roms they wish to scrape. From CLI you can provide a single file or use the startat and endat options. There it makes sense.

                  EDIT: Btw, should we consider moving it back to Optional? I seem to recall everything being fixed including the files being copied for the binary release?

                  S 1 Reply Last reply Reply Quote 0
                  • S
                    Silent @muldjord
                    last edited by Silent

                    Would it be then possible to add a "Scrape specific ROM" option in setup script, similar to "Scrape specific systems"? If for Mobygames specifically it had an annoyingly low limit (like, 3-5 games at once max) it should make it slow enough to deter people from trying to scrape stuff from Mobygames en masse.

                    EDIT:
                    That shouldn't have been marked as a reply to a specific post. Ugh.

                    mituM 1 Reply Last reply Reply Quote 0
                    • mituM
                      mitu Global Moderator @Silent
                      last edited by mitu

                      @Silent That would be too much hassle - adding the option to scrape one ROM - with little use. As for the MobyGames part, I don't see a point in using it (either through the GUI via the command line) if they only allow such low number of scrapes.

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        Silent @mitu
                        last edited by

                        @mitu said in Skyscraper now officially part of RetroPie, please test:

                        I don't see a point in using it

                        They seem to have the best base for PC games, which may come in handy soon when Moonlight script gets integrated into RetroPie.

                        But yeah, I guess invoking Skyscraper from CLI is also fine.

                        1 Reply Last reply Reply Quote 0
                        • S
                          Silent
                          last edited by

                          So I've seen that Mobygames is now officially removed from the setup script and been thinking on how to make custom queries as accessible as possible, since right now there is no really easy way to find out what exactly is the setup script invoking when using Skyscraper (yes, you have "comnandline used" but more often than not you need to race against the UI to be able to read it), and I came up with an idea I'd like to discuss before implementing it:

                          "Scrape with custom parameters" option in Advanced options, which opens a text box dialog prefilled with existing commandline options, like
                          -o \whatevever\directory -p psx -s screenscraper

                          and then the user is free to change any settings, then run scraping. This way any possible hurdles with using Skyscraper installed by setup script would be resolved and you could be 100% certain it is doing to scrape to/from exactly the same directories as setup's UI scraping.

                          muldjordM 1 Reply Last reply Reply Quote 0
                          • muldjordM
                            muldjord @Silent
                            last edited by

                            @Silent I don't know, it seems like a cumbersome way to avoid the cli. I'd much rather people just learn to use the cli instead. The "mobygames" module won't ever be used widely in any case as I've explained. You don't have the advantage of tab completion with a free text field for instance. Btw, thegamesdb is pretty good for PC games aswell, just so you know. But if you've run out of requests, that's why you aren't getting any hits from it.

                            S 1 Reply Last reply Reply Quote 0
                            • muldjordM
                              muldjord
                              last edited by muldjord

                              2.9.5 is coming up soon. It's a whopper of a release implementing some of the major features I've been wanting to work on for a while.
                              For instance, it now allows you to do "--purgedb vacuum" which removes all cached resources not related to your romset.
                              I've also implemented the "aliasMap.csv" file which allows you to add aliases for any filename in your romsets. This helps if you are having trouble scraping single games.
                              The new "--symlink" option also made it in. This forces Skyscraper to symlink videos from the cache instead of copying them. Useful for saving a bit of space.
                              The "esgamelist" module is also pretty cool. It's basically a scraping module that scrapes data from an existing EmulationStation gamelist.xml file (including artwork and videos). This means that you can now use any other scraper to generate a gamelist.xml file and then import it into the Skyscraper cache.

                              I'm not sure if I'll release it before of after christmas. I kind of feel like postponing it until after for the sake of piece of mind in case any bugs have snuck in.

                              1 Reply Last reply Reply Quote 2
                              • S
                                Silent @muldjord
                                last edited by

                                @muldjord said in Skyscraper now officially part of RetroPie, please test:

                                I don't know, it seems like a cumbersome way to avoid the cli.

                                It's more about consistency between setup's GUI vs CLI outputs. If I run it with any arguments whatsoever I get wildly different output to what I get from a setup script (because of its implicit directory commandline arguments). Having pre-filled CLI makes it hard to go wrong.

                                muldjordM 1 Reply Last reply Reply Quote 0
                                • muldjordM
                                  muldjord @Silent
                                  last edited by muldjord

                                  @Silent I'm not really sure what you mean by "wildly different output". The gui uses a few options to make it better suited for use in a GUI. Other than those options (which basically just skips the initial questions about overwriting and skipping) the output is similar to running it on command line.

                                  S 1 Reply Last reply Reply Quote 0
                                  • S
                                    Silent @muldjord
                                    last edited by

                                    @muldjord said in Skyscraper now officially part of RetroPie, please test:

                                    I'm not really sure what you mean by "wildly different output"

                                    Paths, for once. You can get them from reading the source code or from pausing the scraper at a right moment (at "COMMANDLINE USED:" line), but I feel like it's still not foolproof enough.

                                    Don't get me wrong, I am perfectly able to use it from CLI as is, but I am trying to think of as user friendly option as possible.

                                    muldjordM 1 Reply Last reply Reply Quote 0
                                    • muldjordM
                                      muldjord @Silent
                                      last edited by

                                      @Silent So basically you want the default command line options used by the GUI to be configurable? I'm not against that although some of them are locked. RetroPie doesn't allow questions during running a script for instance, so --unattend is locked. It could make sense though. Some of them would be locked, but it could have some "extras" that you could provide yourself. But I'm also thinking that you already have the config.ini for that purpose. Which is actually better.

                                      S 1 Reply Last reply Reply Quote 1
                                      • S
                                        spurlingD
                                        last edited by spurlingD

                                        @muldjord i have installed it with retropie but when i type in skyscraper it not do anything

                                        muldjordM 1 Reply Last reply Reply Quote 0
                                        • muldjordM
                                          muldjord @spurlingD
                                          last edited by

                                          @spurlingD If you typed in "skyscraper" that's why. The command is "Skyscraper".

                                          S 1 Reply Last reply Reply Quote 0
                                          • S
                                            Silent @muldjord
                                            last edited by

                                            @muldjord said in Skyscraper now officially part of RetroPie, please test:

                                            @Silent So basically you want the default command line options used by the GUI to be configurable?

                                            Kinda. Cases where it'd be useful would be any "odd" stuff you'd want to do, like:

                                            • scrape single rom (afaik you cannot do that with config.ini now)
                                            • scrape a range of roms (you can use startat and endat in config.ini but it's awkward to edit every time)
                                            • scrape with a query (probably impossible from config.ini?)
                                            • easy access to any upcoming features whcih didn't make it to GUI yet (like now - vacuuming, ES scraper)
                                            muldjordM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.