Skyscraper now officially part of RetroPie, please test

Hyperlord

@mitu Thanks for the fast reply!
I'll try this at the weekend since the scraping will take approx 2 days :-)

After I installed the skyscraper from the retropi-setup and moved the db I ran a quick scraping to see if it's still working. I guess I started this from the menu but currently I'm not sure.

But my questions has been answered so far by now ...

brimby

Is there a subreddit yet? I tried to find one but just found a lot of stuff about skyscrapers, heh.

If there is no subreddit, creating one might be something to consider so that people can share their opinions about priorities.xml, artwork.xml, or post their local dbs.

Or maybe there already is some other community hub in existence, like a discord or slack or idk? Something besides this single retropie.org.uk thread?

muldjord

@brimby yes, try r/retropie

yyandros

I've got a question about scraping images for Mega Drive/Genesis...is there a way to tell the scraper to grab the Genesis box art as opposed to the Mega Drive box art? I'm an absolute newb when it comes to this stuff, so I'm assuming I'm overlooking a way to do it since this scraper is really robust. Also, great work! This is by far my favorite scraper.

mitu

@yyandros You can try asking for the US region when scraping from the command lina, with the --region us parameter.

muldjord

@yyandros I just updated it yesterday to support auto-region detection, so if the filenames have (USA) in them, it will automatically try to grab the Genesis versions. Otherwise, if you wish to force a certain region for it (megadrive and genesis are usually seen as the same platform) you can use the "--region us" or set it in config.ini with "region=us" under the "[main]" section. Remember to uncomment the lines by removing the "#" in front of them, otherwise they won't apply.

muldjord

@mitu It has been asked that the "mobygames" scraping module is removed from the RetroPie script. This is a fair point, as the module is limited to 25 files at a time and only meant to be used for single file command line provided scrapings. Thus it is both pretty useless to have in the script and also not obvious to the user why it won't work for most scrapings.

I would therefore ask for it to be removed, unless you have an idea for how we can handle this and force the use of it to be restricted to just 1 file at a time.

The reason for the limit on the "mobygames" module is that the key I have for it is limiting the ENTIRE Skyscraper user base. So if one user is using it when you want to use it, it will run into issues. It is therefore imperative that it be used only for single game scrapings. It's basically meant to be used only for those few singular roms you have left that the other sources won't provide data for. :)

mitu

@muldjord Sure, I'll prepare a modification this week-end and send a PR. But, as I'm sure you know, this will not make it inaccessible to the CLI.

muldjord

@mitu Great. And yes, that's fine. It's supposed to be used from CLI to the degree I mentioned. But this way at least users won't get frustrated that it seemingly just quits every time they try to use it. When they use it from the CLI they will be able to ready the yellow "quit text" that tells why it's quitting.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

When they use it from the CLI they will be able to ready the yellow "quit text" that tells why it's quitting.

Yes, I was thinking about that. Although I've put a small sleep delay at the end of the scraping command (when run from the GUI), I don't think that would be enough to spot the message on a first run.

muldjord

@mitu Yes, there's not much point in having it in the GUI as users can't restrict how many roms they wish to scrape. From CLI you can provide a single file or use the startat and endat options. There it makes sense.

EDIT: Btw, should we consider moving it back to Optional? I seem to recall everything being fixed including the files being copied for the binary release?

Silent

Would it be then possible to add a "Scrape specific ROM" option in setup script, similar to "Scrape specific systems"? If for Mobygames specifically it had an annoyingly low limit (like, 3-5 games at once max) it should make it slow enough to deter people from trying to scrape stuff from Mobygames en masse.

EDIT:
That shouldn't have been marked as a reply to a specific post. Ugh.

mitu

@Silent That would be too much hassle - adding the option to scrape one ROM - with little use. As for the MobyGames part, I don't see a point in using it (either through the GUI via the command line) if they only allow such low number of scrapes.

Silent

@mitu said in Skyscraper now officially part of RetroPie, please test:

I don't see a point in using it

They seem to have the best base for PC games, which may come in handy soon when Moonlight script gets integrated into RetroPie.

But yeah, I guess invoking Skyscraper from CLI is also fine.

Silent

So I've seen that Mobygames is now officially removed from the setup script and been thinking on how to make custom queries as accessible as possible, since right now there is no really easy way to find out what exactly is the setup script invoking when using Skyscraper (yes, you have "comnandline used" but more often than not you need to race against the UI to be able to read it), and I came up with an idea I'd like to discuss before implementing it:

"Scrape with custom parameters" option in Advanced options, which opens a text box dialog prefilled with existing commandline options, like
-o \whatevever\directory -p psx -s screenscraper

and then the user is free to change any settings, then run scraping. This way any possible hurdles with using Skyscraper installed by setup script would be resolved and you could be 100% certain it is doing to scrape to/from exactly the same directories as setup's UI scraping.

muldjord

@Silent I don't know, it seems like a cumbersome way to avoid the cli. I'd much rather people just learn to use the cli instead. The "mobygames" module won't ever be used widely in any case as I've explained. You don't have the advantage of tab completion with a free text field for instance. Btw, thegamesdb is pretty good for PC games aswell, just so you know. But if you've run out of requests, that's why you aren't getting any hits from it.

muldjord

2.9.5 is coming up soon. It's a whopper of a release implementing some of the major features I've been wanting to work on for a while.
For instance, it now allows you to do "--purgedb vacuum" which removes all cached resources not related to your romset.
I've also implemented the "aliasMap.csv" file which allows you to add aliases for any filename in your romsets. This helps if you are having trouble scraping single games.
The new "--symlink" option also made it in. This forces Skyscraper to symlink videos from the cache instead of copying them. Useful for saving a bit of space.
The "esgamelist" module is also pretty cool. It's basically a scraping module that scrapes data from an existing EmulationStation gamelist.xml file (including artwork and videos). This means that you can now use any other scraper to generate a gamelist.xml file and then import it into the Skyscraper cache.

I'm not sure if I'll release it before of after christmas. I kind of feel like postponing it until after for the sake of piece of mind in case any bugs have snuck in.

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I don't know, it seems like a cumbersome way to avoid the cli.

It's more about consistency between setup's GUI vs CLI outputs. If I run it with any arguments whatsoever I get wildly different output to what I get from a setup script (because of its implicit directory commandline arguments). Having pre-filled CLI makes it hard to go wrong.

muldjord

@Silent I'm not really sure what you mean by "wildly different output". The gui uses a few options to make it better suited for use in a GUI. Other than those options (which basically just skips the initial questions about overwriting and skipping) the output is similar to running it on command line.

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I'm not really sure what you mean by "wildly different output"

Paths, for once. You can get them from reading the source code or from pausing the scraper at a right moment (at "COMMANDLINE USED:" line), but I feel like it's still not foolproof enough.

Don't get me wrong, I am perfectly able to use it from CLI as is, but I am trying to think of as user friendly option as possible.