Skyscraper now officially part of RetroPie, please test

Silent

Would it be then possible to add a "Scrape specific ROM" option in setup script, similar to "Scrape specific systems"? If for Mobygames specifically it had an annoyingly low limit (like, 3-5 games at once max) it should make it slow enough to deter people from trying to scrape stuff from Mobygames en masse.

EDIT:
That shouldn't have been marked as a reply to a specific post. Ugh.

mitu

@Silent That would be too much hassle - adding the option to scrape one ROM - with little use. As for the MobyGames part, I don't see a point in using it (either through the GUI via the command line) if they only allow such low number of scrapes.

Silent

@mitu said in Skyscraper now officially part of RetroPie, please test:

I don't see a point in using it

They seem to have the best base for PC games, which may come in handy soon when Moonlight script gets integrated into RetroPie.

But yeah, I guess invoking Skyscraper from CLI is also fine.

Silent

So I've seen that Mobygames is now officially removed from the setup script and been thinking on how to make custom queries as accessible as possible, since right now there is no really easy way to find out what exactly is the setup script invoking when using Skyscraper (yes, you have "comnandline used" but more often than not you need to race against the UI to be able to read it), and I came up with an idea I'd like to discuss before implementing it:

"Scrape with custom parameters" option in Advanced options, which opens a text box dialog prefilled with existing commandline options, like
-o \whatevever\directory -p psx -s screenscraper

and then the user is free to change any settings, then run scraping. This way any possible hurdles with using Skyscraper installed by setup script would be resolved and you could be 100% certain it is doing to scrape to/from exactly the same directories as setup's UI scraping.

muldjord

@Silent I don't know, it seems like a cumbersome way to avoid the cli. I'd much rather people just learn to use the cli instead. The "mobygames" module won't ever be used widely in any case as I've explained. You don't have the advantage of tab completion with a free text field for instance. Btw, thegamesdb is pretty good for PC games aswell, just so you know. But if you've run out of requests, that's why you aren't getting any hits from it.

muldjord

2.9.5 is coming up soon. It's a whopper of a release implementing some of the major features I've been wanting to work on for a while.
For instance, it now allows you to do "--purgedb vacuum" which removes all cached resources not related to your romset.
I've also implemented the "aliasMap.csv" file which allows you to add aliases for any filename in your romsets. This helps if you are having trouble scraping single games.
The new "--symlink" option also made it in. This forces Skyscraper to symlink videos from the cache instead of copying them. Useful for saving a bit of space.
The "esgamelist" module is also pretty cool. It's basically a scraping module that scrapes data from an existing EmulationStation gamelist.xml file (including artwork and videos). This means that you can now use any other scraper to generate a gamelist.xml file and then import it into the Skyscraper cache.

I'm not sure if I'll release it before of after christmas. I kind of feel like postponing it until after for the sake of piece of mind in case any bugs have snuck in.

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I don't know, it seems like a cumbersome way to avoid the cli.

It's more about consistency between setup's GUI vs CLI outputs. If I run it with any arguments whatsoever I get wildly different output to what I get from a setup script (because of its implicit directory commandline arguments). Having pre-filled CLI makes it hard to go wrong.

muldjord

@Silent I'm not really sure what you mean by "wildly different output". The gui uses a few options to make it better suited for use in a GUI. Other than those options (which basically just skips the initial questions about overwriting and skipping) the output is similar to running it on command line.

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

I'm not really sure what you mean by "wildly different output"

Paths, for once. You can get them from reading the source code or from pausing the scraper at a right moment (at "COMMANDLINE USED:" line), but I feel like it's still not foolproof enough.

Don't get me wrong, I am perfectly able to use it from CLI as is, but I am trying to think of as user friendly option as possible.

muldjord

@Silent So basically you want the default command line options used by the GUI to be configurable? I'm not against that although some of them are locked. RetroPie doesn't allow questions during running a script for instance, so --unattend is locked. It could make sense though. Some of them would be locked, but it could have some "extras" that you could provide yourself. But I'm also thinking that you already have the config.ini for that purpose. Which is actually better.

spurlingD

@muldjord i have installed it with retropie but when i type in skyscraper it not do anything

muldjord

@spurlingD If you typed in "skyscraper" that's why. The command is "Skyscraper".

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@Silent So basically you want the default command line options used by the GUI to be configurable?

Kinda. Cases where it'd be useful would be any "odd" stuff you'd want to do, like:

scrape single rom (afaik you cannot do that with config.ini now)
scrape a range of roms (you can use startat and endat in config.ini but it's awkward to edit every time)
scrape with a query (probably impossible from config.ini?)
easy access to any upcoming features whcih didn't make it to GUI yet (like now - vacuuming, ES scraper)

muldjord

@Silent For your examples, why not just use the command line? I don't see how a free text field would provide a user friendly way of running commands, quite the contrary. You don't have tab completion in the GUI, so you'd have to know the exact filename to do what you want (and not do any spelling errors). And for that you'd need to look them up on the command line beforehand. In which case you could just run the command there. If the user knows about these features you describe, they have already checked the output of --help where the commands are listed. So they even know how to run the executable from the correct location. So yeah, I don't think this makes much sense. I'd much rather put in some better help text telling people how to actually run it from command line. I mean, you've already exited EmulationStation to even open up the GUI. So the command line is right there.

spurlingD

@muldjord i have typed Skyscraper and it say command not found

mitu

@spurlingD If you installed Skyscraper using the RetroPie scriptmodule, then the command is located in /opt/retropie/supplementary/skyscraper/Skyscraper. If you add the installation folder to your $PATH, then you don't have to type the full path to the command.

Silent

@muldjord said in Skyscraper now officially part of RetroPie, please test:

For your examples, why not just use the command line?

Figured I'd post the way I "solved" my personal gripe I had with this approach:

I never use "Use ROM folders" so I could just add those to config.ini as defaults:

inputFolder="/home/pi/RetroPie/roms"
gamelistFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"

and now I don't need to provide those as -o and -g arguments, like RetroPie-Setup script does.

Tested it when scraping for a single ROM and it worked flawlessly!

EDIT:
In setup like this, I hope that commandline parameters take precedence over what is in INI file? Just in case RetroPie-Setup ever provides paths different to these.

EDIT2:
Does --pretend imply --refresh? When scraping for single ROMs it's always redownloading from the website, bypassing the cache.

spurlingD

@mitu how do i get it show it not in text from but the menu

mitu

@spurlingD I don't understand what you mean. Can you give an example ?

spurlingD

@mitu like this 1539341719679-2d4d68da-bea6-4f9c-a44f-bee87d3d5ecd-image.png