Skyscraper now officially part of RetroPie, please test

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

Actually, since running the scraper involves exiting Emulationstation and then starting the RetroPie-Setup script, this means using a keyboard (or a SSH section).
However, some of them might not be aware they can stop the process with Ctrl+C (being unfamiliar with the command line). I think we can show them this info in a confirmation dialog before the start of the process.

Good point, I forgot about that.

slutsteg

Can you do so that you do not have to quit emulationstation when you need to download pictures or auto quit and relaunch emulationstation?

muldjord

@slutsteg said in Skyscraper now officially part of RetroPie, please test:

Can you do so that you do not have to quit emulationstation when you need to download pictures or auto quit and relaunch emulationstation?

Nope, can't be done. The reason being that ES writes the in-memory game list back to disc when it quits or you restart the system. And when it does, it will overwrite the game list created by any external scraper.

You could run Skyscraper from command line over SSH and gather data while ES is running, that won't be a problem though. Just remember that if you start generating game lists afterwards, they will disappear when ES is restarted or closed down. I tend to forget this myself from time to time and wondering where the hell the new entries I just created had gone. So yeah, even after all this time, I still forget sometimes.

mitu

@muldjord I've made some modifications to the scriptmodule, incorporating some of the feedback from the last discussion.

I removed the 'Gather for all platforms' entry from the menu, which should be now simpler. The gather action should now be similar to the gamelist generation.

I added a warning when starting scraping

Added the vacuum option under the Cache options and commands and switched the purge command from a straight delete to the corresponding --cache purge:all command.
Other small changes in the verbiage and included files.

I tried to test with having a single Select platforms option (for both gathering and generation), but it turned a bit complicated to pass on the parameters and frankly a bit cumbersone - you have to enter a menu, select something, go back, select another entry, then confirm you want to scrape, etc.

muldjord

@mitu This looks good, great work. It's clearer without the "Gather for all" option and the prompt clears up any confusion about how to stop the process, so thumbs up for that aswell.
Using the now included purge options is definitely a great addition too.

mitu

@muldjord Eh, it's mostly incremental, it's not that much.

I had one idea for Skyscraper - the ES built in scraper has a way to modify the search string when querying the scraping source (the Input button in the image below)

One option when modifying the search string is to use the syntax ID:number and the scraper would perform a search-by-id on the site.

I thought you could add this to the --query syntax:

--query id=<gameID>

and query by the ID. Of course, it will depend on the scraper source, but I think it's easier to enter than - for instance - the MD5/CRC when searching on ScreenScraper.

muldjord

@mitu This is already possible for the screenscraper source I believe - but it's probably called gameid or so. For the file name search based sources it is not though.

EDIT: No, that doesn't seem to work. I wonder why.

mitu

@muldjord I thought you might have implemented this and checked the docs (clihelp.md), but I didn't see it mentioned (and it doesn't appear in the --help output also).
I reported a bug in the V2 WebApi last week (the same forum thread on the ScreenScraper site) where searching by gameid didn't work just with the ID, so it might not have worked before - should work now, I tested it in the ES scraper. I think TGDB should also work with the ID - I know it worked before and I've seen it in the new added to the new ES TGDB scraper version (see here).

muldjord

@mitu, are you aware of this? https://github.com/muldjord/skyscraper/issues/134

I've tested the purge command and it works just fine. It shouldn't even be looking for artwork.xml if the command line is correct for it. I don't use the script myself so I can't test.

mitu

@muldjord No, but I'll take a look.

EDIT: I think I know why it fails - I'll submit a fix tomorrow. The old purge used simple rm commands, the new method uses the Skyscraper's command, but I forgot to make it run as the normal user. Since the RetroPie setup runs as root, the error is probably because that user doesn't have Skyscraper set-up properly.

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

EDIT: I think I know why it fails - I'll submit a fix tomorrow. The old purge used simple rm commands, the new method uses the Skyscraper's command, but I forgot to make it run as the normal user. Since the RetroPie setup runs as root, the error is probably because that user doesn't have Skyscraper set-up properly.

Sounds plausible. Thanks for looking into it.

2fst4u

@mitu hi there, submitter of that issue here. Thanks for looking into it and good to hear that it shouldn't be too difficult (relatively) to sort out.

In the meantime, is there anything manual that I can try to make it work or would it be best to just wait for a fix?

mitu

@2fst4u Hello and thank you for the bug report.
I've submitted a fix for the problem you reported (https://github.com/RetroPie/RetroPie-Setup/pull/2658) - I think you should wait to be added to the RetroPie-Setup script and then just update to get the fix.

2fst4u

@mitu ok thank you. How can I check that it's ready to be updated? Will you let us know?

mitu

@2fst4u Sure, I'll add a comment when the fix is added.

ZestyTS

Just found this thread, I believe I'm having trouble with the scriptmodule.

@mitu

Using Skyscraper (3.1.3) on RetroPie (4.4.9) I'm unable to select some of the options for Skyscraper from the RetroPie setup, either through command line or ssh. Example, upon loading up the GUI, clicking gather resources, I'm unable to select any of the platforms for resource gathering. I've tried reinstalling SkyScraper and RetroPie, but that hasn't fixed the issue.

Like if we look at the picture you uploaded earlier, I'm unable to select "Gather and cache resources" it just refreshes the screen as opposed to bringing me to a new screen about starting the scraping process.

mitu

@ZestyTS said in Skyscraper now officially part of RetroPie, please test:

Like if we look at the picture you uploaded earlier, I'm unable to select "Gather and cache resources" it just refreshes the screen as opposed to bringing me to a new screen about starting the scraping process

That's not an action, it's just a heading for the actions beneath it. It's supposed to act like that. Use the Gather Resources - the next menu beneath it.

ZestyTS

@mitu

I see that explains why the page refreshes. So I tried "Gather resources" before but I'm unable to check/select any of the platforms, I just tried it again and I'm still having the issue. I guess, the easy question is, "what are the key combinations required to select an option?"

mitu

@ZestyTS The Space key acts as a toggle/checkbox, that's how you select.

iainjh

Hi! when will that purge issue be addressed please? is there a manual workaround? My image is full up and I have no way to delete.

Its a fantastic scraper btw! - sorry for hassle