Skyscraper now officially part of RetroPie, please test

mitu · 18 Mar 2019, 07:31

@2fst4u Hello and thank you for the bug report.
I've submitted a fix for the problem you reported (https://github.com/RetroPie/RetroPie-Setup/pull/2658) - I think you should wait to be added to the RetroPie-Setup script and then just update to get the fix.

2fst4u · 18 Mar 2019, 07:49

@mitu ok thank you. How can I check that it's ready to be updated? Will you let us know?

mitu · 18 Mar 2019, 08:23

@2fst4u Sure, I'll add a comment when the fix is added.

ZestyTS · 18 Mar 2019, 20:23

Just found this thread, I believe I'm having trouble with the scriptmodule.

@mitu

Using Skyscraper (3.1.3) on RetroPie (4.4.9) I'm unable to select some of the options for Skyscraper from the RetroPie setup, either through command line or ssh. Example, upon loading up the GUI, clicking gather resources, I'm unable to select any of the platforms for resource gathering. I've tried reinstalling SkyScraper and RetroPie, but that hasn't fixed the issue.

Like if we look at the picture you uploaded earlier, I'm unable to select "Gather and cache resources" it just refreshes the screen as opposed to bringing me to a new screen about starting the scraping process.

mitu · 18 Mar 2019, 20:26

@ZestyTS said in Skyscraper now officially part of RetroPie, please test:

Like if we look at the picture you uploaded earlier, I'm unable to select "Gather and cache resources" it just refreshes the screen as opposed to bringing me to a new screen about starting the scraping process

That's not an action, it's just a heading for the actions beneath it. It's supposed to act like that. Use the Gather Resources - the next menu beneath it.

ZestyTS · 18 Mar 2019, 21:43

@mitu

I see that explains why the page refreshes. So I tried "Gather resources" before but I'm unable to check/select any of the platforms, I just tried it again and I'm still having the issue. I guess, the easy question is, "what are the key combinations required to select an option?"

mitu · 18 Mar 2019, 21:52

@ZestyTS The Space key acts as a toggle/checkbox, that's how you select.

iainjh · 18 Mar 2019, 22:22

Hi! when will that purge issue be addressed please? is there a manual workaround? My image is full up and I have no way to delete.

Its a fantastic scraper btw! - sorry for hassle

ZestyTS · 18 Mar 2019, 22:53

@mitu Thanks for getting back to me, that works over ssh, but not with my keyboard plugged into the raspberry pi, so I'll have to figure out what that's configured as, but thanks!

mitu · 19 Mar 2019, 05:18

@iainjh Use the command line parameters to purge/vacuum - https://github.com/muldjord/skyscraper/blob/master/docs/CLIHELP.md.

mitu · 19 Mar 2019, 16:51

@2fst4u @iainjh The fix has been added to the module in the project's repository - update your RetroPie-Setup script and then try again the purge/vacuum functions to see if you still get the error.

2fst4u · 19 Mar 2019, 17:15

@mitu awesome that works fine now. Thank you very much for your help.

mitu · 19 Mar 2019, 17:18

@2fst4u Alright then - thank you again for reporting the bug.
If you find any bugs/problems/questions with the menu/scriptmodule, please use this topic, I'm always notified when someone posts.

iainjh · 20 Mar 2019, 23:46

Lovely job, working beautifully now. thank you!

2fst4u · 24 Mar 2019, 07:14

Hi, just a question out of curiosity:

What happens when you scrape with a second source after having already scraped everything once? Does Skyscraper only scrape for information/files that were missing, or does it re-scrape everything? I see in the docs it mentions a couple of times about using a second source to fill in gaps, but I can't find anywhere what it actually does when you use a second source.

I also understand there is a way to set priorities for scraping sources. If the answer to the above question is "it only scrapes missing information", then what does this priority do? Does it overwrite data from a lower priority source and leave higher priority data untouched?

muldjord · 24 Mar 2019, 08:17

@2fst4u said in Skyscraper now officially part of RetroPie, please test:

What happens when you scrape with a second source after having already scraped everything once? Does Skyscraper only scrape for information/files that were missing, or does it re-scrape everything? I see in the docs it mentions a couple of times about using a second source to fill in gaps, but I can't find anywhere what it actually does when you use a second source.

Please read this: https://github.com/muldjord/skyscraper/blob/master/docs/USECASE.md

If you gather resources from the same source twice and it didn't finish all of your roms the first time around, it will load all of the ones you already gathered for with that source from the cache until it reaches one it doesn't have cached data for. Then it will start fetching it from the source directly. So in a way, it "skips" the ones you already gathered for using said source. You can tell from the From cache: YES/NO in the output. But it's much easier to use the --startat and --endat options so you don't have to wait for it to load each entry from cache again.

Concerning gathering using a /second/ source, it will regather for /every/ file. Skyscraper can cache each resource once per source. So when gathering from a second source it will cache all data once more.

The clever part of this is that you can at /any/ time change anything in your source priorities or your artwork config, and even decide to change your frontend to attractmode. And all you have to do is run a game list generation run again, and you will have a new game list for that frontend using the new configs you've applied. No need to gather new data at all because you already have all of it cached.

I also understand there is a way to set priorities for scraping sources. If the answer to the above question is "it only scrapes missing information", then what does this priority do? Does it overwrite data from a lower priority source and leave higher priority data untouched?

There is no overwriting going on anywhere. The priorities are used /only/ when generating the game lists. If you prefer the screenshots from a particular source, you'd prioritize that higher than other sources for the screenshots and so on. You can do this will all resource types for each platform.

2fst4u · 24 Mar 2019, 08:20

@muldjord thanks, good to know. One more question about priorities then, what happens to your gamelist.xml if you don't set priorities but you have data from multiple sources scraped?

muldjord · 24 Mar 2019, 08:28

@2fst4u said in Skyscraper now officially part of RetroPie, please test:

@muldjord thanks, good to know. One more question about priorities then, what happens to your gamelist.xml if you don't set priorities but you have data from multiple sources scraped?

Then it uses the default priorities.xml. If the source isn't mentioned it uses timestamps, so the latest gathered resource for any type has priority over the others.

2fst4u · 24 Mar 2019, 23:04

@muldjord oh cool that's awesome.

frezeen · 28 Mar 2019, 01:43

~~the windows version used with cmder look perfect~~
https://cmder.net/

~~can u update the windows version to last one?~~
~~or can u write a guide for to compile it for windows. i dont ask everytime for updated version~~~~

edit: forgot all , i success to compile it with qt 5.

Starting scraping run on 2825 files using 1 threads.
Sit back, relax and let me do the work! :)

ScreenScraper APIv2 returned invalid XML
First 256 characters of answer was:

#1/2825 (T1) Pass 1 ---- Game '005' not found :( ----

0/1/1
Elapsed time : 00:00:04
Est. time left : 03:11:25

User wants to quit, trying to exit nicely. This can take a few seconds depending on how many threads you have running...
Request timed out, aborting request...

arcadedb work great, screenscraper got error.
is this a windows problem or what?
version 3.1.5
very very thanks for your work