Skyscraper now officially part of RetroPie, please test

Silent

Since artwork.xml and config.ini can be edited from the GUI already, how about also allowing to edit aliasMap.csv? For completeness.

muldjord

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

So you'd enter that menu and choose any of the scraping modules from inside it and it would do 1 run gathering resources from that 1 module, then return to the menu. Then go back to the main menu and choose "Generate ES game list(s) from cache".

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

I don't understand :). The first 2 choices initiate a scrape (gather), the 4th option is just a configuration toggle. So we would rename the entries

Gather and cache resources for all systems - runs the scrape for all systems, with the source from 4.
Gather and cache resources for chosen systems - runs the scrape for selected systems, with the source from 4.

Is that what you meant ?

EDIT: something like

mitu

@Silent That would be a good candidate; while an 'advanced' feature, it doesn't require any other CLI operations the user must perform.
I see there is a sample (empty one) provided, I'll have to check during the install to not overwrite it. Right now, I think you may be the only user that is using this feature :).

muldjord

@mitu Yes, I forgot that's how they work, sorry.

I think it should be like this for the best workflow and to best split the two things apart (gather and generate):

HEADLINE: "Gather and cache resources:" (just a non-functional headline option)
"Source: ScreensScraper (Online)"
- All of the sources as they are now
"Gather for: All systems"
"Gather for: Chosen systems"
"Cache options -->"
- Force cache refresh
- Cache covers, cache screenshots etc... (Enabled)
- Vacuum cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb vacuum" command)
- Delete cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb all" command)
HEADLINE: "Game list generation:" (just a non-functional headline option)
Generate EmulationStation game list
Generate options -->
- ROM Names (Source name)
- Remove bracket info (Disabled)
- Use Rom folders for game list and media (Enabled)
HEADLINE: "General:" (just a non-functional headline option)
Advanced Skyscraper options -->
- Enable videos
- Edit config.ini
- Edit artwork.xml
- Edit aliasMap.csv

mitu

This post is deleted!

mitu

@muldjord So here's a draft version

Re-organized the menu a bit, per your layout

the scraping source and Cache options are next to each other, since both are sub-menus opening up
the cache options have the actions of vacuum/purge system/purge all (now using Skyscraper instead of manually erasing files and moved from advanced options).
added the video download toggle inside the cache options
advanced options now only have the config files editing.

One question for game generation - do --videos needs to be enabled when scraping from localdb to add the video tag ?

muldjord

@mitu

Looks good!

--videos should be moved to general though as per my suggestion, as it's not related to caching any more than it's related to game list generation. It's not like the --nocovers and such options. It's an option related to Skyscraper in general. So to answer your question, yes, it needs to be enabled for videos to be used in the game list. And also for videos to be cached.

mitu

@muldjord Yeah, the video stuff does belong to both option sections, will move it back - adding it in both sections wouldn't make sense. The new version is mostly done, just need to test it a bit more.

muldjord

@mitu I would also argue to remove the "Gather for All systems" entirely, and rename the "Gather for Chosen systems" to "Gather for system(s)". That will align better with the "Generate game list(s)" option further down.

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

EDIT: Btw, when all this is done I think I'll move the "Simple Mode" of Skyscraper down the ranks in the docs, and instead simply refer to the RetroPie wiki and this script. I'm not gonna remove it completely just yet, just in case non-RetroPie'ers use it. But I feel we're reaching a point where it's becoming obsoleted by this script.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

I though about it - it would make the 2 phases of the scraping symmetrical and it doesn't quite fit the Skyscraper's process (I see it mostly as a one-shot scraping option for quickly getting something for all systems).
I still think it has its usefulness for first time scraping - when you've got a few of your systems populated and you're just wanting to get the info in Emulationstation. Gradually, as you add more ROMs, it's probably not going to get much use since you'd be scraping just one system at a time, switching sources and checking for missing info/roms.
I try to find a way add an 'All' toggle in one gathering option, but dialog won't let me combine radio buttons with a multiple selection list.

muldjord

@mitu Yes, I can see how it's useful certainly. It's only because I feel like is mismatches (and clutters it up a bit) with how there's only one option for game list generation. I'm ok with having it there, I just hope people don't get confused by the difference.

muldjord

@mitu Just to summarize the changes you need to pay attention to in regards to the script when moving from 2.x to 3.x

--refreshhas become --cache refresh but I'll leave --refresh in 3.x for the time being to ease migration. It will be removed at some point in the future and I'll be sure to let you know when I do remove it.
The --purgedb vacuum option has become --cache vacuum.
The --purgedb all option has become --cache purge:all which will remove all resources for the selected platform.
- I'll let you decide whether it makes sense to use these for the time being as they are not compatible with 2.x
- You could just keep the old cache clearing options from the old script for the time being and switch to the new ones when users have migrated. But please note the next point, that dbs has changed.
The ~/.skyscraper/dbs folder is now ~/.skyscraper/cache. This is auto-migrated when Skyscraper 3.x is run for the first time after updating (it simply moves dbs to cache).

If I think of other things, I will update this post.

mitu

@muldjord (I'll reply here, instead of the other topic)

Most of the changes are ok (as far as the package is concerned), as long as you have some backwards compatibility. The script could certainly detect the version and prompt the user to upgrade when it detects an incompatible version, but I wouldn't impose that.
SScraper has a Upgrade to the latest version option, maybe we can add that after the Advanced options, since Skyscraper adds new features and fixes faster and I have the feeling some users don't have the habit or just don't want to break things doing an full upgrade once in a while.

For now the only thing I've yet to figure out (just because I didn't have the time) is the cache location chance, since the script would migrate it on first install (for existing users).

I think that as long as you keep --refresh around, we can introduce the purgedb options later (though I'll have to cut vacuum or make it conditional, depending on the version).

So, one request from me - can you add a --version parameter ? Right now the script just parses the --help invocation, but it would be nice to provide a shorter version :).

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

So, one request from me - can you add a --version parameter ? Right now the script just parses the --help invocation, but it would be nice to provide a shorter version :).

Yes, no problem. :) I'll do that right away

EDIT: Done! It accepts --version or -v and outputs exactly Skyscraper 3.0.0.

For now the only thing I've yet to figure out (just because I didn't have the time) is the cache location chance, since the script would migrate it on first install (for existing users).

I'm sure you already thought of this, but it would probably be something like "if 'dbs' exists, move that, else if 'cache' exists, move that".

CapNinja

"Sit back, relax and let me do the rest" hahaha, awesome.
First, thanks. This is a really cool project, just seeing all colors and details while scraping made my eyes jump from my face with delight (Tom & Jerry style).
Well, after this deserved compliment, I want to make some suggestions and questions. Forgive my poor english and rough comprehension beforehand and let's go.

After scrapping from three sources, I just wanted to sort genres and to my surprise there was a ton of new genres. Is there any way to simplify?
for example, "Soccer" was distributed on three different tags and some did not even appeared (Captain Tsubasa, for example)

1.a. I don't know if the miracle of tags multiplication was related to multiple scrape sources. But, how can you decide to description differences after 3 different scrapes (if this plural exists)? It's something like "first come, first served"?

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here
Suggestion. Select multiple sources to scrape in one go. So then I can sleep the sleep of the righteous and wake with all done.
On the (well made) section of picture-cover-wheel structure, instead of change text, multiple examples to choose.

There is some questions, but I need more time to formulate in (quasi coherent) english. Thanks for read my blabbering until here.

Edit: 1 more

"Est. time left" is only for one system, right? Can we have another estimated time but for all systems selected?

mitu

@CapNinja said in Skyscraper now officially part of RetroPie, please test:

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here

That's more dependant on the front-end, and ES doesn't supports only one source for the image tag.

muldjord

@CapNinja said in Skyscraper now officially part of RetroPie, please test:

After scrapping from three sources, I just wanted to sort genres and to my surprise there was a ton of new genres. Is there any way to simplify?

Each source returns a bunch of genres related to the game. I have no control of each individual genre for each source. It comes as a package so to speak. You can prioritize which source you want to use the "genre package" from though. Read more here. I general, please check out the documents in this folder. It will explain pretty much all you need to know. Be aware that some of it is "under construction" as I am currently moving towards the 3.0.0 release.

1.a. I don't know if the miracle of tags multiplication was related to multiple scrape sources. But, how can you decide to description differences after 3 different scrapes (if this plural exists)? It's something like "first come, first served"?

See answer above.

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here

Not possible and not really part of "scraping". :) This would have to be handled by the frontend instead, which it doesn't at the moment as far as I know.

Suggestion. Select multiple sources to scrape in one go. So then I can sleep the sleep of the righteous and wake with all done.

You can always script it to do so. Skyscraper won't be supporting this as it's just as easy to simply create a script. You can use the "simple mode" or the RetroPie-Setup script for easier access if you don't like working on command line. :)

On the (well made) section of picture-cover-wheel structure, instead of change text, multiple examples to choose.

Check the artwork docs as answered above.

"Est. time left" is only for one system, right? Can we have another estimated time but for all systems selected?

Can't be done. Skyscraper only knows about the current running Skyscraper, not what happens after that. :)

Have fun with it!

muldjord

@mitu Just a heads up. Skyscraper 3.0.0 is pretty much ready for release. I've been testing it over the past week or so and I think it's ready for prime time.

muldjord

User at reddit reported having issues if his rom folder inside /home/pi/RetroPie/roms is symlinked to some other place such as /media/usb0/roms. The script reports "no populated rom folders were found in /home/pi/RetroPi/roms" and quits. Might be permission related, he's currently investigating that. Tested this with Skyscraper on command line and it works.

Links to thread: https://www.reddit.com/r/RetroPie/comments/ahs14m/skyscraper_with_symlinked_rom_folder_not_working/