Skyscraper now officially part of RetroPie, please test

mitu

@muldjord So here's a draft version

Re-organized the menu a bit, per your layout

the scraping source and Cache options are next to each other, since both are sub-menus opening up
the cache options have the actions of vacuum/purge system/purge all (now using Skyscraper instead of manually erasing files and moved from advanced options).
added the video download toggle inside the cache options
advanced options now only have the config files editing.

One question for game generation - do --videos needs to be enabled when scraping from localdb to add the video tag ?

muldjord

@mitu

Looks good!

--videos should be moved to general though as per my suggestion, as it's not related to caching any more than it's related to game list generation. It's not like the --nocovers and such options. It's an option related to Skyscraper in general. So to answer your question, yes, it needs to be enabled for videos to be used in the game list. And also for videos to be cached.

mitu

@muldjord Yeah, the video stuff does belong to both option sections, will move it back - adding it in both sections wouldn't make sense. The new version is mostly done, just need to test it a bit more.

muldjord

@mitu I would also argue to remove the "Gather for All systems" entirely, and rename the "Gather for Chosen systems" to "Gather for system(s)". That will align better with the "Generate game list(s)" option further down.

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

EDIT: Btw, when all this is done I think I'll move the "Simple Mode" of Skyscraper down the ranks in the docs, and instead simply refer to the RetroPie wiki and this script. I'm not gonna remove it completely just yet, just in case non-RetroPie'ers use it. But I feel we're reaching a point where it's becoming obsoleted by this script.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

I though about it - it would make the 2 phases of the scraping symmetrical and it doesn't quite fit the Skyscraper's process (I see it mostly as a one-shot scraping option for quickly getting something for all systems).
I still think it has its usefulness for first time scraping - when you've got a few of your systems populated and you're just wanting to get the info in Emulationstation. Gradually, as you add more ROMs, it's probably not going to get much use since you'd be scraping just one system at a time, switching sources and checking for missing info/roms.
I try to find a way add an 'All' toggle in one gathering option, but dialog won't let me combine radio buttons with a multiple selection list.

muldjord

@mitu Yes, I can see how it's useful certainly. It's only because I feel like is mismatches (and clutters it up a bit) with how there's only one option for game list generation. I'm ok with having it there, I just hope people don't get confused by the difference.

muldjord

@mitu Just to summarize the changes you need to pay attention to in regards to the script when moving from 2.x to 3.x

--refreshhas become --cache refresh but I'll leave --refresh in 3.x for the time being to ease migration. It will be removed at some point in the future and I'll be sure to let you know when I do remove it.
The --purgedb vacuum option has become --cache vacuum.
The --purgedb all option has become --cache purge:all which will remove all resources for the selected platform.
- I'll let you decide whether it makes sense to use these for the time being as they are not compatible with 2.x
- You could just keep the old cache clearing options from the old script for the time being and switch to the new ones when users have migrated. But please note the next point, that dbs has changed.
The ~/.skyscraper/dbs folder is now ~/.skyscraper/cache. This is auto-migrated when Skyscraper 3.x is run for the first time after updating (it simply moves dbs to cache).

If I think of other things, I will update this post.

mitu

@muldjord (I'll reply here, instead of the other topic)

Most of the changes are ok (as far as the package is concerned), as long as you have some backwards compatibility. The script could certainly detect the version and prompt the user to upgrade when it detects an incompatible version, but I wouldn't impose that.
SScraper has a Upgrade to the latest version option, maybe we can add that after the Advanced options, since Skyscraper adds new features and fixes faster and I have the feeling some users don't have the habit or just don't want to break things doing an full upgrade once in a while.

For now the only thing I've yet to figure out (just because I didn't have the time) is the cache location chance, since the script would migrate it on first install (for existing users).

I think that as long as you keep --refresh around, we can introduce the purgedb options later (though I'll have to cut vacuum or make it conditional, depending on the version).

So, one request from me - can you add a --version parameter ? Right now the script just parses the --help invocation, but it would be nice to provide a shorter version :).

muldjord

@mitu said in Skyscraper now officially part of RetroPie, please test:

So, one request from me - can you add a --version parameter ? Right now the script just parses the --help invocation, but it would be nice to provide a shorter version :).

Yes, no problem. :) I'll do that right away

EDIT: Done! It accepts --version or -v and outputs exactly Skyscraper 3.0.0.

For now the only thing I've yet to figure out (just because I didn't have the time) is the cache location chance, since the script would migrate it on first install (for existing users).

I'm sure you already thought of this, but it would probably be something like "if 'dbs' exists, move that, else if 'cache' exists, move that".

CapNinja

"Sit back, relax and let me do the rest" hahaha, awesome.
First, thanks. This is a really cool project, just seeing all colors and details while scraping made my eyes jump from my face with delight (Tom & Jerry style).
Well, after this deserved compliment, I want to make some suggestions and questions. Forgive my poor english and rough comprehension beforehand and let's go.

After scrapping from three sources, I just wanted to sort genres and to my surprise there was a ton of new genres. Is there any way to simplify?
for example, "Soccer" was distributed on three different tags and some did not even appeared (Captain Tsubasa, for example)

1.a. I don't know if the miracle of tags multiplication was related to multiple scrape sources. But, how can you decide to description differences after 3 different scrapes (if this plural exists)? It's something like "first come, first served"?

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here
Suggestion. Select multiple sources to scrape in one go. So then I can sleep the sleep of the righteous and wake with all done.
On the (well made) section of picture-cover-wheel structure, instead of change text, multiple examples to choose.

There is some questions, but I need more time to formulate in (quasi coherent) english. Thanks for read my blabbering until here.

Edit: 1 more

"Est. time left" is only for one system, right? Can we have another estimated time but for all systems selected?

mitu

@CapNinja said in Skyscraper now officially part of RetroPie, please test:

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here

That's more dependant on the front-end, and ES doesn't supports only one source for the image tag.

muldjord

@CapNinja said in Skyscraper now officially part of RetroPie, please test:

After scrapping from three sources, I just wanted to sort genres and to my surprise there was a ton of new genres. Is there any way to simplify?

Each source returns a bunch of genres related to the game. I have no control of each individual genre for each source. It comes as a package so to speak. You can prioritize which source you want to use the "genre package" from though. Read more here. I general, please check out the documents in this folder. It will explain pretty much all you need to know. Be aware that some of it is "under construction" as I am currently moving towards the 3.0.0 release.

1.a. I don't know if the miracle of tags multiplication was related to multiple scrape sources. But, how can you decide to description differences after 3 different scrapes (if this plural exists)? It's something like "first come, first served"?

See answer above.

Games with multiple covers (like the genesis - mega drive situation), there is a way to show both? Like, every 5-10 seconds change photos or something... I don't know if I made my self clear here

Not possible and not really part of "scraping". :) This would have to be handled by the frontend instead, which it doesn't at the moment as far as I know.

Suggestion. Select multiple sources to scrape in one go. So then I can sleep the sleep of the righteous and wake with all done.

You can always script it to do so. Skyscraper won't be supporting this as it's just as easy to simply create a script. You can use the "simple mode" or the RetroPie-Setup script for easier access if you don't like working on command line. :)

On the (well made) section of picture-cover-wheel structure, instead of change text, multiple examples to choose.

Check the artwork docs as answered above.

"Est. time left" is only for one system, right? Can we have another estimated time but for all systems selected?

Can't be done. Skyscraper only knows about the current running Skyscraper, not what happens after that. :)

Have fun with it!

muldjord

@mitu Just a heads up. Skyscraper 3.0.0 is pretty much ready for release. I've been testing it over the past week or so and I think it's ready for prime time.

muldjord

User at reddit reported having issues if his rom folder inside /home/pi/RetroPie/roms is symlinked to some other place such as /media/usb0/roms. The script reports "no populated rom folders were found in /home/pi/RetroPi/roms" and quits. Might be permission related, he's currently investigating that. Tested this with Skyscraper on command line and it works.

Links to thread: https://www.reddit.com/r/RetroPie/comments/ahs14m/skyscraper_with_symlinked_rom_folder_not_working/

mitu

@muldjord I've seen the thread, I'll check what's going on.
The new module is also ready, forgot to post the new UI screenshot. I've been re-testing the installation and the way it works with both versions. I have one more check to do and I'll probably submit the new version tonight.

muldjord

@mitu Looks really good! Really looking forward to getting this out there so I can start focusing on new features again.

Silent

So is there any reason for Skyscraper to still be in "experimental"?

muldjord

@Silent For the time being I think so yes. The module needs to be tested thoroughly for a while before I think it should be moved to Optional. And docs needs to be updated and so on (I have a lot of stuff I need to document). So while it's ready for use, there's some peripheral stuff that needs to solidify before it's ready for Optional. That's my opinion anyways.

Silent

Also, for the sake of "noob friendliness" I think the nice end goal would be to see Skyscraper as a backend for ES scraping. Keep in mind however that I am saying this without checking how that ES feature works at all, so perhaps it's never going to be possible.

mitu

@muldjord PR submitted - https://github.com/RetroPie/RetroPie-Setup/pull/2584. Since it should work with both versions, can be merged before you release 3.x.
Fixed both bugs ~~fixed~~ reported so far (the one with the ROM folder containing only sub-folders and the one with the symlink-ed ROMs folder).