Skyscraper now officially part of RetroPie, please test

muldjord

@mitu An issue has been reported at the Skyscraper github which relates to the script and not Skyscraper itself. If a user has all of their files in subfolders inside a platform folder such as /home/pi/RetroPie/roms/mame-libretro/subfolder/[all files here] the script exits without scraping the subfolder. To make this work the user had to have at least one file in the /home/pi/RetroPie/roms/mame-libretro folder for the script to run Skyscraper.

Ronoh55

MobyGames is not in the gui but is on the Github as an option to scrape from. It is still an option correct? Only through CLI for now?

mitu

@muldjord I'll take a look, looks like a script issue.

mitu

@Ronoh55 Yes, it's not in the GUI, since it's not suitable for bulk scraping due to the low number of allowed scrapes (25 I think).

Ronoh55

Ah, that makes sense. Didn't know it has such a small limit

muldjord

@Ronoh55 Yes, the mobygames module is very limited and should only ever be used to scrape single roms. So that makes it unusable in the GUI. That's the reason for the removal. It can still be used on command line though and works well.

mitu

@muldjord Here's a draft for the new UI. I removed the Localdb as a scraping source and added the new ESGamelist local source.

There is now an explicit option to generate the gamelist(s) from the menu

When chosen, you'll be asked to choose which systems you want to include in the gamelist generation (similar to 'Scrape chosen systems'). I'm not so sure about renaming 'Scrape' to 'Gather', and 'Download' doesn't seem appropriate (it's not only downloading artwork, it's also information retrieval). I'm open to ideas :).

I think I'll re-arrange the menu, so that any configuration options are below the actions (Scrape/Scrape/Generate and then the rest of the options).

Silent

Since artwork.xml and config.ini can be edited from the GUI already, how about also allowing to edit aliasMap.csv? For completeness.

muldjord

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

So you'd enter that menu and choose any of the scraping modules from inside it and it would do 1 run gathering resources from that 1 module, then return to the menu. Then go back to the main menu and choose "Generate ES game list(s) from cache".

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

I don't understand :). The first 2 choices initiate a scrape (gather), the 4th option is just a configuration toggle. So we would rename the entries

Gather and cache resources for all systems - runs the scrape for all systems, with the source from 4.
Gather and cache resources for chosen systems - runs the scrape for selected systems, with the source from 4.

Is that what you meant ?

EDIT: something like

mitu

@Silent That would be a good candidate; while an 'advanced' feature, it doesn't require any other CLI operations the user must perform.
I see there is a sample (empty one) provided, I'll have to check during the install to not overwrite it. Right now, I think you may be the only user that is using this feature :).

muldjord

@mitu Yes, I forgot that's how they work, sorry.

I think it should be like this for the best workflow and to best split the two things apart (gather and generate):

HEADLINE: "Gather and cache resources:" (just a non-functional headline option)
"Source: ScreensScraper (Online)"
- All of the sources as they are now
"Gather for: All systems"
"Gather for: Chosen systems"
"Cache options -->"
- Force cache refresh
- Cache covers, cache screenshots etc... (Enabled)
- Vacuum cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb vacuum" command)
- Delete cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb all" command)
HEADLINE: "Game list generation:" (just a non-functional headline option)
Generate EmulationStation game list
Generate options -->
- ROM Names (Source name)
- Remove bracket info (Disabled)
- Use Rom folders for game list and media (Enabled)
HEADLINE: "General:" (just a non-functional headline option)
Advanced Skyscraper options -->
- Enable videos
- Edit config.ini
- Edit artwork.xml
- Edit aliasMap.csv

mitu

This post is deleted!

mitu

@muldjord So here's a draft version

Re-organized the menu a bit, per your layout

the scraping source and Cache options are next to each other, since both are sub-menus opening up
the cache options have the actions of vacuum/purge system/purge all (now using Skyscraper instead of manually erasing files and moved from advanced options).
added the video download toggle inside the cache options
advanced options now only have the config files editing.

One question for game generation - do --videos needs to be enabled when scraping from localdb to add the video tag ?

muldjord

@mitu

Looks good!

--videos should be moved to general though as per my suggestion, as it's not related to caching any more than it's related to game list generation. It's not like the --nocovers and such options. It's an option related to Skyscraper in general. So to answer your question, yes, it needs to be enabled for videos to be used in the game list. And also for videos to be cached.

mitu

@muldjord Yeah, the video stuff does belong to both option sections, will move it back - adding it in both sections wouldn't make sense. The new version is mostly done, just need to test it a bit more.

muldjord

@mitu I would also argue to remove the "Gather for All systems" entirely, and rename the "Gather for Chosen systems" to "Gather for system(s)". That will align better with the "Generate game list(s)" option further down.

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

EDIT: Btw, when all this is done I think I'll move the "Simple Mode" of Skyscraper down the ranks in the docs, and instead simply refer to the RetroPie wiki and this script. I'm not gonna remove it completely just yet, just in case non-RetroPie'ers use it. But I feel we're reaching a point where it's becoming obsoleted by this script.

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

It feels like the "Gather for All systems" is cluttering it up a bit right now. And people still have the option to simply select all systems if they want. There could also simply be a "Select all" inside of the "Gather for systems(s)" option. That would also be better I think. Thoughts?

I though about it - it would make the 2 phases of the scraping symmetrical and it doesn't quite fit the Skyscraper's process (I see it mostly as a one-shot scraping option for quickly getting something for all systems).
I still think it has its usefulness for first time scraping - when you've got a few of your systems populated and you're just wanting to get the info in Emulationstation. Gradually, as you add more ROMs, it's probably not going to get much use since you'd be scraping just one system at a time, switching sources and checking for missing info/roms.
I try to find a way add an 'All' toggle in one gathering option, but dialog won't let me combine radio buttons with a multiple selection list.

muldjord

@mitu Yes, I can see how it's useful certainly. It's only because I feel like is mismatches (and clutters it up a bit) with how there's only one option for game list generation. I'm ok with having it there, I just hope people don't get confused by the difference.

muldjord

@mitu Just to summarize the changes you need to pay attention to in regards to the script when moving from 2.x to 3.x

--refreshhas become --cache refresh but I'll leave --refresh in 3.x for the time being to ease migration. It will be removed at some point in the future and I'll be sure to let you know when I do remove it.
The --purgedb vacuum option has become --cache vacuum.
The --purgedb all option has become --cache purge:all which will remove all resources for the selected platform.
- I'll let you decide whether it makes sense to use these for the time being as they are not compatible with 2.x
- You could just keep the old cache clearing options from the old script for the time being and switch to the new ones when users have migrated. But please note the next point, that dbs has changed.
The ~/.skyscraper/dbs folder is now ~/.skyscraper/cache. This is auto-migrated when Skyscraper 3.x is run for the first time after updating (it simply moves dbs to cache).

If I think of other things, I will update this post.