Skyscraper now officially part of RetroPie, please test

ohmycommodore

Of course, was just looking for clarification as I don't terminal too much, and there's a warning about mixing methods of running this thing. I also haven't used it since early on, and there were recent changes. Thanks.

goobatroopa

@ohmycommodore I ran it from the terminal after exiting ES. I know it can be installed whilst ES is running but I think you have to scrape without ES running.

mitu

@ohmycommodore I don't understand - you can run it through the GUI, which ultimately runs the CLI with some parameters pre-set. Each setting in the GUI has a Help option which shows which CLI parameter is used for that option. So ultimately running it through the GUI is the same think as running it from the command line - as stated, the command line has more options than what is exposed through the GUI.
If you're not comfortable with the CLI, then I suggest you use the GUI and give it a try.

Ronoh55

Working great for me so far. I did have a small issue where I had games in folders and it found them but refused to write a gameslist.xml. It was easily fixed for me because ES has the functionality now that made me use folders to begin with (genre filtering) and I just removed everything from folders.

One thing I'd like to see in the GUI though is scraping images by region. Default seems to be EU and I'd love to scrape US without going to command line.
Or did I miss a setting?

mitu

@Ronoh55 There is no default for region set from the GUI, however you can edit the .ini file and set your preferred region there.

muldjord

@mitu An issue has been reported at the Skyscraper github which relates to the script and not Skyscraper itself. If a user has all of their files in subfolders inside a platform folder such as /home/pi/RetroPie/roms/mame-libretro/subfolder/[all files here] the script exits without scraping the subfolder. To make this work the user had to have at least one file in the /home/pi/RetroPie/roms/mame-libretro folder for the script to run Skyscraper.

Ronoh55

MobyGames is not in the gui but is on the Github as an option to scrape from. It is still an option correct? Only through CLI for now?

mitu

@muldjord I'll take a look, looks like a script issue.

mitu

@Ronoh55 Yes, it's not in the GUI, since it's not suitable for bulk scraping due to the low number of allowed scrapes (25 I think).

Ronoh55

Ah, that makes sense. Didn't know it has such a small limit

muldjord

@Ronoh55 Yes, the mobygames module is very limited and should only ever be used to scrape single roms. So that makes it unusable in the GUI. That's the reason for the removal. It can still be used on command line though and works well.

mitu

@muldjord Here's a draft for the new UI. I removed the Localdb as a scraping source and added the new ESGamelist local source.

There is now an explicit option to generate the gamelist(s) from the menu

When chosen, you'll be asked to choose which systems you want to include in the gamelist generation (similar to 'Scrape chosen systems'). I'm not so sure about renaming 'Scrape' to 'Gather', and 'Download' doesn't seem appropriate (it's not only downloading artwork, it's also information retrieval). I'm open to ideas :).

I think I'll re-arrange the menu, so that any configuration options are below the actions (Scrape/Scrape/Generate and then the rest of the options).

Silent

Since artwork.xml and config.ini can be edited from the GUI already, how about also allowing to edit aliasMap.csv? For completeness.

muldjord

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

So you'd enter that menu and choose any of the scraping modules from inside it and it would do 1 run gathering resources from that 1 module, then return to the menu. Then go back to the main menu and choose "Generate ES game list(s) from cache".

mitu

@muldjord said in Skyscraper now officially part of RetroPie, please test:

@mitu Looks good! One note: The "Gather resources from X" should not include the name since it should be initiated from within the menu. It's not an option, just a submenu. And I think it makes sense to call it "Gather and cache resources" for instance. That's more specific and descriptive.

I don't understand :). The first 2 choices initiate a scrape (gather), the 4th option is just a configuration toggle. So we would rename the entries

Gather and cache resources for all systems - runs the scrape for all systems, with the source from 4.
Gather and cache resources for chosen systems - runs the scrape for selected systems, with the source from 4.

Is that what you meant ?

EDIT: something like

mitu

@Silent That would be a good candidate; while an 'advanced' feature, it doesn't require any other CLI operations the user must perform.
I see there is a sample (empty one) provided, I'll have to check during the install to not overwrite it. Right now, I think you may be the only user that is using this feature :).

muldjord

@mitu Yes, I forgot that's how they work, sorry.

I think it should be like this for the best workflow and to best split the two things apart (gather and generate):

HEADLINE: "Gather and cache resources:" (just a non-functional headline option)
"Source: ScreensScraper (Online)"
- All of the sources as they are now
"Gather for: All systems"
"Gather for: Chosen systems"
"Cache options -->"
- Force cache refresh
- Cache covers, cache screenshots etc... (Enabled)
- Vacuum cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb vacuum" command)
- Delete cache: For selected systems (Use the "Skyscraper -p [platform] --purgedb all" command)
HEADLINE: "Game list generation:" (just a non-functional headline option)
Generate EmulationStation game list
Generate options -->
- ROM Names (Source name)
- Remove bracket info (Disabled)
- Use Rom folders for game list and media (Enabled)
HEADLINE: "General:" (just a non-functional headline option)
Advanced Skyscraper options -->
- Enable videos
- Edit config.ini
- Edit artwork.xml
- Edit aliasMap.csv

mitu

This post is deleted!

mitu

@muldjord So here's a draft version

Re-organized the menu a bit, per your layout

the scraping source and Cache options are next to each other, since both are sub-menus opening up
the cache options have the actions of vacuum/purge system/purge all (now using Skyscraper instead of manually erasing files and moved from advanced options).
added the video download toggle inside the cache options
advanced options now only have the config files editing.

One question for game generation - do --videos needs to be enabled when scraping from localdb to add the video tag ?

muldjord

@mitu

Looks good!

--videos should be moved to general though as per my suggestion, as it's not related to caching any more than it's related to game list generation. It's not like the --nocovers and such options. It's an option related to Skyscraper in general. So to answer your question, yes, it needs to be enabled for videos to be used in the game list. And also for videos to be cached.