Skyscraper now officially part of RetroPie, please test

muldjord · 8 Jan 2019, 21:07

@mitu Hard to give an estimate as I am currently battling IGDB's unstable servers to integrate their new API. And I also have some stuff with cache entry editing I'd like to get in. So... Maybe 2 weeks? Rough estimate.

mitu · 8 Jan 2019, 21:18

@muldjord Ok, then I should do some testing in the week-end with your current 3.0.0.
I do have a question regarding the changes in 3.x.
The new modus operandi will be that online scraping would only add resources to the cache, without modifying the users' gamelist. So let's say I scrape from X, Y, Z, etc.. and then with localdb - resulting in the gamelist.xml being updated (or whatever the AttractMode equivalent is).
What happens if I do

scrape with X, then with localdb
scrape with Y, then with localdb.
etc.
Would I get the same results as before ?

muldjord · 1 Aug 2019, 21:32

@mitu Any module other than 'localdb' will only add data to the cache. So whenever the user wants a game list generated, they need to use 'localdb'. It's the gather and combine phases I am trying to clarify to the users. There's a lot of confusion as it is right now. Many users think they should just scrape with one module and then stop. But that was never the intention of course.

So the proper way from 3.x forward is, as you describe, to do Z, Y, X or whatever order you want of the non-localdb sources. Then when you feel like you have all the data you need cached, you will do one last run with 'localdb' which generated the game list from all cached data. You can always go back and do Z, Y again if you want to, as long as you remember to add --refresh. And then, again, do one final run with localdb.

EDIT: Actually, I think I'll start referring to it as the gather and generate phases instead. Makes more sense, right?

mitu · 1 Aug 2019, 21:35

@muldjord said in Skyscraper now officially part of RetroPie, please test:

EDIT: Actually, I think I'll start referring to it as the gather and generate phases instead. Makes more sense, right?

Well, it already made sense - for me - with the actual behavior :). The (CL)UI though refers to both as 'scraping', which I guess it's what you're trying to clarify more in the 3.x series (code and docs).

muldjord · 1 Aug 2019, 21:54

@mitu Yes, it would be cool to clarify it somehow. Perhaps move the localdb to the bottom of the list and add a title above it with "GENERATE GAMELIST:" and at the very top add a title with "GATHER RESOURCES:" or similar. the import and esgamelist modules and others should also be at the top to clearly remove them from the localdb module which is the one "GENERATE" module so to speak.

EDIT: I think it would actually make sense to change this already pre-3.x as the process is the same. I've just forced it more in 3.x.

goobatroopa · 1 Oct 2019, 15:29

Used Skyscraper today for the first time. Great piece of software and worked like a charm. Thanks @muldjord

muldjord · 10 Jan 2019, 16:59

@goobatroopa Awesome, thanks! Glad you like it :)

ohmycommodore · 1 Oct 2019, 18:48

@muldjord
"Remember to quit Emulationstation before using Skyscraper"... is this still essential? If so, what is the command to run Skyscraper from terminal? I just have a couple of empty systems to scrape and thought I should check first on the best approach.

goobatroopa · 1 Oct 2019, 19:00

@ohmycommodore sudo ~/RetroPie-Setup/retropie_setup.sh
And then choose experimental

Instructions in the first post of this thread.

ohmycommodore · 10 Jan 2019, 19:08

Thanks, was just wondering whether there's a different command for running it, vs installing it. I already have it installed from a while back. Perhaps it's the same command and that's just not stated up there.

mitu · 10 Jan 2019, 19:10

@ohmycommodore said in Skyscraper now officially part of RetroPie, please test:

Perhaps it's the same command and that's just not stated up there.

"Up there" - where ?

ohmycommodore · 10 Jan 2019, 19:10

@mitu Ah sorry, first post, up at the top I meant.

mitu · 10 Jan 2019, 19:12

@ohmycommodore Did you read the Wiki page ? It's stated in there:

If you wish to run it from the command line, for more advanced options and parameters, you can find it installed in /opt/retropie/supplementary/skyscraper/Skyscraper.

ohmycommodore · 1 Oct 2019, 19:15

Of course, was just looking for clarification as I don't terminal too much, and there's a warning about mixing methods of running this thing. I also haven't used it since early on, and there were recent changes. Thanks.

goobatroopa · 10 Jan 2019, 19:17

@ohmycommodore I ran it from the terminal after exiting ES. I know it can be installed whilst ES is running but I think you have to scrape without ES running.

mitu · 10 Jan 2019, 19:19

@ohmycommodore I don't understand - you can run it through the GUI, which ultimately runs the CLI with some parameters pre-set. Each setting in the GUI has a Help option which shows which CLI parameter is used for that option. So ultimately running it through the GUI is the same think as running it from the command line - as stated, the command line has more options than what is exposed through the GUI.
If you're not comfortable with the CLI, then I suggest you use the GUI and give it a try.

Ronoh55 · 11 Jan 2019, 02:10

Working great for me so far. I did have a small issue where I had games in folders and it found them but refused to write a gameslist.xml. It was easily fixed for me because ES has the functionality now that made me use folders to begin with (genre filtering) and I just removed everything from folders.

One thing I'd like to see in the GUI though is scraping images by region. Default seems to be EU and I'd love to scrape US without going to command line.
Or did I miss a setting?

mitu · 11 Jan 2019, 04:41

@Ronoh55 There is no default for region set from the GUI, however you can edit the .ini file and set your preferred region there.

muldjord · 13 Jan 2019, 21:59

@mitu An issue has been reported at the Skyscraper github which relates to the script and not Skyscraper itself. If a user has all of their files in subfolders inside a platform folder such as /home/pi/RetroPie/roms/mame-libretro/subfolder/[all files here] the script exits without scraping the subfolder. To make this work the user had to have at least one file in the /home/pi/RetroPie/roms/mame-libretro folder for the script to run Skyscraper.

Ronoh55 · 13 Jan 2019, 22:05

MobyGames is not in the gui but is on the Github as an option to scrape from. It is still an option correct? Only through CLI for now?