You may use also this Python script to scrape the majority of information from the worldofspectrum site ad interim

https://gist.github.com/Gemba/c323e8036b921a1aa2fb927bb4958928

Usage: See comments in header of script.

"The real world is a special case."

FTR, the RetroPie version 4.8.6 onwards deploys the configurable platform information (peas.json). The upcoming Skyscraper release 3.14 will assess the RetroPie Setup version and will provide the hint on how to proceed if a older version of RetroPie Setup is found.

The scriptmodule is thought as a "ease of use" / for default setups, if you want to unleash the full potential of Skyscraper use the command line.

Within the scriptmodule you can only alter the (output) gamelist (-g) folder and media folder (-o).

With Use ROM folder (Enabled) (see generate options in the menu) the gamelist location is
/home/pi/RetroPie/roms/<platform> and /home/pi/RetroPie/roms/<platform>/media.

With Use ROM folder (Disabled) the gamelist location is
/home/pi/.emulationstation/gamelists/<platform> and /home/pi/.emulationstation/gamelists/<platform>/media.

The inputFolder is in any case picked up from the config.ini, however, the scriptmodule only uses $romdir which is currently taken from the default RetroPie value (which is /home/pi/RetroPie/roms). See _list_systems_skyscraper() in the scriptmodule skyscraper.sh.

TL;DR: Using a symlink as you did is the right way.

@sleve_mcdichael - ah thanks for the info! Last time I set up a pi system (3b) was about 6 years ago so I'm out of touch :)

@Lolonois - thanks for replying so quickly. I was reading the MobyGames section on https://gemba.github.io/skyscraper/SCRAPINGMODULES/ and it sounded like it could be something cool, but maybe i'm trying to solve a problem that doesn't exist. So far screenscraper, openretro, etc are covering things really well. I might look into it more and report back if there is anything worthwhile there. Cheers!

@Flexo55 Good you did find the cause. Please, as a courtesy to other users of this forum, do edit your first post and add to the subject something like [solved].

It’s working fine, thank you!

Thank you very much for the detail answer! i will check all options carefully.

Even if it has been around ever since, the scriptmodule opens a separate config path outside the config ini and it is not transparent to the user that some flags are hardwired in the scriptmodule.

I understand that this may cause some confusion, but as I said, I'm not aware of any reported issues related to this.

I ditched the initial idea to have exactly one config source thus, to have any config switch set in/get from the scriptmodule as it would be a royal annoyance to rewrite the evaluation/precedence logic of the config ini in bash.

I don't see why would be necessary. Ultimately the dialog based UI maintained in RetroPie is just a wrapper over the CLI options, meant to be used by users who don't need many options besides maybe turning on 'video' downloads. Avanced/experienced users are using the CLI to scrape and can (and do) ignore the UI. For the former users, the current .cfg is (ubenknownst to them) the config.ini, since they just use a couple of options to scrape and don't seek or need other parameters.

As a compromise I would see that the all applied switches from the scriptmodule are prompted to the user before he/she starts the scraping and/or have the skipped flag also configurable.

Yes, I can see this working - start anew without any .cfg so the user would always need to set the parameters and those parameters (including the skipped flag if we don't want to default to a safe value) are not persisted to any configuration. We can make also make it more clear that options set in the UI override the config.ini options.

Okay I think I sorted it out.

I edited my xml files to remove any absolute paths and replace them with the old /home/<user>/RetroPie/roms/... equivalent line.
Then I deleted the <folder> lines from the bottom of the file.

Everything works again.

Yes, you can modify the theme and remove it from the view. I don't think it has anything to do with Skyscraper.

@sleve_mcdichael said in Skyscraper - generate gamelist only once in the end?:

If you want to re-download data from a source again (or gather additional data such as marquees or videos that weren't enabled the first time), use --cache refresh.

Almost: --refresh / --cache refresh does update every enabled property (ie. cacheCovers=true and others in config.ini) of the scraped ROM(s).

By using only --flags video (or videos=true in config.ini) or setting in the ini: cacheMarquees=true (if it was false before) without --refresh will scrape the missing properties for the ROM(s), unless they are present in the cache for that scraper module.

Thank you both for catching and fixing this. It escaped me.

Thanks that you deployed a fix for this too 😃. I havent tested the fix but I will also mark this thread as solved 😊

@m33ts4k0z Thanks for reporting back. Please add to the subject [solved] or similar.

I just had a convo on discord with the screenscraper guys and it looks that version 3.9.1 is broken.
This command yields no results on 3.9.1, but it does work on 3.7.7

Skyscraper -p switch -s screenscraper --query "romnom=Super%20Mario%20Odyssey%20(World)%20(En,Ja,Fr,De,Es,It,Nl,Zh,Ru)%20(Rev%203).xci" /home/retropie/RetroPie/roms/switch/super-mario-odyssey.xci

[...]
My sd-card die recently. i transfered my metadata, download_images and gamelists folders, to a new setup and everything looks fine except that, if i run again skyscraper, it starts to download the pics that i have already download. My question: Is this a normal behavior or i missing something? Thanks!

Yes, it's normal - Skyscraper downloads files into it's cache first, then it copies them to the final location. If you haven't transferred your cache (from $HOME/.skyscraper/cache), then it's normal that the artwork/videos are re-downloaded.

Now I officially announce the release of Skyscraper 3.15. (3.15.3)

As always find the changes here

Update the RetroPie setup before you update Skyscraper to get a new helper script deployed (which you may use when you get the shown warning below).

Important: In some cases you may get a warning when starting Skyscraper

Looks like '/home/pi/.skyscraper/peas.json' has local changes. Please transfer local changes to another file to mute this warning. See topic 'Transferring Local Platform Changes' in the documentation for guidance.

Why do you get this warning?
This happens when you made changes to the peas.json or platforms_idmap.csv in the past, which was intended to be that way.
See the documentation here on how to remediate the situation.

A later release will change the current behaviour of the RetroPie scriptmodule of Skyscraper and will no longer put updated peas.json as peas.json.rp-dist (same for platforms_idmap.csv). Instead it will always replace your existing ~/.skyscraper/peas.json / platforms_idmap.csv with the upstream version. Thus, do follow the guide, when you get the warning.

There will be an extra announcement when a new Skyscraper version will change the copy behaviour. But as there is no ETA for 3.16 we are talking at least about months.

Any questions let me know.

@mitu Is it so simple then? OK, thank you for your answer and for your time.

@jadams what platform and source (nvm I see screenscraper -- what platform and game(s)?) do you use? I just tested arcade frogger.zip on screenscraper and got a description:

$ /opt/retropie/supplementary/skyscraper/Skyscraper -p arcade -s screenscraper ~/RetroPie/roms/arcade/frogger.zip ------------------------------------------ Running Skyscraper v3.7.7 by Lars Muldjord ------------------------------------------ Platform: 'arcade' Scraping module: 'screenscraper' (...) #1/1 (T1) Pass 1 ---- Game 'frogger' found! :) ---- Scraper: screenscraper From cache: NO Search match: 100 % Compare title: 'Frogger' Result title: 'Frogger' () Platform: 'Konami Classics' () Release Date: '1981-01-01' () Developer: 'Konami' () Publisher: 'Konami' () Players: '2' () Ages: '' () Tags: 'Action / Labyrinth, Action' () Rating (0-1): '0.9' () Cover: YES () Screenshot: YES () Wheel: YES () Marquee: NO (uncached) () Video: YES () Description: () 'Frogger is a one or two-player game in which players must overcome a number of hazards to safely guide the green amphibian from the bottom of the screen to one of five home bases situated at the top of the screen. A round is completed once five frogs have been guided safely home, after which the next round begins with an increased level of difficulty. This includes faster-moving vehicles on the road section and new, deadly hazards on the river section, including crocodiles, otters and snakes.' (...)

@Lolonois oh yah, it looks like this is the same issue as that.

I'm using linapple. It does not appear to support .zip files (platforms.cfg), but adding the Apple2 suffixes to that list in nametools.cpp does look like it might do the trick.

Edit: yah that seems to have worked: https://github.com/muldjord/skyscraper/compare/master...s1eve-mcdichae1:skyscraper:apple2exts