Skyscraper is back! (v3.17)

mitu

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

The default seems to be still to output the full path, instead of a relative path as before. Why the change ? Also, folder entries are still generated.

YFZdude

@Lolonois This seems to straighten out my issue.
I moved the existing gamelist to generate a new one and got the expected paths containing /home/<user>/RetroPie/roms/ etc.

I did not see any <folder> entries at the bottom of my newly generated list either. I was also not using the relative paths setting in any config file.

Thanks

Lolonois

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

[1] https://github.com/RetroPie/EmulationStation/blob/master/GAMELISTS.md#folder

Lolonois

@YFZdude Thanks for reporting back. This means your case is ok?

YFZdude

@Lolonois
Yes I have it working fine on my setup.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Yes, but that part is not present in most RetroPie installations - it's been only added to the -dev branch. Until this feature is present by default in RetroPie, I'm asking you to hide this under an option and default to not generate this info.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

I have the default .ini, without anything else set. I didn't scrape for video. The default ini has:

[esgamelist]
cacheRefresh="true"
cacheScreenshots="false"

[import]
cacheRefresh="true"

enabled only.

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

I don't have any symlinks that way, I only use $HOME/roms symlinked to $HOME/RetroPie/roms, but I don't see how that should affect scraping since the default RetroPie structure for ROM folders is intact.

Lolonois

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

Since I have some SNES images in my cache, I just remove my gamelist.xml and run:

Skyscraper -p snes

and the gamelist.xml is re-created (in the ROMs folder - snes).

Lolonois

@mitu I tried to reconstruct the issue but had no luck. I put a also symlink from ~/roms to ~/RetroPie/roms deleted the extisting gamelist file and created from cache a new one.

relativePaths=false: The absolute paths show up in the gamelist file, the path up to the systems/platforms is expanded to the regular rom folder (/home/pi/RetroPie/roms) unless a specific inputFolder is given.
relativePaths=true: Entries in the gamelist are generated as relative, starting from the regular rom folder plus system/platform, f.i. the file on the filesystem /home/pi/RetroPie/roms/snes/Frogger (USA).zip is generated as ./Frogger (USA).zip in gamelist.

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

If there is still something with the generated paths then it must be a very specific setup I can not imagine.

I will follow-up on the folder thingie later.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

Yes, I checked also in the code and the default hasn't changed. The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off. Sorry for the confusion.

I will follow-up on the folder thingie later.

OK, thank you.

Lolonois

@mitu said in Skyscraper is back! (v3.10.0):

The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off.

Nvm. But good hint, reminds me to consider the dialog based scraping too on development.

Lolonois

The <folder/> situation. (Long post but needed to create the context).

First off:

Skyscraper generates only <folder/> elements in gamelist when there are filesystem directories used for ROM organization inside a platform folder.
Even with ES 2.11.2rp the FileData FOLDER-type entries are created in memory (which then get persisted as <folder/> upon metadata edit+save): https://github.com/RetroPie/EmulationStation/blob/cda7de687924c4c1ab83d6b0ceb88aa734fe6cfe/es-app/src/Gamelist.cpp#L73
It seems to me very few users use the metadata edit on folders as the issues below in ES 2.11.2rp have been present several ES releases.

I tested forth and back between ES (2.11.2rp) and the ES-dev (main/head) in these combinations:

relativePaths=true and false in Skyscraper
gamelist with Skyscraper generated <folder/> entries and without.
Editing and persisting of folders in ES and ES-dev

Resume: I did not identify any regression with ES 2.11.2rp (and also not with ES-dev) introduced with the generated <folder/> elements of Skyscraper.

Here is the directory structure I have tested with

$ tree RetroPie/roms/apple2/
RetroPie/roms/apple2/
├── aaa
│   ├── bbbbb
│   │   └── Thief (1981)(Datamost)(San Inc & TRex crack).dsk
│   └── cccc
│       ├── MECC-A157 The Oregon Trail v1.1 (4am crack) side A.dsk
│       └── MECC-A157 The Oregon Trail v1.1 (4am crack) side B.dsk
├── ddddd
│   └── Multi Disk - Dung Beetles - Guardian - Zenith.dsk
├── eee
│   └── fff
│       └── ggg
│           └── Snakebyte.dsk
├── gamelist.xml
├── loderunr.zip
├── media
.   └── [truncated]
├── Mystery House.dsk
├── Sabotage (19xx)(Mark Allen).dsk
└── Three Mile Island Special Version (1980)(Richard Orban)[a].dsk

However, with ES 2.11.2rp the persistence has flaws. I tested with a gamelist which contained only <game/> elements:

1. Whenever metadata is edited for a folder in ES a new entry is added to the gamelist on save.

E.g. Edit fff metadata in ES once yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Edit fff twice yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>
        <folder>                         <!-- erroneous 2nd entry for same path -->
                <path>eee/fff</path>
                <name>fff 2nd edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

This "entry cumulation" can also be reproduced when ES is quitted in between.

Eventually, the end-user will not notice, as in ES the last <folder/> entry "wins".

If such gamelist is updated with the current Skyscraper the first <folder/> entry wins, as Skyscraper expects there is in maximum one such entry for the same <path/>. Before v3.10 existing <folder/> entries were not preserved by Skyscraper when present in an existing gamelist.

2. Whenever metadata is edited for a folder which is direct child of the platform directory, the platform name is stored as part of the path on save.

E.g. edit aaa metadata in ES yields:

        <folder>
                <path>apple2/aaa</path>  <!-- note the surplus platform folder -->
                <name>aaa 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Upon restart of ES an error is logged (correct, as the path is invalid):

Feb 29 20:12:28 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/apple2/apple2/aaa", skipping.

Also in this case the entries get cumulated with each edit+save of aaa's metadata.

NB: These issues are not present ES-dev, i.e. ES-dev creates exactly one folder element or updates the existing one for each path, when metadata is edited.

TL;DR: I see these options.

Tell the few users of ES 2.11.2rp not to edit folder metadata (game metadata edit is not affected) in ES and then run Skyscraper (as they will lose their ES folder edits, unless there is in maximum one <folder/> entry per <path/>).
Make ES-dev the new ES-stable anytime soon.
Create a workaround for the ES 2.11.2rp issues in Skyscraper, that preserve the last <folder/> element when there are several <folder/> entries with the same <path/>. Additionally drop/repair those folder entries with an non-existing path (the aaa case above). Once the today's ES-dev is the new ES-stable, I can remove that workaround again. :/
Having a flag with "do-not-create-folders" in Skyscraper does not remediate the situation completely, because if there are two folder entries (the eee/fff case above) Skyscraper will pick only the first. Again, user edits from ES may get lost.

Do you see other options? What is your preferred option?

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

Do you see other options? What is your preferred option?

As said before, don't create folder entries by default. I'm not keen on workarounds that may change behavior today or tomorrow based on a release, so having a clear and documented method to disable the folder elements from Skyscraper is preferred. Thank you.

Lolonois

Version 3.10.3 restores the pre v3.10.0 behaviour, unless the config switch addFolders is deliberately set true.

Existing <folder/> entries in a gamelist XML will be kept, for any state of addFolders. If this is not wanted: Remove the gamelist.xml and let Skyscraper generate a new one.

https://github.com/Gemba/skyscraper/releases/tag/3.10.3

Lolonois

Another month, another Skyscraper release: Main additions are ES-DE frontend support and aliasMap entries now applicable also for Screenscraper.

All changes of v3.11.0

abj

I am prepering for the upcoming 3.11 version and the changes that will bring. I checked the docs and i solve all my questions, about the various parameters of config.ini file, except one that i didn't understand very well.

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

When generating gamelist xml, by default, if a rom file that has no scraped resources associated with it, will not be an entry created in the gamelist. Setting this skipped="true" causes empty "dummy" entries be created for these skipped roms.

When gathering data, by default, if a rom has any assets from this source (eg. screenscraper, thegamesdb) it will be skipped but if it has assets from other source, it will still be processed for this source also. By using the CLI option --flags onlymissing, roms with assets from any source will be skipped, and only those with no assets from any source are processed. I think this is probably what you want, in this case.

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

Lolonois

Behavior is as @sleve_mcdichael has written, thanks.

To amend to this:
Skyscraper uses three sources to create a new gamelist.xml: The online DB (like Screenscraper, aso.), it's local cache and if present the existing gamelist.xml file.

Manual changes of an existing gamelist.xml are preserved, whenever there is no data available either by the scraping from an online DB or in the cache.

If you want to have the manual edits stored in the skyscraper cache you may use the -s esgamelist to "scrape" from your existing gamelist.xml. Most likely you should add an entry to the priorities.xml file for that platform to use also esgamelist as cache source. There is more info on this topic here

To cover also the case you may referring to, @abj : Skyscraper writes always all entries of a gamelist.xml fresh (from the beforementioned sources), there is no option to skip entries while Skyscraper is in the last stage and creates the gamelist.xml

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.11):

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

That is right, it only exists as CLI option. It can be as useful permanent option, esp. for MobyGames which has a limit of 35 games to scrape at once, thus it might be easier with this flag to stay below that limit. In a technical nutshell: The limit is checked in Skyscraper by the size of the queue of worker threads before the actual scraping (and check if an entry exists in the cache) is done within a worker thread. However, let me think about it this idea.

abj

I have the feeling that the paths for the media and gamelist aren't correct in the latest version. If i remember correct, in the past, the metadata was into .emulastation folder. If i run the current Skyscraper the metadata are going into the usb -that i have my roms- without choose this. THanks.

From a fresh install of skyscraper :

;gameListFolder="/home/pi/RetroPie/roms"

;mediaFolder="/home/pi/RetroPie/roms"

The entire config.ini file:

; --------------------------------------------------------------------
; Skyscraper by Lars Muldjord (https://github.com/Gemba/skyscraper)
; --------------------------------------------------------------------

; This is an example config file for use with Skyscraper. Use it as a template for creating
;   your own. If you copy this file to 'config.ini' it will be used by default.
;   You can always copy config.ini.example to a filename of your choice and make Skyscraper
;   use it with the '-c CONFIGFILE' command line option.

; You can have several platform sections in one config file. Further down is an example of
;   a generic platform setup. Check '--help' for more info on supported platforms.

; You can have several scraping module sections in one config file. Further down is an example
;   of a generic module setup. Check '--help' for more info on supported modules.

; Remember that most of these settings can also be set as command line options. Doing so
;   will overrule the options seen here except for the scraping module configs which
;   overrules everything else.
; Platform specific configs overrule main configs.

; Uncomment the lines you wish to use (remove the ';' in front of the lines).
; Also remember to uncomment or create the sections they belong to such as '[snes]'.

[main]
;inputFolder="/home/pi/RetroPie/roms"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms"
;cacheFolder="/home/pi/.skyscraper/cache"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import"
;unpack="false"
;frontend="emulationstation"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;symlink="false"
;brackets="true"
;maxLength="10000"
;threads="2"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;maxFails="30"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;artworkXml="artwork.xml"
;relativePaths="false"
;addExtensions="*.zst"
;hints="false"
;subdirs="true"
;spaceCheck="true"
;scummIni="/full/path/to/scummvm.ini"
;tidyDesc="true"

; The following is an example of configs that only affect the 'snes' platform.
;[snes]
;inputFolder="/home/pi/RetroPie/roms/amiga"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes/snes/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes/snes/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;cacheFolder="/home/pi/.skyscraper/cache/amiga"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import/amiga"
;unpack="false"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;symlink="false"
;brackets="true"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;maxLength="10000"
;threads="2"
;startAt="filename"
;endAt="filename"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;artworkXml="artwork.xml"
;relativePaths="false"
;extensions="*.zip *.uae *.adf"
;addExtensions="*.zst *.rom"
;subdirs="true"

; The following is an example of configs that only affect the 'emulationstation' frontend.
[emulationstation]
;addFolders="false"

; The following is an example of configs that only affect the 'pegasus' frontend.
;[pegasus]
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;artworkXml="artwork.xml"
;emulator=""
;launch=""
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;skipped="false"
;brackets="true"
;videos="false"
;symlink="false"
;startAt="filename"
;endAt="filename"
;unattend="false"
;unattendSkip="false"
;forceFilename="false"
;verbosity="1"
;maxLength="10000"

; The following is an example of configs that only affect the 'screenscraper' scraping module.
;[screenscraper]
;userCreds="user:password"
;threads="1"
;minMatch="0"
;maxLength="10000"
;interactive="false"
;cacheResize="false"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;;The following option is only applicable to 'screenscraper'
;videoPreferNormalized="true"

[esgamelist]
cacheRefresh="true"
; https://github.com/RetroPie/RetroPie-Setup/pull/3704
cacheScreenshots="false"