Skyscraper is back! (v3.12)

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.10.0):

However, I am still puzzled why the symlink noted above is already on your system @sleve_mcdichael as it is not created with the current skyscraper.sh scriptmodule,

I have manually added this symlink a long time ago, as suggested in the CLIHELP.md document.

If you've installed Skyscraper through the RetroPie-Setup script, it is recommended to create a symbolic link to the executable. Do this by running sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper. This will allow you to just type Skyscraper when running it from command line.

sleve_mcdichael

@mitu @Lolonois seems to work if placed here, with no other modifications on my part:

~/.local/share/bash-completion/completions/Skyscraper.bash

Edit: https://github.com/RetroPie/RetroPie-Setup/pull/3872 is working for me (but am I understanding this might be system-dependent for different people?)

mitu

@sleve_mcdichael let's do the system install option.

YFZdude

@Lolonois
For the first time I used your updated version to scrape a system.
The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file. This broke the artwork in that system (and any others that were re-scraped). It also caused there to be warnings left in the terminal window about not updating FileData for unknown items when exiting ES.

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Also I do have a symlink in place of the standard RetroPie folder in the home folder so I can point it at a secondary hard disk. Not sure if that would affect this.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Hm, that's not good.
@Lolonois is this intended ? I don't see anything in the changelog about this change. This will break some setups where gamelists are saved in the ROMs folder and they rely on a relative path to work - this makes them non-portable.

The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file ...

That's not good, stable ES doesn't know about this. @Lolonois is there a way to disable the generation of folder items by default and hide it behind a parameter/option ?

YFZdude

@mitu
For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

That's not an issue. If it worked so far, it should continue to work.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

I'm not sure if they're necessary, but let's wait for @Lolonois - maybe they'd like to see the logs.

Lolonois

I had a fix for the regression ready, just pushed out 3.10.2.

I fixed a issue when a symlink is used for single romfiles provided to Skyscraper, but apperently introduced this regression.

Anyhow, now both use cases should be fine. Esp. the one reported here and in #38 on Github.

I gave your setup @YFZdude a testdrive with 3.10.2 and it looked good:

$ pwd
/home/pi
$ ls -l RetroPie
lrwxrwxrwx 1 pi pi 27 Feb 27 19:16 RetroPie -> /mnt/yabba/dabba/_RetroPie/
# below RetroPie/ it is roms/, BIOS/ aso. as directories.

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

Pls let us know if it remediates the situation on your side.

mitu

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

The default seems to be still to output the full path, instead of a relative path as before. Why the change ? Also, folder entries are still generated.

YFZdude

@Lolonois This seems to straighten out my issue.
I moved the existing gamelist to generate a new one and got the expected paths containing /home/<user>/RetroPie/roms/ etc.

I did not see any <folder> entries at the bottom of my newly generated list either. I was also not using the relative paths setting in any config file.

Thanks

Lolonois

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

[1] https://github.com/RetroPie/EmulationStation/blob/master/GAMELISTS.md#folder

Lolonois

@YFZdude Thanks for reporting back. This means your case is ok?

YFZdude

@Lolonois
Yes I have it working fine on my setup.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Yes, but that part is not present in most RetroPie installations - it's been only added to the -dev branch. Until this feature is present by default in RetroPie, I'm asking you to hide this under an option and default to not generate this info.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

I have the default .ini, without anything else set. I didn't scrape for video. The default ini has:

[esgamelist]
cacheRefresh="true"
cacheScreenshots="false"

[import]
cacheRefresh="true"

enabled only.

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

I don't have any symlinks that way, I only use $HOME/roms symlinked to $HOME/RetroPie/roms, but I don't see how that should affect scraping since the default RetroPie structure for ROM folders is intact.

Lolonois

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

Since I have some SNES images in my cache, I just remove my gamelist.xml and run:

Skyscraper -p snes

and the gamelist.xml is re-created (in the ROMs folder - snes).

Lolonois

@mitu I tried to reconstruct the issue but had no luck. I put a also symlink from ~/roms to ~/RetroPie/roms deleted the extisting gamelist file and created from cache a new one.

relativePaths=false: The absolute paths show up in the gamelist file, the path up to the systems/platforms is expanded to the regular rom folder (/home/pi/RetroPie/roms) unless a specific inputFolder is given.
relativePaths=true: Entries in the gamelist are generated as relative, starting from the regular rom folder plus system/platform, f.i. the file on the filesystem /home/pi/RetroPie/roms/snes/Frogger (USA).zip is generated as ./Frogger (USA).zip in gamelist.

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

If there is still something with the generated paths then it must be a very specific setup I can not imagine.

I will follow-up on the folder thingie later.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

Yes, I checked also in the code and the default hasn't changed. The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off. Sorry for the confusion.

I will follow-up on the folder thingie later.

OK, thank you.

Lolonois

@mitu said in Skyscraper is back! (v3.10.0):

The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off.

Nvm. But good hint, reminds me to consider the dialog based scraping too on development.

Lolonois

The <folder/> situation. (Long post but needed to create the context).

First off:

Skyscraper generates only <folder/> elements in gamelist when there are filesystem directories used for ROM organization inside a platform folder.
Even with ES 2.11.2rp the FileData FOLDER-type entries are created in memory (which then get persisted as <folder/> upon metadata edit+save): https://github.com/RetroPie/EmulationStation/blob/cda7de687924c4c1ab83d6b0ceb88aa734fe6cfe/es-app/src/Gamelist.cpp#L73
It seems to me very few users use the metadata edit on folders as the issues below in ES 2.11.2rp have been present several ES releases.

I tested forth and back between ES (2.11.2rp) and the ES-dev (main/head) in these combinations:

relativePaths=true and false in Skyscraper
gamelist with Skyscraper generated <folder/> entries and without.
Editing and persisting of folders in ES and ES-dev

Resume: I did not identify any regression with ES 2.11.2rp (and also not with ES-dev) introduced with the generated <folder/> elements of Skyscraper.

Here is the directory structure I have tested with

$ tree RetroPie/roms/apple2/
RetroPie/roms/apple2/
├── aaa
│   ├── bbbbb
│   │   └── Thief (1981)(Datamost)(San Inc & TRex crack).dsk
│   └── cccc
│       ├── MECC-A157 The Oregon Trail v1.1 (4am crack) side A.dsk
│       └── MECC-A157 The Oregon Trail v1.1 (4am crack) side B.dsk
├── ddddd
│   └── Multi Disk - Dung Beetles - Guardian - Zenith.dsk
├── eee
│   └── fff
│       └── ggg
│           └── Snakebyte.dsk
├── gamelist.xml
├── loderunr.zip
├── media
.   └── [truncated]
├── Mystery House.dsk
├── Sabotage (19xx)(Mark Allen).dsk
└── Three Mile Island Special Version (1980)(Richard Orban)[a].dsk

However, with ES 2.11.2rp the persistence has flaws. I tested with a gamelist which contained only <game/> elements:

1. Whenever metadata is edited for a folder in ES a new entry is added to the gamelist on save.

E.g. Edit fff metadata in ES once yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Edit fff twice yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>
        <folder>                         <!-- erroneous 2nd entry for same path -->
                <path>eee/fff</path>
                <name>fff 2nd edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

This "entry cumulation" can also be reproduced when ES is quitted in between.

Eventually, the end-user will not notice, as in ES the last <folder/> entry "wins".

If such gamelist is updated with the current Skyscraper the first <folder/> entry wins, as Skyscraper expects there is in maximum one such entry for the same <path/>. Before v3.10 existing <folder/> entries were not preserved by Skyscraper when present in an existing gamelist.

2. Whenever metadata is edited for a folder which is direct child of the platform directory, the platform name is stored as part of the path on save.

E.g. edit aaa metadata in ES yields:

        <folder>
                <path>apple2/aaa</path>  <!-- note the surplus platform folder -->
                <name>aaa 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Upon restart of ES an error is logged (correct, as the path is invalid):

Feb 29 20:12:28 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/apple2/apple2/aaa", skipping.

Also in this case the entries get cumulated with each edit+save of aaa's metadata.

NB: These issues are not present ES-dev, i.e. ES-dev creates exactly one folder element or updates the existing one for each path, when metadata is edited.

TL;DR: I see these options.

Tell the few users of ES 2.11.2rp not to edit folder metadata (game metadata edit is not affected) in ES and then run Skyscraper (as they will lose their ES folder edits, unless there is in maximum one <folder/> entry per <path/>).
Make ES-dev the new ES-stable anytime soon.
Create a workaround for the ES 2.11.2rp issues in Skyscraper, that preserve the last <folder/> element when there are several <folder/> entries with the same <path/>. Additionally drop/repair those folder entries with an non-existing path (the aaa case above). Once the today's ES-dev is the new ES-stable, I can remove that workaround again. :/
Having a flag with "do-not-create-folders" in Skyscraper does not remediate the situation completely, because if there are two folder entries (the eee/fff case above) Skyscraper will pick only the first. Again, user edits from ES may get lost.

Do you see other options? What is your preferred option?