Skyscraper is back! (v3.14)

Lolonois

@mitu No worries. I will happily put it below /etc... to keep the XDG police / policy out. :-D

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.10.0):

However, I am still puzzled why the symlink noted above is already on your system @sleve_mcdichael as it is not created with the current skyscraper.sh scriptmodule,

I have manually added this symlink a long time ago, as suggested in the CLIHELP.md document.

If you've installed Skyscraper through the RetroPie-Setup script, it is recommended to create a symbolic link to the executable. Do this by running sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper. This will allow you to just type Skyscraper when running it from command line.

sleve_mcdichael

@mitu @Lolonois seems to work if placed here, with no other modifications on my part:

~/.local/share/bash-completion/completions/Skyscraper.bash

Edit: https://github.com/RetroPie/RetroPie-Setup/pull/3872 is working for me (but am I understanding this might be system-dependent for different people?)

mitu

@sleve_mcdichael let's do the system install option.

YFZdude

@Lolonois
For the first time I used your updated version to scrape a system.
The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file. This broke the artwork in that system (and any others that were re-scraped). It also caused there to be warnings left in the terminal window about not updating FileData for unknown items when exiting ES.

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Also I do have a symlink in place of the standard RetroPie folder in the home folder so I can point it at a secondary hard disk. Not sure if that would affect this.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Hm, that's not good.
@Lolonois is this intended ? I don't see anything in the changelog about this change. This will break some setups where gamelists are saved in the ROMs folder and they rely on a relative path to work - this makes them non-portable.

The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file ...

That's not good, stable ES doesn't know about this. @Lolonois is there a way to disable the generation of folder items by default and hide it behind a parameter/option ?

YFZdude

@mitu
For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

That's not an issue. If it worked so far, it should continue to work.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

I'm not sure if they're necessary, but let's wait for @Lolonois - maybe they'd like to see the logs.

Lolonois

I had a fix for the regression ready, just pushed out 3.10.2.

I fixed a issue when a symlink is used for single romfiles provided to Skyscraper, but apperently introduced this regression.

Anyhow, now both use cases should be fine. Esp. the one reported here and in #38 on Github.

I gave your setup @YFZdude a testdrive with 3.10.2 and it looked good:

$ pwd
/home/pi
$ ls -l RetroPie
lrwxrwxrwx 1 pi pi 27 Feb 27 19:16 RetroPie -> /mnt/yabba/dabba/_RetroPie/
# below RetroPie/ it is roms/, BIOS/ aso. as directories.

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

Pls let us know if it remediates the situation on your side.

mitu

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

The default seems to be still to output the full path, instead of a relative path as before. Why the change ? Also, folder entries are still generated.

YFZdude

@Lolonois This seems to straighten out my issue.
I moved the existing gamelist to generate a new one and got the expected paths containing /home/<user>/RetroPie/roms/ etc.

I did not see any <folder> entries at the bottom of my newly generated list either. I was also not using the relative paths setting in any config file.

Thanks

Lolonois

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

[1] https://github.com/RetroPie/EmulationStation/blob/master/GAMELISTS.md#folder

Lolonois

@YFZdude Thanks for reporting back. This means your case is ok?

YFZdude

@Lolonois
Yes I have it working fine on my setup.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu The folder generation is deliberately added as it is part of the gamelist spec [1] if one organizes roms in directories per platform.

Yes, but that part is not present in most RetroPie installations - it's been only added to the -dev branch. Until this feature is present by default in RetroPie, I'm asking you to hide this under an option and default to not generate this info.

Can you provide the parts and sections of the config.ini with relativePaths settings? Do you use videos=true and have symlink also set true?

I have the default .ini, without anything else set. I didn't scrape for video. The default ini has:

[esgamelist]
cacheRefresh="true"
cacheScreenshots="false"

[import]
cacheRefresh="true"

enabled only.

And most importantly which symlinks are present up to /home/pi/RetroPie/roms/<platform>?

I don't have any symlinks that way, I only use $HOME/roms symlinked to $HOME/RetroPie/roms, but I don't see how that should affect scraping since the default RetroPie structure for ROM folders is intact.

Lolonois

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

@mitu thanks, as I am still on reconstruct the behaviour on my side, which commands do you issue from Skyscraper?

Since I have some SNES images in my cache, I just remove my gamelist.xml and run:

Skyscraper -p snes

and the gamelist.xml is re-created (in the ROMs folder - snes).

Lolonois

@mitu I tried to reconstruct the issue but had no luck. I put a also symlink from ~/roms to ~/RetroPie/roms deleted the extisting gamelist file and created from cache a new one.

relativePaths=false: The absolute paths show up in the gamelist file, the path up to the systems/platforms is expanded to the regular rom folder (/home/pi/RetroPie/roms) unless a specific inputFolder is given.
relativePaths=true: Entries in the gamelist are generated as relative, starting from the regular rom folder plus system/platform, f.i. the file on the filesystem /home/pi/RetroPie/roms/snes/Frogger (USA).zip is generated as ./Frogger (USA).zip in gamelist.

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

If there is still something with the generated paths then it must be a very specific setup I can not imagine.

I will follow-up on the folder thingie later.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

Yes, I checked also in the code and the default hasn't changed. The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off. Sorry for the confusion.

I will follow-up on the folder thingie later.

OK, thank you.

Lolonois

@mitu said in Skyscraper is back! (v3.10.0):

The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off.

Nvm. But good hint, reminds me to consider the dialog based scraping too on development.