Skyscraper is back! (v3.13)

mitu

@sleve_mcdichael It works for me, so could be a local config issue. Can you post your skyscraper.ini, removing any personal/password fields ?

sleve_mcdichael

@mitu

[main]

;; define paths
; inputFolder="/home/pi/RetroPie/roms"
gameListFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"
; cacheFolder="/home/pi/.skyscraper/cache"
; importFolder="/home/pi/.skyscraper/import"

;; cache settings
; cacheCovers="true"
; cacheScreenshots="true"
; cacheWheels="true"
cacheMarquees="false"
cacheTextures="false"

;; video settings
videos="true"
videoSizeLimit="42"
videoConvertCommand="videoconvert.sh %i %o"
videoConvertExtension="mp4"
symlink="true"

;; artwork
; artworkXml="artwork-wide.xml"
artworkXml="artwork-cover.xml"

;; generic
relativePaths="true"
brackets="false"
subdirs="false"
theInFront="true"
unattendSkip="true"
maxLength="10000"

[arcade]
; artworkXml="artwork-tall.xml"

[gb]
; artworkXml="artwork-tall.xml"
;; allow gbc titles in gb folder
addExtensions="*.gbc"

[gba]
; artworkXml="artwork-wide.xml"

[gbc]
; artworkXml="artwork-tall.xml"

[genesis]
;; allow sega cd titles in genesis folder
addExtensions="*.iso *.cue *.chd"

[pcengine]
;; use 'tg16' folders for pce titles
;; (deprecated - use custom system in peas.json/platforms_idmap.csv instead)
;inputFolder="/home/pi/RetroPie/roms/tg16"
;gameListFolder="/home/pi/.emulationstation/gamelists/tg16"
;mediaFolder="/home/pi/.emulationstation/downloaded_media/tg16"
;cacheFolder="/home/pi/.skyscraper/cache/tg16"
;importFolder="/home/pi/.skyscraper/import/tg16"

[screenscraper]
;; userCreds="user:pass"
userCreds="[user]:[pass]"

[esgamelist]
;; import customized ES textual data
cacheRefresh="true"
unattend="true"
videos="false"
cacheCovers="false"
cacheScreenshots="false"
cacheWheels="false"
cacheMarquees="false"

[import]
cacheRefresh="true"

mitu

@sleve_mcdichael Hm, it doesn't seem related to the config file.
Are you sure that the bash completion file is loaded ? Can you try running source $HOME/.bash_completion.d/Skyscraper.bash and try again ?

sleve_mcdichael

@mitu said in Skyscraper is back! (v3.10.0):

Can you try running source $HOME/.bash_completion.d/Skyscraper.bash and try again ?

Yes, this works. Only until I log off and back on again (SSH), then it not longer works again.

Do you suppose it's related to one of them 251 packages, or why else won't this file be loading?

@sleve_mcdichael said in Skyscraper is back! (v3.10.0):

My RP-Setup is current, but underlying Raspbian Buster is not (251 packages can be upgraded. "If it ain't broke, don't fix it.") Should that make a difference?

https://pastebin.com/KGfQri07

mitu

@sleve_mcdichael said in Skyscraper is back! (v3.10.0):

Do you suppose it's related to one of them 251 packages, or why else won't this file be loading?

No, I don't thinks so. Do you have the bash-completion package installed ?

sleve_mcdichael

@mitu yep:

$ apt list bash-completion 
Listing... Done
bash-completion/oldoldstable,now 1:2.8-6 all [installed]

mitu

@sleve_mcdichael then it's bug in the installation instructions - I think it's not working also on my system OOB.

@Lolonois are you sure that using $HOME/.bash_completion.d is the right place to copy user-defined Bash completion files ? It doesn't seem to sourced automatically by bash_completion.

Lolonois

Do you have a ~/.bash_completion @sleve_mcdichael ?

If not create one to pick the Skyscraper bash completion up:

cat <<'EOF' >~/.bash_completion
for bcfile in ~/.bash_completion.d/* ; do
    [ -f "$bcfile" ] && source "$bcfile"
done
EOF

Another bash -i should give you the completion and also this should survive reboots.

Apperently I forgot that I did this ~/.bash_completion creation for another bash completion already back in December 2021. :-/

However, I am still puzzled why the symlink noted above is already on your system @sleve_mcdichael as it is not created with the current skyscraper.sh scriptmodule, but is needed also for bash completion.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

Apperently I forgot that I did this ~/.bash_completion creation for another bash completion already back in December 2021. :-/

Then maybe is better to install the completion file system-wide (/etc/bash_completion.d/ ?) if there's no standard user defined location for completion files.

Lolonois

@mitu ok for me to put it in the global /etc/bash_completion.d/ when you maintainers are. This reduces the logic in the scriptmodule where to put the completion file and do additional actions, depending on bash completion version and XDG compliant setups.

There is a lot of context here, FWIW.

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

There is a lot of context here, FWIW.

Oh, no, not the XDG_ police !

Lolonois

@mitu No worries. I will happily put it below /etc... to keep the XDG police / policy out. :-D

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.10.0):

However, I am still puzzled why the symlink noted above is already on your system @sleve_mcdichael as it is not created with the current skyscraper.sh scriptmodule,

I have manually added this symlink a long time ago, as suggested in the CLIHELP.md document.

If you've installed Skyscraper through the RetroPie-Setup script, it is recommended to create a symbolic link to the executable. Do this by running sudo ln -s /opt/retropie/supplementary/skyscraper/Skyscraper /usr/local/bin/Skyscraper. This will allow you to just type Skyscraper when running it from command line.

sleve_mcdichael

@mitu @Lolonois seems to work if placed here, with no other modifications on my part:

~/.local/share/bash-completion/completions/Skyscraper.bash

Edit: https://github.com/RetroPie/RetroPie-Setup/pull/3872 is working for me (but am I understanding this might be system-dependent for different people?)

mitu

@sleve_mcdichael let's do the system install option.

YFZdude

@Lolonois
For the first time I used your updated version to scrape a system.
The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file. This broke the artwork in that system (and any others that were re-scraped). It also caused there to be warnings left in the terminal window about not updating FileData for unknown items when exiting ES.

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Also I do have a symlink in place of the standard RetroPie folder in the home folder so I can point it at a secondary hard disk. Not sure if that would affect this.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

Am I just missing something simple like enabling the relative paths option in the config file? It never used to do this in the past. Old gamelists generated using 3.7 and lower had either relative paths, or paths that started with /home/<user>/RetroPie.

Hm, that's not good.
@Lolonois is this intended ? I don't see anything in the changelog about this change. This will break some setups where gamelists are saved in the ROMs folder and they rely on a relative path to work - this makes them non-portable.

The resulting gamelist file started using absolute paths and sticking a bunch of <folder> stuff at the bottom of the file ...

That's not good, stable ES doesn't know about this. @Lolonois is there a way to disable the generation of folder items by default and hide it behind a parameter/option ?

YFZdude

@mitu
For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

mitu

@YFZdude said in Skyscraper is back! (v3.10.0):

For reference, I only symlinked the RetroPie folder in the home directory to a different drive. Anything in the /opt/retropie folder should be standard path.

That's not an issue. If it worked so far, it should continue to work.

If it helps I can upload some files/screenshots to show the things I see when it breaks.

I'm not sure if they're necessary, but let's wait for @Lolonois - maybe they'd like to see the logs.

Lolonois

I had a fix for the regression ready, just pushed out 3.10.2.

I fixed a issue when a symlink is used for single romfiles provided to Skyscraper, but apperently introduced this regression.

Anyhow, now both use cases should be fine. Esp. the one reported here and in #38 on Github.

I gave your setup @YFZdude a testdrive with 3.10.2 and it looked good:

$ pwd
/home/pi
$ ls -l RetroPie
lrwxrwxrwx 1 pi pi 27 Feb 27 19:16 RetroPie -> /mnt/yabba/dabba/_RetroPie/
# below RetroPie/ it is roms/, BIOS/ aso. as directories.

Gamelists get created as expected (depending on the relativePaths setting with or without the RetropIe platform rom path (/home/pi/RetroPie/roms/<platform>/)).

Pls let us know if it remediates the situation on your side.