Problems with Skyscraper 3.17.3

sleve_mcdichael

I have unattendSkip="true"

First I was on 3.17.2 binary. I added one game & scraped. Instead of doing just the new game (previous, correct behavior), it ran through all 141 games & downloaded their assets again.

Saw on github there is an update to address this so I installed 3.17.3 from source but it only partially helped.

In 3.17.3 it does not download all the assets again, but it does still runs through all the 141 games, scraping "from cache" the ones already there, instead of skipping entirely ones already in gamelist.

On build mode, it again runs through all games, not just the new one, and creating duplicate (triplicate, etc. on subsequent runs) entries in the gamelist for the ones already there.

Reproduce:

Have existing SNES roms, cache, and gamelist. Add a new game zip to the rom folder, or manually remove one game from gamelist, and then scrape or build:

Skyscraper -p snes -s screenscraper --flags unattendskip

Skyscraper -p snes --flags unattendskip

Expected: only the one new game is processed

Observed: all 141 games are processed. In scrape, most are quickly scraped "from cache" and only the new one downloads any assets, but all are ran across the screen when most should be skipped entirely. In build, all 141 games are processed and duplicate entries are created (gamelist roughly doubles in size).

When revert to 3.16.1 then it works again as expected. Can repair the duplicated-entries list by running through once with --flags unattend (not skip).

Cache might be corrupted (after 3.17.2?) When I do rebuild the list this way I'm only getting the original data from screenscraper and not my custom edits from esgamelist. Works again after I restore a backup list and scrape esgamelist again before rebuild.

...

Additionally, all paths in the 3.17.3 list appear as relative to the gamelist folder:

<path>./../../../RetroPie/roms/snes/F-Zero (USA).zip</path>

<image>./../../downloaded_media/snes/screenshots/F-Zero (USA).png</image>

...but EmulationStation reads this as if it's relative to the ROM folder. Which works for the rom paths (*well sort of, and by mere coincidence), but not for the downloaded media.

*In my SNES system in ES now I have 2 copies of every game: one with the "generic" name and no data (from the rom file in the rom folder, with no gamelist xml entry), and then a second one by entering the folders . > . > . > RetroPie > ROMS > SNES (back where we started, but the long way -- also they do all show as single dot . in the ES list like that even though double dot .. in the gamelist XML), these with correct names and descriptions from the gamelist entry, but still no image. Both ways launch and play the game.

Lolonois

Well, seems I did not catch these edge cases.
Anyhow, thanks for tabling them.

To sum up you noticed:

Enabling unattendSkip does relookup existing entries in the gamelist and rewrites also these entries
Rerunning gamelist creation with unattendSkip does add another set of the first complete set to the gamelist
Data from -s esgamelist scraped earlier, present in the cache, does not show up in the generated gamelist, even priorities.xml for that platform advises to rank esgamelist higher than screenscraper.
A gamelist not residing in ~/RetroPie/roms/<platform> does mislead ES to show another set of the first set of games but with one dot as path and not .. (assuming relativePaths=true and in ES settings ParseGamelistOnly="false").

To be able to easier dissect these could you pls provide (longer snippets as private gist, pastebin, ...):

The ES version you are using
The paths of one system to its ROMs, to its gamelist and its media and respective any changes made to inputFolder, gamelistFolder, mediaFolder or CLI switches provided (-i, -g, -o). (I do not assume you applied the cacheFolder / -d between 3.16.1 and the current version)
The (truncated) gamelist as it was in 3.16 and how it was generated with 3.17, also as a "twice", "triple" effect version
A truncated gamelist which you used to scrape from with -s esgamelist

Thanks a lot!

PS: FYI the path processing has been reworked with 3.17

sleve_mcdichael

@Lolonois

The ES version you are using

I have just updated and the lists still act weird in emulationstation-dev latest binary:
Branch: master
Commit: 9e594dfe
Date: 2025-05-19

...but I think this is more about the XML's format being messed up, than any bug in ES.

The paths of one system to its ROMs, to its gamelist and its media and respective any changes made to inputFolder, gamelistFolder, mediaFolder or CLI switches provided (-i, -g, -o). (I do not assume you applied the cacheFolder / -d between 3.16.1 and the current version)

For the SNES paths I have:

/home/pi/RetroPie/roms/snes
/home/pi/.emulationstation/gamelists/snes
/home/pi/.emulationstation/downloaded_media/snes

In config.ini this looks like:

[main]

;; define paths
; inputFolder="/home/pi/RetroPie/roms"
gameListFolder="/home/pi/.emulationstation/gamelists"
mediaFolder="/home/pi/.emulationstation/downloaded_media"
; cacheFolder="/home/pi/.skyscraper/cache"
; importFolder="/home/pi/.skyscraper/import"

;; misc
unattendSkip="true"

The (truncated) gamelist as it was in 3.16 and how it was generated with 3.17, also as a "twice", "triple" effect version

In 3.16.1, a fresh list generated with --flags unattend looks like:

gamelist.xml.3161.1.unattend.clean

https://pastebin.com/mzAZrdAG

In 3.17.3 and ran with unattendSkip this becomes:

gamelist.xml.3173.1

https://pastebin.com/XuH9PYY5

...and running it one more time produces:

gamelist.xml.3173.2

https://pastebin.com/RWG8aSfp

A "clean" 3.17.3 with --flags unattend:

gamelist.xml.3173.3.unattend

https://pastebin.com/TyRN0Dae

And once more with unattendSkip, looks much like before but missing the original 3.16.1 entry:

gamelist.xml.3173.4

https://pastebin.com/6zV4KWSE

...all of these so far have been with relativePaths="true". If I change that to "false" and then run --flags unattend once more, I get this working list:

gamelist.xml.3173.5.unattend.relativePathsFalse

https://pastebin.com/dzwh2PAm

At this point, it seems to behave normally. If I run unattendSkip then it finds No files to process in cache for platform 'snes'., and then Skyscraper came to an untimely end. Or if I manually remove a game from the XML and rerun, then it processes just the missing entry.

This is the correct behavior I expected, previously.

However, with relativePaths="false", then if I take the original 3.16.1 list and run it unattendskip then it still duplicates the entries, additionally misinterpreting the path (./ActRaiser (USA).zip) from the original entry:

gamelist.xml.3173.6.relativePathsFalse.from3161

https://pastebin.com/tvemFHsN

A truncated gamelist which you used to scrape from with -s esgamelist

This I can't reproduce without using 3.17.2. The cache did become corrupted after I scraped a single file in 3.17.2 -- the subsequent built list contained only that single game and "not found" for all the rest. But since this only happen in the outdated version I'm calling it moot.

PS: FYI the path processing has been reworked with 3.17

"Front end output: input folder and media folder are relative to the gamelist folder." Isn't that wrong (for ES, anyway)?

I think it should be "gamelist and media folders are relative to the input folder."

This works, and always has:

<path>./ActRaiser (USA).zip</path>

As I understand, the ./ there refers to the <path> defined in es_systems.cfg, or ie. the rom folder/input folder.

This...is a very long route back to the same place:

<path>./../../../RetroPie/roms/snes/ActRaiser (USA).zip</path>

For images, this works:

<image>./../../../.emulationstation/downloaded_media/snes/screenshots/ActRaiser (USA).png</image>

Meanwhile, this doesn't:

<image>./../../downloaded_media/snes/screenshots/ActRaiser (USA).png</image>

Edited to add:

After playing a game, then ES itself writes the paths as (rom, relative to romdir):

<path>./ActRaiser (USA).zip</path>

And (image, absolute with ~ shortcut):

<image>~/.emulationstation/downloaded_media/snes/screenshots/ActRaiser (USA).png</image>

Compare Zombies Ate My Neighbors (entry generated by Skyscraper 3.17.3 w/ relative paths OFF) vs ActRaiser (entry generated by ES):

https://pastebin.com/G938dJWY

Lolonois

Thanks again @sleve_mcdichael. This will make the identification of the culprit much easier, but there is a catch: I will be away for a few weeks, but will definitely fix this end of June.

I was just asking for the ES version, to test on the same version, not that I was thinking ES has an issue.

After some thinking, I guess my oversight that the gamelists may also reside inside ~/.emulationstation/gamelists/ could be the cause of some if not all issues.

Setting the gamelistFolder to the same as the inputFolder and copy the existing gamelists to the inputFolder (i.e. sibling to the games) could be a workaround. However, it is a guess, I did not verify it by myself.

Lolonois

@sleve_mcdichael testing looks good on my side, I have fixed the root cause and the knock-on effects.

The issue with ES is special (replacing the absolute $HOME path with ~ in a gamelist file when absolute paths used) but it is okay: I deal with it in Skyscraper from now.

Could you please test this branch [1] with your config for ES frontend as time permits?

Thanks for reporting back, then I can ship 3.17.4.

[1] https://github.com/Gemba/skyscraper/tree/3.17.3_fix_relpath_ES

sleve_mcdichael

@Lolonois it's functional but still have some minor issue. It does process just the one new game now instead of trying to run through all existing ones, and entries are no longer duplicated on rebuild.

But in specific circumstance (change relativePaths="true" to false, add 1 or more new game, then rebuild list with unattendSkip) then the existing games end up with media entries looking like:

    <image>/home/pi/RetroPie/roms/snes/../../../.emulationstation/downloaded_media/snes/screenshots/ActRaiser (USA).png</image>
    <marquee>/home/pi/RetroPie/roms/snes/../../../.emulationstation/downloaded_media/snes/marquees/ActRaiser (USA).png</marquee>

Note how it path into the roms directory first, before it reverse back out and then go into .emulationstation/downloaded_media.

Self-corrects if ran with unattend:

    <image>/home/pi/.emulationstation/downloaded_media/snes/screenshots/ActRaiser (USA).png</image>
    <marquee>/home/pi/.emulationstation/downloaded_media/snes/marquees/ActRaiser (USA).png</marquee>

...

Re: unattend - in order to speed up this step of testing I ran with flags skipexisting on for screenshots, marquees, and videos, as to not unnecessarily reprocess/copy these assets -- and it seems that skipexistingvideos is broken, it results in empty video tags every time:

    <video/>

If ran again without skipexistingvideos, then the video tags are restored:

    <video>/home/pi/.emulationstation/downloaded_media/snes/videos/ActRaiser (USA).mp4</video>

This happens either I have symlinks="true" or not.

Lolonois

@sleve_mcdichael Thanks for reporting back, and thanks for unearthing the additional two bugs. Both confirmed and squashed. (See update in github branch).

The skipExistingVideo (and manual btw), was easy to reproduce but tricky to fix, as the code section in question was not my masterpiece. I ironed out it a bit.

The other effect was tricky to reproduce but easy to fix: To reproduce I did

Generate a full gamelist with relativePaths ON
Deliberately remove at least one game entry from the gamelist to simulate a new entry in the next run
Set unattend to OFF and unattendSkip to ON and relativePaths to OFF
Recreate the gamelist

With the fix and this procedure above the de-normalized path segments (e.g. a/b/c/../../../) at the existing gamelist entries is no longer reproducable, means they appear normalized as for the the new entry.

Once you confirm and have no other issues I will release 3.17.4.

Thanks again for spotting these issues, I assume you have eagle eyes.

sleve_mcdichael

@Lolonois said in Problems with Skyscraper 3.17.3:

Once you confirm and have no other issues I will release 3.17.4.

Confirm, I can no longer reproduce these in 8224e449.

Thanks again for spotting these issues, I assume you have eagle eyes.

My special talent lies in finding new and interesting ways to break stuff :)

Lolonois

3.17.4 is out to fix all the reported problems. 🙂