Skyscraper is back! (v3.13)

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

The setting relativePaths=false is the default since Lars was developing Skyscraper, I did not change that default since then.

Yes, I checked also in the code and the default hasn't changed. The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off. Sorry for the confusion.

I will follow-up on the folder thingie later.

OK, thank you.

Lolonois

@mitu said in Skyscraper is back! (v3.10.0):

The relative flags is added by RetroPie's GUI when scraping, if the ROMs folder is chosen as destination for gamelist/artwork - that's what threw me off.

Nvm. But good hint, reminds me to consider the dialog based scraping too on development.

Lolonois

The <folder/> situation. (Long post but needed to create the context).

First off:

Skyscraper generates only <folder/> elements in gamelist when there are filesystem directories used for ROM organization inside a platform folder.
Even with ES 2.11.2rp the FileData FOLDER-type entries are created in memory (which then get persisted as <folder/> upon metadata edit+save): https://github.com/RetroPie/EmulationStation/blob/cda7de687924c4c1ab83d6b0ceb88aa734fe6cfe/es-app/src/Gamelist.cpp#L73
It seems to me very few users use the metadata edit on folders as the issues below in ES 2.11.2rp have been present several ES releases.

I tested forth and back between ES (2.11.2rp) and the ES-dev (main/head) in these combinations:

relativePaths=true and false in Skyscraper
gamelist with Skyscraper generated <folder/> entries and without.
Editing and persisting of folders in ES and ES-dev

Resume: I did not identify any regression with ES 2.11.2rp (and also not with ES-dev) introduced with the generated <folder/> elements of Skyscraper.

Here is the directory structure I have tested with

$ tree RetroPie/roms/apple2/
RetroPie/roms/apple2/
├── aaa
│   ├── bbbbb
│   │   └── Thief (1981)(Datamost)(San Inc & TRex crack).dsk
│   └── cccc
│       ├── MECC-A157 The Oregon Trail v1.1 (4am crack) side A.dsk
│       └── MECC-A157 The Oregon Trail v1.1 (4am crack) side B.dsk
├── ddddd
│   └── Multi Disk - Dung Beetles - Guardian - Zenith.dsk
├── eee
│   └── fff
│       └── ggg
│           └── Snakebyte.dsk
├── gamelist.xml
├── loderunr.zip
├── media
.   └── [truncated]
├── Mystery House.dsk
├── Sabotage (19xx)(Mark Allen).dsk
└── Three Mile Island Special Version (1980)(Richard Orban)[a].dsk

However, with ES 2.11.2rp the persistence has flaws. I tested with a gamelist which contained only <game/> elements:

1. Whenever metadata is edited for a folder in ES a new entry is added to the gamelist on save.

E.g. Edit fff metadata in ES once yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Edit fff twice yields:

        <folder>
                <path>eee/fff</path>
                <name>fff 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>
        <folder>                         <!-- erroneous 2nd entry for same path -->
                <path>eee/fff</path>
                <name>fff 2nd edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

This "entry cumulation" can also be reproduced when ES is quitted in between.

Eventually, the end-user will not notice, as in ES the last <folder/> entry "wins".

If such gamelist is updated with the current Skyscraper the first <folder/> entry wins, as Skyscraper expects there is in maximum one such entry for the same <path/>. Before v3.10 existing <folder/> entries were not preserved by Skyscraper when present in an existing gamelist.

2. Whenever metadata is edited for a folder which is direct child of the platform directory, the platform name is stored as part of the path on save.

E.g. edit aaa metadata in ES yields:

        <folder>
                <path>apple2/aaa</path>  <!-- note the surplus platform folder -->
                <name>aaa 1st edit</name>
                <rating>0</rating>
                <releasedate>19700101T010000</releasedate>
        </folder>

Upon restart of ES an error is logged (correct, as the path is invalid):

Feb 29 20:12:28 lvl0: 	Error finding/creating FileData for "/home/pi/RetroPie/roms/apple2/apple2/aaa", skipping.

Also in this case the entries get cumulated with each edit+save of aaa's metadata.

NB: These issues are not present ES-dev, i.e. ES-dev creates exactly one folder element or updates the existing one for each path, when metadata is edited.

TL;DR: I see these options.

Tell the few users of ES 2.11.2rp not to edit folder metadata (game metadata edit is not affected) in ES and then run Skyscraper (as they will lose their ES folder edits, unless there is in maximum one <folder/> entry per <path/>).
Make ES-dev the new ES-stable anytime soon.
Create a workaround for the ES 2.11.2rp issues in Skyscraper, that preserve the last <folder/> element when there are several <folder/> entries with the same <path/>. Additionally drop/repair those folder entries with an non-existing path (the aaa case above). Once the today's ES-dev is the new ES-stable, I can remove that workaround again. :/
Having a flag with "do-not-create-folders" in Skyscraper does not remediate the situation completely, because if there are two folder entries (the eee/fff case above) Skyscraper will pick only the first. Again, user edits from ES may get lost.

Do you see other options? What is your preferred option?

mitu

@Lolonois said in Skyscraper is back! (v3.10.0):

Do you see other options? What is your preferred option?

As said before, don't create folder entries by default. I'm not keen on workarounds that may change behavior today or tomorrow based on a release, so having a clear and documented method to disable the folder elements from Skyscraper is preferred. Thank you.

Lolonois

Version 3.10.3 restores the pre v3.10.0 behaviour, unless the config switch addFolders is deliberately set true.

Existing <folder/> entries in a gamelist XML will be kept, for any state of addFolders. If this is not wanted: Remove the gamelist.xml and let Skyscraper generate a new one.

https://github.com/Gemba/skyscraper/releases/tag/3.10.3

Lolonois

Another month, another Skyscraper release: Main additions are ES-DE frontend support and aliasMap entries now applicable also for Screenscraper.

All changes of v3.11.0

abj

I am prepering for the upcoming 3.11 version and the changes that will bring. I checked the docs and i solve all my questions, about the various parameters of config.ini file, except one that i didn't understand very well.

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

When generating gamelist xml, by default, if a rom file that has no scraped resources associated with it, will not be an entry created in the gamelist. Setting this skipped="true" causes empty "dummy" entries be created for these skipped roms.

When gathering data, by default, if a rom has any assets from this source (eg. screenscraper, thegamesdb) it will be skipped but if it has assets from other source, it will still be processed for this source also. By using the CLI option --flags onlymissing, roms with assets from any source will be skipped, and only those with no assets from any source are processed. I think this is probably what you want, in this case.

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

Lolonois

Behavior is as @sleve_mcdichael has written, thanks.

To amend to this:
Skyscraper uses three sources to create a new gamelist.xml: The online DB (like Screenscraper, aso.), it's local cache and if present the existing gamelist.xml file.

Manual changes of an existing gamelist.xml are preserved, whenever there is no data available either by the scraping from an online DB or in the cache.

If you want to have the manual edits stored in the skyscraper cache you may use the -s esgamelist to "scrape" from your existing gamelist.xml. Most likely you should add an entry to the priorities.xml file for that platform to use also esgamelist as cache source. There is more info on this topic here

To cover also the case you may referring to, @abj : Skyscraper writes always all entries of a gamelist.xml fresh (from the beforementioned sources), there is no option to skip entries while Skyscraper is in the last stage and creates the gamelist.xml

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.11):

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

That is right, it only exists as CLI option. It can be as useful permanent option, esp. for MobyGames which has a limit of 35 games to scrape at once, thus it might be easier with this flag to stay below that limit. In a technical nutshell: The limit is checked in Skyscraper by the size of the queue of worker threads before the actual scraping (and check if an entry exists in the cache) is done within a worker thread. However, let me think about it this idea.

abj

I have the feeling that the paths for the media and gamelist aren't correct in the latest version. If i remember correct, in the past, the metadata was into .emulastation folder. If i run the current Skyscraper the metadata are going into the usb -that i have my roms- without choose this. THanks.

From a fresh install of skyscraper :

;gameListFolder="/home/pi/RetroPie/roms"

;mediaFolder="/home/pi/RetroPie/roms"

The entire config.ini file:

; --------------------------------------------------------------------
; Skyscraper by Lars Muldjord (https://github.com/Gemba/skyscraper)
; --------------------------------------------------------------------

; This is an example config file for use with Skyscraper. Use it as a template for creating
;   your own. If you copy this file to 'config.ini' it will be used by default.
;   You can always copy config.ini.example to a filename of your choice and make Skyscraper
;   use it with the '-c CONFIGFILE' command line option.

; You can have several platform sections in one config file. Further down is an example of
;   a generic platform setup. Check '--help' for more info on supported platforms.

; You can have several scraping module sections in one config file. Further down is an example
;   of a generic module setup. Check '--help' for more info on supported modules.

; Remember that most of these settings can also be set as command line options. Doing so
;   will overrule the options seen here except for the scraping module configs which
;   overrules everything else.
; Platform specific configs overrule main configs.

; Uncomment the lines you wish to use (remove the ';' in front of the lines).
; Also remember to uncomment or create the sections they belong to such as '[snes]'.

[main]
;inputFolder="/home/pi/RetroPie/roms"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms"
;cacheFolder="/home/pi/.skyscraper/cache"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import"
;unpack="false"
;frontend="emulationstation"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;symlink="false"
;brackets="true"
;maxLength="10000"
;threads="2"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;maxFails="30"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;artworkXml="artwork.xml"
;relativePaths="false"
;addExtensions="*.zst"
;hints="false"
;subdirs="true"
;spaceCheck="true"
;scummIni="/full/path/to/scummvm.ini"
;tidyDesc="true"

; The following is an example of configs that only affect the 'snes' platform.
;[snes]
;inputFolder="/home/pi/RetroPie/roms/amiga"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes/snes/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes/snes/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;cacheFolder="/home/pi/.skyscraper/cache/amiga"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import/amiga"
;unpack="false"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;symlink="false"
;brackets="true"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;maxLength="10000"
;threads="2"
;startAt="filename"
;endAt="filename"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;artworkXml="artwork.xml"
;relativePaths="false"
;extensions="*.zip *.uae *.adf"
;addExtensions="*.zst *.rom"
;subdirs="true"

; The following is an example of configs that only affect the 'emulationstation' frontend.
[emulationstation]
;addFolders="false"

; The following is an example of configs that only affect the 'pegasus' frontend.
;[pegasus]
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;artworkXml="artwork.xml"
;emulator=""
;launch=""
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;skipped="false"
;brackets="true"
;videos="false"
;symlink="false"
;startAt="filename"
;endAt="filename"
;unattend="false"
;unattendSkip="false"
;forceFilename="false"
;verbosity="1"
;maxLength="10000"

; The following is an example of configs that only affect the 'screenscraper' scraping module.
;[screenscraper]
;userCreds="user:password"
;threads="1"
;minMatch="0"
;maxLength="10000"
;interactive="false"
;cacheResize="false"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;;The following option is only applicable to 'screenscraper'
;videoPreferNormalized="true"

[esgamelist]
cacheRefresh="true"
; https://github.com/RetroPie/RetroPie-Setup/pull/3704
cacheScreenshots="false"

Lolonois

@abj just to understand more of this:

Do you use the frontend EmulationStation during scraping (if you don't use the -f flag, EmulationStation is the default)?
Do you scrape from the command line or from the RetroPie-Setup dialog?
Do you have your previous config.ini still around?

You can still edit the gameListFolder entry in the [main] section of the config.ini to "/home/pi/.emulationstation/gamelists" which should get you the previous experience.

abj

@Lolonois

I runned the Skyscraper from the terminal window (i had exit from ES with F4) and without using the RetroPie-Setup dialog, just to get familiar with the commands for the upcoming v.3.11 .

Then i scrape from the command line with

Skyscraper -p psx -s screenscraper
Skyscraper -p psx

I don't have the previous config.ini file, because it was a fresh install of RetroPie and Skyscraper (Branch 3.10.4).

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug. Thanks for everything!

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug.

(Skyscraper default has always been to use the ROM folder for gamelists and media. Only the RP-Setup dialog (or CLI option, or manual edit config.ini) redirects it to RetroPie configs area instead.)

abj

@sleve_mcdichael This is the reason then, because the previous times i was using the gui. Thanks!

abj

Hello, it's me again (sorry :P).

I was trying to scrape my neogeocd games that i have into neogeo folder from the terminal window, without the use of RetroPie-Setup Gui.
The command i run was :

Skyscraper -p neogeo -s screenscraper

The command run but i didn't saw any metadata, it was like i didn't have roms in the folder.

I check the peas.json file and it looks that something is missing from the formats section:

},
    "neogeo": {
        "aliases": [
            "arcade",
            "neo geo",
            "neo geo aes",
            "neo geo cd",
            "neo geo mvs",
            "neo-geo",
            "neo-geo cd"
        ],
        "formats": [],
        "scrapers": [
            "arcadedb",
            "openretro",
            "screenscraper"
        ]
    },

sleve_mcdichael

@abj I'm not sure but you could try adding some "formats"? So:

        "formats": [],

...becomes:

        "formats": [
            "*.chd",
            "*.cue",
            "*.fba",
            "*.iso"
        ],

(NeoGeo formats from https://github.com/RetroPie/RetroPie-Setup/blob/master/platforms.cfg#L146, not include "*.zip", "*.7z" as none other systems include these either, so I suppose they are "hardwired" somewhere else.)

abj

@sleve_mcdichael

Thanks, this fix it!

Lolonois

if the option extension is unset (default) then these extensions are automatically used as base *.7z *.ml *.zip by Skyscraper.

Another option is to add the wanted extensions with addExtensions, these get appended to the base extensions.

Eventually the extensions from peas.json / formats are added also to the list of extensions.

Lolonois

Version 3.12.0 is out.

Implementing a batch of community suggestions, all details here

Thank you all and happy scraping!

Edit: Thanks, @mitu for spotting the wrong URL, corrected now. Seems I copied from the wrong browser tab yesterday.