Skyscraper is back! (v3.17)

abj

I am prepering for the upcoming 3.11 version and the changes that will bring. I checked the docs and i solve all my questions, about the various parameters of config.ini file, except one that i didn't understand very well.

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

I want to skip the already scraped roms. To do this, i have to enable this flag : ;skipped="false" ? Thanks.

When generating gamelist xml, by default, if a rom file that has no scraped resources associated with it, will not be an entry created in the gamelist. Setting this skipped="true" causes empty "dummy" entries be created for these skipped roms.

When gathering data, by default, if a rom has any assets from this source (eg. screenscraper, thegamesdb) it will be skipped but if it has assets from other source, it will still be processed for this source also. By using the CLI option --flags onlymissing, roms with assets from any source will be skipped, and only those with no assets from any source are processed. I think this is probably what you want, in this case.

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

Lolonois

Behavior is as @sleve_mcdichael has written, thanks.

To amend to this:
Skyscraper uses three sources to create a new gamelist.xml: The online DB (like Screenscraper, aso.), it's local cache and if present the existing gamelist.xml file.

Manual changes of an existing gamelist.xml are preserved, whenever there is no data available either by the scraping from an online DB or in the cache.

If you want to have the manual edits stored in the skyscraper cache you may use the -s esgamelist to "scrape" from your existing gamelist.xml. Most likely you should add an entry to the priorities.xml file for that platform to use also esgamelist as cache source. There is more info on this topic here

To cover also the case you may referring to, @abj : Skyscraper writes always all entries of a gamelist.xml fresh (from the beforementioned sources), there is no option to skip entries while Skyscraper is in the last stage and creates the gamelist.xml

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.11):

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

That is right, it only exists as CLI option. It can be as useful permanent option, esp. for MobyGames which has a limit of 35 games to scrape at once, thus it might be easier with this flag to stay below that limit. In a technical nutshell: The limit is checked in Skyscraper by the size of the queue of worker threads before the actual scraping (and check if an entry exists in the cache) is done within a worker thread. However, let me think about it this idea.

abj

I have the feeling that the paths for the media and gamelist aren't correct in the latest version. If i remember correct, in the past, the metadata was into .emulastation folder. If i run the current Skyscraper the metadata are going into the usb -that i have my roms- without choose this. THanks.

From a fresh install of skyscraper :

;gameListFolder="/home/pi/RetroPie/roms"

;mediaFolder="/home/pi/RetroPie/roms"

The entire config.ini file:

; --------------------------------------------------------------------
; Skyscraper by Lars Muldjord (https://github.com/Gemba/skyscraper)
; --------------------------------------------------------------------

; This is an example config file for use with Skyscraper. Use it as a template for creating
;   your own. If you copy this file to 'config.ini' it will be used by default.
;   You can always copy config.ini.example to a filename of your choice and make Skyscraper
;   use it with the '-c CONFIGFILE' command line option.

; You can have several platform sections in one config file. Further down is an example of
;   a generic platform setup. Check '--help' for more info on supported platforms.

; You can have several scraping module sections in one config file. Further down is an example
;   of a generic module setup. Check '--help' for more info on supported modules.

; Remember that most of these settings can also be set as command line options. Doing so
;   will overrule the options seen here except for the scraping module configs which
;   overrules everything else.
; Platform specific configs overrule main configs.

; Uncomment the lines you wish to use (remove the ';' in front of the lines).
; Also remember to uncomment or create the sections they belong to such as '[snes]'.

[main]
;inputFolder="/home/pi/RetroPie/roms"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms"
;cacheFolder="/home/pi/.skyscraper/cache"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import"
;unpack="false"
;frontend="emulationstation"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;symlink="false"
;brackets="true"
;maxLength="10000"
;threads="2"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;maxFails="30"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;artworkXml="artwork.xml"
;relativePaths="false"
;addExtensions="*.zst"
;hints="false"
;subdirs="true"
;spaceCheck="true"
;scummIni="/full/path/to/scummvm.ini"
;tidyDesc="true"

; The following is an example of configs that only affect the 'snes' platform.
;[snes]
;inputFolder="/home/pi/RetroPie/roms/amiga"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes/snes/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes/snes/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;cacheFolder="/home/pi/.skyscraper/cache/amiga"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import/amiga"
;unpack="false"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;symlink="false"
;brackets="true"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;maxLength="10000"
;threads="2"
;startAt="filename"
;endAt="filename"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;artworkXml="artwork.xml"
;relativePaths="false"
;extensions="*.zip *.uae *.adf"
;addExtensions="*.zst *.rom"
;subdirs="true"

; The following is an example of configs that only affect the 'emulationstation' frontend.
[emulationstation]
;addFolders="false"

; The following is an example of configs that only affect the 'pegasus' frontend.
;[pegasus]
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;artworkXml="artwork.xml"
;emulator=""
;launch=""
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;skipped="false"
;brackets="true"
;videos="false"
;symlink="false"
;startAt="filename"
;endAt="filename"
;unattend="false"
;unattendSkip="false"
;forceFilename="false"
;verbosity="1"
;maxLength="10000"

; The following is an example of configs that only affect the 'screenscraper' scraping module.
;[screenscraper]
;userCreds="user:password"
;threads="1"
;minMatch="0"
;maxLength="10000"
;interactive="false"
;cacheResize="false"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;;The following option is only applicable to 'screenscraper'
;videoPreferNormalized="true"

[esgamelist]
cacheRefresh="true"
; https://github.com/RetroPie/RetroPie-Setup/pull/3704
cacheScreenshots="false"

Lolonois

@abj just to understand more of this:

Do you use the frontend EmulationStation during scraping (if you don't use the -f flag, EmulationStation is the default)?
Do you scrape from the command line or from the RetroPie-Setup dialog?
Do you have your previous config.ini still around?

You can still edit the gameListFolder entry in the [main] section of the config.ini to "/home/pi/.emulationstation/gamelists" which should get you the previous experience.

abj

@Lolonois

I runned the Skyscraper from the terminal window (i had exit from ES with F4) and without using the RetroPie-Setup dialog, just to get familiar with the commands for the upcoming v.3.11 .

Then i scrape from the command line with

Skyscraper -p psx -s screenscraper
Skyscraper -p psx

I don't have the previous config.ini file, because it was a fresh install of RetroPie and Skyscraper (Branch 3.10.4).

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug. Thanks for everything!

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug.

(Skyscraper default has always been to use the ROM folder for gamelists and media. Only the RP-Setup dialog (or CLI option, or manual edit config.ini) redirects it to RetroPie configs area instead.)

abj

@sleve_mcdichael This is the reason then, because the previous times i was using the gui. Thanks!

abj

Hello, it's me again (sorry :P).

I was trying to scrape my neogeocd games that i have into neogeo folder from the terminal window, without the use of RetroPie-Setup Gui.
The command i run was :

Skyscraper -p neogeo -s screenscraper

The command run but i didn't saw any metadata, it was like i didn't have roms in the folder.

I check the peas.json file and it looks that something is missing from the formats section:

},
    "neogeo": {
        "aliases": [
            "arcade",
            "neo geo",
            "neo geo aes",
            "neo geo cd",
            "neo geo mvs",
            "neo-geo",
            "neo-geo cd"
        ],
        "formats": [],
        "scrapers": [
            "arcadedb",
            "openretro",
            "screenscraper"
        ]
    },

sleve_mcdichael

@abj I'm not sure but you could try adding some "formats"? So:

        "formats": [],

...becomes:

        "formats": [
            "*.chd",
            "*.cue",
            "*.fba",
            "*.iso"
        ],

(NeoGeo formats from https://github.com/RetroPie/RetroPie-Setup/blob/master/platforms.cfg#L146, not include "*.zip", "*.7z" as none other systems include these either, so I suppose they are "hardwired" somewhere else.)

abj

@sleve_mcdichael

Thanks, this fix it!

Lolonois

if the option extension is unset (default) then these extensions are automatically used as base *.7z *.ml *.zip by Skyscraper.

Another option is to add the wanted extensions with addExtensions, these get appended to the base extensions.

Eventually the extensions from peas.json / formats are added also to the list of extensions.

Lolonois

Version 3.12.0 is out.

Implementing a batch of community suggestions, all details here

Thank you all and happy scraping!

Edit: Thanks, @mitu for spotting the wrong URL, corrected now. Seems I copied from the wrong browser tab yesterday.

mitu

@Lolonois said in Skyscraper is back! (v3.12):

Implementing a batch of community suggestions, all details here

Hm, here points to localhost, probably should be https://gemba.github.io/skyscraper/CHANGELOG/#version-3120-2024-07-01 ?

Lolonois

3.13.0 is out.

Mainly a maintenance release, removed flags and options marked deprecated since version 3.7.0 (almost three years ago).
You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.
Added the platform Vircon32 (@Vircon32 ).
Many smaller fixes and improvements.

https://github.com/Gemba/skyscraper/releases/tag/3.13.0

Vircon32

@Lolonois that's great! Thank you for the addition.

Good timing too. Recently several Vircon32 games have been added to some scraping sites, like Screenscaper and IGDB.

Lolonois

Just released 3.14 with a bunch of changes:

You may benefit from the three new configuration options innerBracketsReplace, keepDiscInfo and ignoreYearInFilename) or if you use Skyscraper also on a non-Retropie Setup from the XDG Base Directory support.

https://gemba.github.io/skyscraper/CHANGELOG/#version-3140-2024-12-08

For 2025 I plan to implement fanart and bezel support and maybe also frontend support for Batocera Emulationstation.

If you have features in mind which would benefit the RetroPie community feel free to discuss them here.

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.14):

3.13.0 is out.

You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.

This is a great and welcome addition -- ideally, this will allow official upstream changes to co-exist with custom user mods, maintenance-free. Right now though, function copyDefaultConfig in the RP module still puts the new upstream version to peas.json.rp-dist and so update is no applied without manual copy this file to overwrite existing peas.json.

Eventually, will probably want to just overwrite with a regular cp so that changes are applied automatically, but, this could clobber any custom config for users not yet migrated to peas_local.json. Maybe first can use some if diff (looks like RP already have a diffFiles function for this) and ~~if so~~ (I think actually diffFiles check if they are the same, so if not), then save a peas.json.old. This would only give users one "update" worth of grace period to retrieve their edits before the .old version gets overwrite as well, but is still better than no grace.

Something along the lines of:

local json="$scraper_conf_dir/peas.json"
local pristine="$md_inst/.pristine_configs/peas.json"

if [[ -f "$json" ]] && ! diffFiles "$json" "$pristine"; then
    cp -f "$json" "$json.old"
fi
cp -f "$pristine" "$json"

...

Also, I do still have to put some custom entries into platforms_idmap.csv to scrape Screenscraper for the US version genesis and tg16 systems that I have added to peas_local. Can that one have the "local version" file option, as well? (And then it would have the same change to backup & overwrite in the RP module.)

Lolonois

Always welcomed to get feedback on how the software is used. :)

I did not consider a _local file for the CSV file, as I assumed it would be easy to maintain as it would be a single line compare. But you are right, it still has to be maintained. I will take the platforms_idmap_local.csv on the 3.15 release with a guide on how to separate the local additions from the distributed/"upstream" configs for the platform info.

However, I would not overwrite the existing peas/platform files automagically (even after a "grace" update as you noted) unless a user has opt-ed in for that (e.g. with a empty file like .platformcfg_overwrite_ok) and thus confirms that he/she saved additions in a _local file (or didn't change anything in the files peas.json / platforms_idmap.csv at all). On juvenile installations the .platformcfg_overwrite_ok would be created right away.