Skyscraper is back! (v3.17)

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.11):

@Lolonois, this (only missing) doesn't appear configurable by config.ini, is that correct or an oversight? I could foresee wanting to set it permanently for certain module for example [mobygames].

That is right, it only exists as CLI option. It can be as useful permanent option, esp. for MobyGames which has a limit of 35 games to scrape at once, thus it might be easier with this flag to stay below that limit. In a technical nutshell: The limit is checked in Skyscraper by the size of the queue of worker threads before the actual scraping (and check if an entry exists in the cache) is done within a worker thread. However, let me think about it this idea.

abj

I have the feeling that the paths for the media and gamelist aren't correct in the latest version. If i remember correct, in the past, the metadata was into .emulastation folder. If i run the current Skyscraper the metadata are going into the usb -that i have my roms- without choose this. THanks.

From a fresh install of skyscraper :

;gameListFolder="/home/pi/RetroPie/roms"

;mediaFolder="/home/pi/RetroPie/roms"

The entire config.ini file:

; --------------------------------------------------------------------
; Skyscraper by Lars Muldjord (https://github.com/Gemba/skyscraper)
; --------------------------------------------------------------------

; This is an example config file for use with Skyscraper. Use it as a template for creating
;   your own. If you copy this file to 'config.ini' it will be used by default.
;   You can always copy config.ini.example to a filename of your choice and make Skyscraper
;   use it with the '-c CONFIGFILE' command line option.

; You can have several platform sections in one config file. Further down is an example of
;   a generic platform setup. Check '--help' for more info on supported platforms.

; You can have several scraping module sections in one config file. Further down is an example
;   of a generic module setup. Check '--help' for more info on supported modules.

; Remember that most of these settings can also be set as command line options. Doing so
;   will overrule the options seen here except for the scraping module configs which
;   overrules everything else.
; Platform specific configs overrule main configs.

; Uncomment the lines you wish to use (remove the ';' in front of the lines).
; Also remember to uncomment or create the sections they belong to such as '[snes]'.

[main]
;inputFolder="/home/pi/RetroPie/roms"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms"
;cacheFolder="/home/pi/.skyscraper/cache"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import"
;unpack="false"
;frontend="emulationstation"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;symlink="false"
;brackets="true"
;maxLength="10000"
;threads="2"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;maxFails="30"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;artworkXml="artwork.xml"
;relativePaths="false"
;addExtensions="*.zst"
;hints="false"
;subdirs="true"
;spaceCheck="true"
;scummIni="/full/path/to/scummvm.ini"
;tidyDesc="true"

; The following is an example of configs that only affect the 'snes' platform.
;[snes]
;inputFolder="/home/pi/RetroPie/roms/amiga"
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;excludeFrom="/home/pi/.skyscraper/excludes/snes/excludes.txt"
;includeFrom="/home/pi/.skyscraper/includes/snes/includes.txt"
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;cacheFolder="/home/pi/.skyscraper/cache/amiga"
;cacheResize="false"
;nameTemplate="%t [%f], %P player(s)"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;importFolder="/home/pi/.skyscraper/import/amiga"
;unpack="false"
;emulator=""
;launch=""
;videos="false"
;videoSizeLimit="42"
;symlink="false"
;brackets="true"
;lang="en"
;region="wor"
;langPrios="en,de,es"
;regionPrios="eu,us,ss,uk,wor,jp"
;minMatch="0"
;maxLength="10000"
;threads="2"
;startAt="filename"
;endAt="filename"
;pretend="false"
;unattend="false"
;unattendSkip="false"
;interactive="false"
;forceFilename="false"
;verbosity="1"
;skipped="false"
;artworkXml="artwork.xml"
;relativePaths="false"
;extensions="*.zip *.uae *.adf"
;addExtensions="*.zst *.rom"
;subdirs="true"

; The following is an example of configs that only affect the 'emulationstation' frontend.
[emulationstation]
;addFolders="false"

; The following is an example of configs that only affect the 'pegasus' frontend.
;[pegasus]
;excludePattern="*[BIOS]*"
;includePattern="Super*"
;artworkXml="artwork.xml"
;emulator=""
;launch=""
;gameListFolder="/home/pi/RetroPie/roms/amiga"
;gameListBackup="false"
;mediaFolder="/home/pi/RetroPie/roms/amiga/media"
;skipped="false"
;brackets="true"
;videos="false"
;symlink="false"
;startAt="filename"
;endAt="filename"
;unattend="false"
;unattendSkip="false"
;forceFilename="false"
;verbosity="1"
;maxLength="10000"

; The following is an example of configs that only affect the 'screenscraper' scraping module.
;[screenscraper]
;userCreds="user:password"
;threads="1"
;minMatch="0"
;maxLength="10000"
;interactive="false"
;cacheResize="false"
;jpgQuality="95"
;cacheCovers="true"
;cacheScreenshots="true"
;cacheWheels="true"
;cacheMarquees="true"
;cacheTextures="true"
;videos="false"
;videoSizeLimit="42"
;videoConvertCommand="ffmpeg -i %i -y -pix_fmt yuv420p -t 00:00:10 -c:v libx264 -crf 23 -c:a aac -b:a 64k -vf scale=640:480:force_original_aspect_ratio=decrease,pad=640:480:(ow-iw)/2:(oh-ih)/2,setsar=1 %o"
;videoConvertExtension="mp4"
;;The following option is only applicable to 'screenscraper'
;videoPreferNormalized="true"

[esgamelist]
cacheRefresh="true"
; https://github.com/RetroPie/RetroPie-Setup/pull/3704
cacheScreenshots="false"

Lolonois

@abj just to understand more of this:

Do you use the frontend EmulationStation during scraping (if you don't use the -f flag, EmulationStation is the default)?
Do you scrape from the command line or from the RetroPie-Setup dialog?
Do you have your previous config.ini still around?

You can still edit the gameListFolder entry in the [main] section of the config.ini to "/home/pi/.emulationstation/gamelists" which should get you the previous experience.

abj

@Lolonois

I runned the Skyscraper from the terminal window (i had exit from ES with F4) and without using the RetroPie-Setup dialog, just to get familiar with the commands for the upcoming v.3.11 .

Then i scrape from the command line with

Skyscraper -p psx -s screenscraper
Skyscraper -p psx

I don't have the previous config.ini file, because it was a fresh install of RetroPie and Skyscraper (Branch 3.10.4).

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug. Thanks for everything!

sleve_mcdichael

@abj said in Skyscraper is back! (v3.11):

Thanks, i modify the entrys and everything is fine. I just reported it just in case that is a bug.

(Skyscraper default has always been to use the ROM folder for gamelists and media. Only the RP-Setup dialog (or CLI option, or manual edit config.ini) redirects it to RetroPie configs area instead.)

abj

@sleve_mcdichael This is the reason then, because the previous times i was using the gui. Thanks!

abj

Hello, it's me again (sorry :P).

I was trying to scrape my neogeocd games that i have into neogeo folder from the terminal window, without the use of RetroPie-Setup Gui.
The command i run was :

Skyscraper -p neogeo -s screenscraper

The command run but i didn't saw any metadata, it was like i didn't have roms in the folder.

I check the peas.json file and it looks that something is missing from the formats section:

},
    "neogeo": {
        "aliases": [
            "arcade",
            "neo geo",
            "neo geo aes",
            "neo geo cd",
            "neo geo mvs",
            "neo-geo",
            "neo-geo cd"
        ],
        "formats": [],
        "scrapers": [
            "arcadedb",
            "openretro",
            "screenscraper"
        ]
    },

sleve_mcdichael

@abj I'm not sure but you could try adding some "formats"? So:

        "formats": [],

...becomes:

        "formats": [
            "*.chd",
            "*.cue",
            "*.fba",
            "*.iso"
        ],

(NeoGeo formats from https://github.com/RetroPie/RetroPie-Setup/blob/master/platforms.cfg#L146, not include "*.zip", "*.7z" as none other systems include these either, so I suppose they are "hardwired" somewhere else.)

abj

@sleve_mcdichael

Thanks, this fix it!

Lolonois

if the option extension is unset (default) then these extensions are automatically used as base *.7z *.ml *.zip by Skyscraper.

Another option is to add the wanted extensions with addExtensions, these get appended to the base extensions.

Eventually the extensions from peas.json / formats are added also to the list of extensions.

Lolonois

Version 3.12.0 is out.

Implementing a batch of community suggestions, all details here

Thank you all and happy scraping!

Edit: Thanks, @mitu for spotting the wrong URL, corrected now. Seems I copied from the wrong browser tab yesterday.

mitu

@Lolonois said in Skyscraper is back! (v3.12):

Implementing a batch of community suggestions, all details here

Hm, here points to localhost, probably should be https://gemba.github.io/skyscraper/CHANGELOG/#version-3120-2024-07-01 ?

Lolonois

3.13.0 is out.

Mainly a maintenance release, removed flags and options marked deprecated since version 3.7.0 (almost three years ago).
You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.
Added the platform Vircon32 (@Vircon32 ).
Many smaller fixes and improvements.

https://github.com/Gemba/skyscraper/releases/tag/3.13.0

Vircon32

@Lolonois that's great! Thank you for the addition.

Good timing too. Recently several Vircon32 games have been added to some scraping sites, like Screenscaper and IGDB.

Lolonois

Just released 3.14 with a bunch of changes:

You may benefit from the three new configuration options innerBracketsReplace, keepDiscInfo and ignoreYearInFilename) or if you use Skyscraper also on a non-Retropie Setup from the XDG Base Directory support.

https://gemba.github.io/skyscraper/CHANGELOG/#version-3140-2024-12-08

For 2025 I plan to implement fanart and bezel support and maybe also frontend support for Batocera Emulationstation.

If you have features in mind which would benefit the RetroPie community feel free to discuss them here.

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.14):

3.13.0 is out.

You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.

This is a great and welcome addition -- ideally, this will allow official upstream changes to co-exist with custom user mods, maintenance-free. Right now though, function copyDefaultConfig in the RP module still puts the new upstream version to peas.json.rp-dist and so update is no applied without manual copy this file to overwrite existing peas.json.

Eventually, will probably want to just overwrite with a regular cp so that changes are applied automatically, but, this could clobber any custom config for users not yet migrated to peas_local.json. Maybe first can use some if diff (looks like RP already have a diffFiles function for this) and ~~if so~~ (I think actually diffFiles check if they are the same, so if not), then save a peas.json.old. This would only give users one "update" worth of grace period to retrieve their edits before the .old version gets overwrite as well, but is still better than no grace.

Something along the lines of:

local json="$scraper_conf_dir/peas.json"
local pristine="$md_inst/.pristine_configs/peas.json"

if [[ -f "$json" ]] && ! diffFiles "$json" "$pristine"; then
    cp -f "$json" "$json.old"
fi
cp -f "$pristine" "$json"

...

Also, I do still have to put some custom entries into platforms_idmap.csv to scrape Screenscraper for the US version genesis and tg16 systems that I have added to peas_local. Can that one have the "local version" file option, as well? (And then it would have the same change to backup & overwrite in the RP module.)

Lolonois

Always welcomed to get feedback on how the software is used. :)

I did not consider a _local file for the CSV file, as I assumed it would be easy to maintain as it would be a single line compare. But you are right, it still has to be maintained. I will take the platforms_idmap_local.csv on the 3.15 release with a guide on how to separate the local additions from the distributed/"upstream" configs for the platform info.

However, I would not overwrite the existing peas/platform files automagically (even after a "grace" update as you noted) unless a user has opt-ed in for that (e.g. with a empty file like .platformcfg_overwrite_ok) and thus confirms that he/she saved additions in a _local file (or didn't change anything in the files peas.json / platforms_idmap.csv at all). On juvenile installations the .platformcfg_overwrite_ok would be created right away.

sleve_mcdichael

Skyscraper not working after 3.15 update:

Duplicate installation of Skyscraper found:
/usr/local/Skyscraper and
/opt/retropie/supplementary/skyscraper/Skyscraper
Please remove one or the other to avoid confusion.

But there is nothing at /usr/local/Skyscraper -- there is one in /usr/local/bin/ but it's the same symlink that's has always been there. (And even with symlink removed, error is still the same.)

Lolonois

@sleve_mcdichael Thanks, maybe I had too much cookies as I implemented it early in Jan. Version 3.15.1 is just out with the fix.

As you are on it, don't try the local platform config yet (see above), We came across a better idea as I proposed the change there.

Lolonois

Now I officially announce the release of Skyscraper 3.15. (3.15.3)

As always find the changes here

Update the RetroPie setup before you update Skyscraper to get a new helper script deployed (which you may use when you get the shown warning below).

Important: In some cases you may get a warning when starting Skyscraper

Looks like '/home/pi/.skyscraper/peas.json' has local changes. Please transfer local changes to another file to mute this warning. See topic 'Transferring Local Platform Changes' in the documentation for guidance.

Why do you get this warning?
This happens when you made changes to the peas.json or platforms_idmap.csv in the past, which was intended to be that way.
See the documentation here on how to remediate the situation.

A later release will change the current behaviour of the RetroPie scriptmodule of Skyscraper and will no longer put updated peas.json as peas.json.rp-dist (same for platforms_idmap.csv). Instead it will always replace your existing ~/.skyscraper/peas.json / platforms_idmap.csv with the upstream version. Thus, do follow the guide, when you get the warning.

There will be an extra announcement when a new Skyscraper version will change the copy behaviour. But as there is no ETA for 3.16 we are talking at least about months.

Any questions let me know.