Skyscraper is back! (v3.17)

abj

@sleve_mcdichael This is the reason then, because the previous times i was using the gui. Thanks!

abj

Hello, it's me again (sorry :P).

I was trying to scrape my neogeocd games that i have into neogeo folder from the terminal window, without the use of RetroPie-Setup Gui.
The command i run was :

Skyscraper -p neogeo -s screenscraper

The command run but i didn't saw any metadata, it was like i didn't have roms in the folder.

I check the peas.json file and it looks that something is missing from the formats section:

},
    "neogeo": {
        "aliases": [
            "arcade",
            "neo geo",
            "neo geo aes",
            "neo geo cd",
            "neo geo mvs",
            "neo-geo",
            "neo-geo cd"
        ],
        "formats": [],
        "scrapers": [
            "arcadedb",
            "openretro",
            "screenscraper"
        ]
    },

sleve_mcdichael

@abj I'm not sure but you could try adding some "formats"? So:

        "formats": [],

...becomes:

        "formats": [
            "*.chd",
            "*.cue",
            "*.fba",
            "*.iso"
        ],

(NeoGeo formats from https://github.com/RetroPie/RetroPie-Setup/blob/master/platforms.cfg#L146, not include "*.zip", "*.7z" as none other systems include these either, so I suppose they are "hardwired" somewhere else.)

abj

@sleve_mcdichael

Thanks, this fix it!

Lolonois

if the option extension is unset (default) then these extensions are automatically used as base *.7z *.ml *.zip by Skyscraper.

Another option is to add the wanted extensions with addExtensions, these get appended to the base extensions.

Eventually the extensions from peas.json / formats are added also to the list of extensions.

Lolonois

Version 3.12.0 is out.

Implementing a batch of community suggestions, all details here

Thank you all and happy scraping!

Edit: Thanks, @mitu for spotting the wrong URL, corrected now. Seems I copied from the wrong browser tab yesterday.

mitu

@Lolonois said in Skyscraper is back! (v3.12):

Implementing a batch of community suggestions, all details here

Hm, here points to localhost, probably should be https://gemba.github.io/skyscraper/CHANGELOG/#version-3120-2024-07-01 ?

Lolonois

3.13.0 is out.

Mainly a maintenance release, removed flags and options marked deprecated since version 3.7.0 (almost three years ago).
You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.
Added the platform Vircon32 (@Vircon32 ).
Many smaller fixes and improvements.

https://github.com/Gemba/skyscraper/releases/tag/3.13.0

Vircon32

@Lolonois that's great! Thank you for the addition.

Good timing too. Recently several Vircon32 games have been added to some scraping sites, like Screenscaper and IGDB.

Lolonois

Just released 3.14 with a bunch of changes:

You may benefit from the three new configuration options innerBracketsReplace, keepDiscInfo and ignoreYearInFilename) or if you use Skyscraper also on a non-Retropie Setup from the XDG Base Directory support.

https://gemba.github.io/skyscraper/CHANGELOG/#version-3140-2024-12-08

For 2025 I plan to implement fanart and bezel support and maybe also frontend support for Batocera Emulationstation.

If you have features in mind which would benefit the RetroPie community feel free to discuss them here.

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.14):

3.13.0 is out.

You may now also maintain local changes to the supported platforms in a separate file called peas_local.json. Any platform entry from peas_local.json with the same platform name as in peas.json takes precedence.

This is a great and welcome addition -- ideally, this will allow official upstream changes to co-exist with custom user mods, maintenance-free. Right now though, function copyDefaultConfig in the RP module still puts the new upstream version to peas.json.rp-dist and so update is no applied without manual copy this file to overwrite existing peas.json.

Eventually, will probably want to just overwrite with a regular cp so that changes are applied automatically, but, this could clobber any custom config for users not yet migrated to peas_local.json. Maybe first can use some if diff (looks like RP already have a diffFiles function for this) and ~~if so~~ (I think actually diffFiles check if they are the same, so if not), then save a peas.json.old. This would only give users one "update" worth of grace period to retrieve their edits before the .old version gets overwrite as well, but is still better than no grace.

Something along the lines of:

local json="$scraper_conf_dir/peas.json"
local pristine="$md_inst/.pristine_configs/peas.json"

if [[ -f "$json" ]] && ! diffFiles "$json" "$pristine"; then
    cp -f "$json" "$json.old"
fi
cp -f "$pristine" "$json"

...

Also, I do still have to put some custom entries into platforms_idmap.csv to scrape Screenscraper for the US version genesis and tg16 systems that I have added to peas_local. Can that one have the "local version" file option, as well? (And then it would have the same change to backup & overwrite in the RP module.)

Lolonois

Always welcomed to get feedback on how the software is used. :)

I did not consider a _local file for the CSV file, as I assumed it would be easy to maintain as it would be a single line compare. But you are right, it still has to be maintained. I will take the platforms_idmap_local.csv on the 3.15 release with a guide on how to separate the local additions from the distributed/"upstream" configs for the platform info.

However, I would not overwrite the existing peas/platform files automagically (even after a "grace" update as you noted) unless a user has opt-ed in for that (e.g. with a empty file like .platformcfg_overwrite_ok) and thus confirms that he/she saved additions in a _local file (or didn't change anything in the files peas.json / platforms_idmap.csv at all). On juvenile installations the .platformcfg_overwrite_ok would be created right away.

sleve_mcdichael

Skyscraper not working after 3.15 update:

Duplicate installation of Skyscraper found:
/usr/local/Skyscraper and
/opt/retropie/supplementary/skyscraper/Skyscraper
Please remove one or the other to avoid confusion.

But there is nothing at /usr/local/Skyscraper -- there is one in /usr/local/bin/ but it's the same symlink that's has always been there. (And even with symlink removed, error is still the same.)

Lolonois

@sleve_mcdichael Thanks, maybe I had too much cookies as I implemented it early in Jan. Version 3.15.1 is just out with the fix.

As you are on it, don't try the local platform config yet (see above), We came across a better idea as I proposed the change there.

Lolonois

Now I officially announce the release of Skyscraper 3.15. (3.15.3)

As always find the changes here

Update the RetroPie setup before you update Skyscraper to get a new helper script deployed (which you may use when you get the shown warning below).

Important: In some cases you may get a warning when starting Skyscraper

Looks like '/home/pi/.skyscraper/peas.json' has local changes. Please transfer local changes to another file to mute this warning. See topic 'Transferring Local Platform Changes' in the documentation for guidance.

Why do you get this warning?
This happens when you made changes to the peas.json or platforms_idmap.csv in the past, which was intended to be that way.
See the documentation here on how to remediate the situation.

A later release will change the current behaviour of the RetroPie scriptmodule of Skyscraper and will no longer put updated peas.json as peas.json.rp-dist (same for platforms_idmap.csv). Instead it will always replace your existing ~/.skyscraper/peas.json / platforms_idmap.csv with the upstream version. Thus, do follow the guide, when you get the warning.

There will be an extra announcement when a new Skyscraper version will change the copy behaviour. But as there is no ETA for 3.16 we are talking at least about months.

Any questions let me know.

Lolonois

@Lolonois said in Skyscraper is back! (v3.15):

no ETA for 3.16 we are talking at least about months.

Well, at least more than one month. :D But anyhow. 3.16 has just arrived.

With this release Skyscraper has gotten its 10th scraping module: This time for GameBase DB. Any GameBase DB is usually provided as Microsoft Access DB but Skyscraper provides a script to convert it to a more compliant format (Sqlite), which also saves you to install some MS blob DB drivers.

In general the other scrapers also did get some attention:

IGDB now scrapes Covers and Screenshots and supports querying for a id=... in the --query parameter,
ArcadeDB does scrape HD-video for any title that provides it,
various things put straight in Mobygames scraper,
Import scraper with XML format now allows the closing XML element to be on a different line than the opening XML element (comes in handy for longer game descriptions).

The scraper documentation got also some (hopefully useful) overview tables.

See the details of changes with all links to the added and updated documentation.

Please update the RetroPie-Setup script before updating Skyscraper to have everything in place.

Happy scraping!

Lolonois

Skyscraper 3.17 just has been released including these highlights

ZX-Spectrum scraping works again
Relative paths can be used for the majority of configuration options and command line options
Artwork options enhancements
...

See all updates here