Skyscraper is back! (v3.13)

Lolonois

Skyscraper v3.8.1 is out.

Contains a couple of updates and fixes (thank you @windg and @sleve_mcdichael). If you use Mobygames for scraping some games, you may want to run a script to update your db.xml files. [1].
Added two features: One for import of rating values and one for OpenRetro scraping the ratings.
Details below:

[1] https://github.com/Gemba/skyscraper/blob/dev_reloaded/README_Skyscraper_reloaded.md#2023-09-23-version-3812309

windg

@Lolonois
Thanks for the new version, the error message gone away.
The purge for chosen platform is working fine, the purge all platform have a small issue i think.
When i choose it, the following message appear with preselect the No option.

Do you wish to continue (y/N)? User chose not to continue, cancelling purge...

By the way, i am using the RetroPie Bullseye weekly image, Thanks again!

(I hope the Devs to replace the older version of skyscraper with this! )

Lolonois

Skyscraper v3.9.0 is out.

For all features and fixes ot this release see release notes.

If you used/changed either platforms.json or screenscraper.json consider reviewing the information in the documentation. If the last version you used was v3.7.7 from muldjord you don't need to take any action.

This Skyscraper fork is now part of RetroPie, see accompanied scriptmodule there, Please wait until the PR is merged in RetroPie-Setup before updating: https://github.com/RetroPie/RetroPie-Setup/pull/3767

beudbeud

Hi @Lolonois, in the github repo i can't open issue, so i asking you here. I wanna know if you can add possibility to use relatif path for roms and media like that

<game id="40045" source="ScreenScraper.fr">
<path>./arabfgt.zip</path>
<name>Arabian Fight (World)</name>
<desc>Arabian Fight est un beat-em-up à défilement latéral. Combattez des vagues d'ennemis avant d'affronter un boss qui doit être vaincu pour passer au niveau suivant. Pendant que vous jouez, vous pouvez ouvrir des coffres et d'autres objets pour obtenir de la nourriture en guise d'énergie ou une lampe magique qui vous permet d'utiliser un sort magique pour tuer tous les ennemis à l'écran. Un effet que le jeu utilise pour ses effets visuels est l'utilisation de la perspective. Tous les personnages à l'écran entrent et sortent de l'écran en devenant plus petits ou plus grands.</desc>
<rating>0.4</rating>
<releasedate>19910101T000000</releasedate>
<developer>SEGA</developer>
<publisher>SEGA</publisher>
<genre>Combat-Combat / 2.5D-Beat'em All</genre>
<players>1-4</players>
<hash/>
<image>./media/images/arabfgt.png</image>
<video>./media/videos/arabfgt.mp4</video>
<genreid>262</genreid>
</game>

Thanks

Lolonois

Without more context I can only advise to use the relativePaths option. It works for all supported frontends, except for attractmode (although the documentation states otherwise, atm.).

https://github.com/Gemba/skyscraper/blob/dev_reloaded/docs/CONFIGINI.md#relativepathsfalse

sleve_mcdichael

In the latest updates, I've been seeing some WARN: messages in the console that didn't use to be there:

 WARN: libpng warning: iCCP: known incorrect sRGB profile WARN: libpng warning: iCCP: known incorrect sRGB profile#1/1 (T1) Pass 1 ---- Game 'Super Mario World (USA)' found! :) ----

Request timed out, server might be busy / overloaded...
 WARN: QIODevice::read (QNetworkReplyHttpImpl): device not openRequest timed out, server might be busy / overloaded...
 WARN: QIODevice::read (QNetworkReplyHttpImpl): device not open#1/1 (T1) Pass 1 ---- Game 'Super Mario World 2 - Yoshi's Island (USA) (Rev 1)' found! :) ----

They are displayed like that, without trailing newlines. The libpng warning is only on certain roms so I suppose it may have gone past me unnoticed before, but I certainly never used to see that additional device not open text on the Request timed out message before.

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.9.0):

iCCP: known incorrect sRGB profile

Root cause is the scraping source provides a non-compliant PNG, it is a warning which has no knock-on effect as far as i can tell. It has been there all the times but in previous Skyscraper releases all Warnings have been muted. I reverted this as "muted warnings" may result in head-scratching debugging. But this one is harmless. I might resolve the warning properly in some of the next releases. For now it is muted (see pre-release below).

@sleve_mcdichael said in Skyscraper is back! (v3.9.0):

device not open

Is also a warning reaching the surface now. If you apply --verbosity 1 or higher Skyscraper should provide some more information why the timeout was reached.

I have provided a pre 3.9.2 version (which also contains many "under-the-hood"-changes [1]. You may give it a try if you change the line in the scriptmodules/supplementary/skyscraper.sh from

rp_module_repo="git https://github.com/Gemba/skyscraper :_get_branch_skyscraper"

to

rp_module_repo="git https://github.com/Gemba/skyscraper dev_reloaded"

As always, thanks for your feedback.

[1] https://gemba.github.io/skyscraper/CHANGELOG/

beudbeud

Do you think it's possible to add a way too keep favorites listing.

Something like keep json list with all favorite, and generate gamelist.xml you add <favorite>true</favorite>

sleve_mcdichael

@beudbeud doesn't it already?

https://github.com/Gemba/skyscraper/blob/dev_reloaded/docs/FRONTENDS.md#metadata-preservation

Skyscraper will preserve the following metadata when re-generating a game list for EmulationStation: favorite, hidden, kidgame, lastplayed, playcount, sortname.

beudbeud

@sleve_mcdichael thanks nice, I always build from another computer and after copy it. So I will see for change my way to generate gamelist

Lolonois

Version 3.9.2 is out.

Two enhancements:

A configuration option to tidy some of the most common description quirks.
If you use a definition.dat in XML style for custom import data you don't have to fit the definitions.dat whitespacing outside the XML elements as your import file is parsed as DOM/XML.

Full release notes here.

sleve_mcdichael

On my build, I keep a separate genesis system, not symlinked with megadrive. In earlier versions of Skyscraper (certainly prior to 3.8.1.2309, and I think prior to 3.9.0), using platform -p genesis worked, out the box with no modification.

Today (in 3.9.2) I tried Skyscraper -p genesis -s screenscraper and got:

Please set a valid platform with '-p <PLATFORM>'
Check '--help' for a list of supported platforms. Qutting.

I added a genesis section to peas.json like:

"genesis": {
    "aliases": [
        "megadrive",
        "mega drive",
        "sega mega drive",
        "sega mega drive/genesis"
    ],
    "formats": [
        "*.bin",
        "*.gen",
        "*.md",
        "*.sg",
        "*.smd"
    ],
    "scrapers": [
        "openretro",
        "screenscraper"
    ]
},

Now Skyscraper -p genesis works, but when I include -s screenscraper I am met with:

Platform not supported by ScreenScraper or it hasn't yet been included in Skyscraper for this module...

I understand screenscraper_platforms.json is a reference file not meant to be edited, plus already has this section anyway:

"1": {
    "nom_eu": "Megadrive",
    "nom_us": "Genesis",
    "nom_recalbox": "megadrive",
    "nom_retropie": "genesis,megadrive",
    "nom_launchbox": "Sega Genesis",
    "nom_hyperspin": "Sega Genesis",
    "noms_commun": "Sega Megadrive,Sega Genesis,Megadrive,Genesis,Super Aladdin Boy"
},

Not sure what else is needed to make -p genesis -s screenscraper work like before?

Lolonois

@sleve_mcdichael this can be solved configuration-wise. You are almost there: Last step should be to copy the line megadrive,1,16,36 from platforms_idmap.csv and add it to that file but change the platform to Genesis, thus it reads genesis,1,16,36.

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.9.2):

platforms_idmap.csv

Thanks. Apparently I had already known this at one point since I've already got a tg16 line added, but I guess I've forgotten it in the meantime.

Cheers, thanks!

windg

I remember in the past, when i was scrapping psx roms (bin/cue), to saw the metadata, i have to enter in the game folder in the ES menu. Did this behavior change in the recent releases ? Now i can see them without enter in the folder. It's a very welcome change!

Edit: I just saw the changelog.Thank you very much to add this feature!!

windg

I am reporting something that maybe is an issue of the current setup (v3.9.2).

With the multidisk psx games, if the .m3u text contents are :

Game_A (Disk1).cue
Game_A (Disk2).cue

Then, metadata, aren't visible without entering the folder. But they are visible if the .m3u file is :

Game_A (Disk1).CD1
Game_A (Disk2).CD2

Lolonois

@windg how is are directory/files structured?

I have tried with this layout inside ~/RetroPie/roms/psx:

$ find Alone\ in\ the\ Dark\:\ The\ New\ Nightmare*
Alone in the Dark: The New Nightmare/
Alone in the Dark: The New Nightmare/Game (Disk1).CD1
Alone in the Dark: The New Nightmare/Game (Disk1).CD2
Alone in the Dark: The New Nightmare.m3u

and this m3u content (also with cue extension for the disks):

$ cat Alone\ in\ the\ Dark\:\ The\ New\ Nightmare.m3u 
Alone in the Dark: The New Nightmare/Game (Disk1).CD1
Alone in the Dark: The New Nightmare/Game (Disk2).CD2

But in all four cases (Skyscraper 3.9.2 and current dev version and disc extension either cue or cd1/2 the metadata showed up instantly when in System psx (no need to navigate to subfolder).

I ran these commands:

Skyscraper -p psx --refresh -s screenscraper '/home/pi/RetroPie/roms/psx/Alone in the Dark: The New Nightmare.m3u'
Skyscraper -p psx

Could you verify on your end with this setup outlined above?

If it does not remediate the situation:
What is the setting of subdirs in your config.ini (in [main] as in [psx])?
which frontend are your using (see config.ini)?
what is the section "psx" showing in your peas.json?
I am running the emulationstation-dev version, what version do you use?
Can you share the part of your gamelist.xml for that section of <game> in question?

windg

@Lolonois

I am using Skyscraper v3.9.2 installed from binary.
Emulationstation frontend
Folder Structure :

Wild Arms 2 (USA)/
Wild Arms 2 (USA) (Disc 1).bin
Wild Arms 2 (USA) (Disc 1).cue
Wild Arms 2 (USA) (Disc 2).bin
Wild Arms 2 (USA) (Disc 2).cue
Wild Arms 2 (USA).m3u

Wild Arms 2 (USA).m3u :

Wild Arms 2 (USA) (Disc 1).cue
Wild Arms 2 (USA) (Disc 2).cue

config.ini file
peas.json :

},
    "psx": {
        "aliases": [
            "playstation",
            "sony playstation"
        ],
        "formats": [
            "*.cbn",
            "*.chd",
            "*.cue",
            "*.img",
            "*.iso",
            "*.m3u",
            "*.mdf",
            "*.pbp",
            "*.toc",
            "*.z",
            "*.znx"
        ],
        "scrapers": [
            "screenscraper",
            "thegamesdb"
        ]
    },

artwork.xml :

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" resource="cover"/>
  <output type="marquee" resource="wheel"/>
</artwork>

gamelist.xml

Indeed, if i use .CDx extenstion instead of .cue, it will show the metadata without move into subfolder. But i haven't test multidisk games for other emulators that don't support the .CDx or if the user is using .chd files. Thanks to look into this !

Lolonois

@windg Thanks for the info. To narrow down the context, a few more questions: Which ES version are you running? What is the setting of Parse Gamelists Only in the ES UI Settings menu or in your ~/.emulationstation/es_settings.cfg ?

If I understand correctly, you have highlighted the folder in ES which is shown in your game list of the system PSX and then there is no metadata shown.

However, when you navigate into the folder you get metadata presented
a) regardless of the extension of the files in that folder? -or-
b) only when extension is CD1 / CD2?

Independently of this: Do you get metadata presented when you highlight the entry in ES which is the m3u file on your storage/filesystem?

windg

The Emulationstation version is v2.11.2RP, RetroPie Buster v4.8.6 .

Gamelist view is the default 'Automatic' . I am using the default theme .

"ParseGamelistOnly" value="false"

es_settings.cfg file

If the .m3u have .cue extentions, then :

No Metadata shown when i highlighted the folder in ES.
Metadata shown when navigate into the folder, for the .m3u, for disk1 (bin/cue) and for the disk2 (bin/cue), 3 times .

If the .m3u have .CDx extentions, then :

-Metadata shown when i highlighted the folder in ES.
-No metadata shown when i navigate into the folder. (only one entry is visible here(.m3u), not three like before)

gamelist.xml file with the both games that i test. The first is the Countdown Vampires, a two disks game - m3u file with .CDx extentions. And the other is Wild Arms 2, a two disk game - m3u file with .cue extentions.

To be more clear, this is happening only with the multi-disk games. Single disk shown correctly the metadata when i highlighted the folder in ES. Thanks again !!

(As a workaround, i am hiding the game folder and i have put the .m3u file in the psx rom folder .)