Skyscraper is back! (v3.12)

sleve_mcdichael

In the latest updates, I've been seeing some WARN: messages in the console that didn't use to be there:

 WARN: libpng warning: iCCP: known incorrect sRGB profile WARN: libpng warning: iCCP: known incorrect sRGB profile#1/1 (T1) Pass 1 ---- Game 'Super Mario World (USA)' found! :) ----

Request timed out, server might be busy / overloaded...
 WARN: QIODevice::read (QNetworkReplyHttpImpl): device not openRequest timed out, server might be busy / overloaded...
 WARN: QIODevice::read (QNetworkReplyHttpImpl): device not open#1/1 (T1) Pass 1 ---- Game 'Super Mario World 2 - Yoshi's Island (USA) (Rev 1)' found! :) ----

They are displayed like that, without trailing newlines. The libpng warning is only on certain roms so I suppose it may have gone past me unnoticed before, but I certainly never used to see that additional device not open text on the Request timed out message before.

Lolonois

@sleve_mcdichael said in Skyscraper is back! (v3.9.0):

iCCP: known incorrect sRGB profile

Root cause is the scraping source provides a non-compliant PNG, it is a warning which has no knock-on effect as far as i can tell. It has been there all the times but in previous Skyscraper releases all Warnings have been muted. I reverted this as "muted warnings" may result in head-scratching debugging. But this one is harmless. I might resolve the warning properly in some of the next releases. For now it is muted (see pre-release below).

@sleve_mcdichael said in Skyscraper is back! (v3.9.0):

device not open

Is also a warning reaching the surface now. If you apply --verbosity 1 or higher Skyscraper should provide some more information why the timeout was reached.

I have provided a pre 3.9.2 version (which also contains many "under-the-hood"-changes [1]. You may give it a try if you change the line in the scriptmodules/supplementary/skyscraper.sh from

rp_module_repo="git https://github.com/Gemba/skyscraper :_get_branch_skyscraper"

to

rp_module_repo="git https://github.com/Gemba/skyscraper dev_reloaded"

As always, thanks for your feedback.

[1] https://gemba.github.io/skyscraper/CHANGELOG/

beudbeud

Do you think it's possible to add a way too keep favorites listing.

Something like keep json list with all favorite, and generate gamelist.xml you add <favorite>true</favorite>

sleve_mcdichael

@beudbeud doesn't it already?

https://github.com/Gemba/skyscraper/blob/dev_reloaded/docs/FRONTENDS.md#metadata-preservation

Skyscraper will preserve the following metadata when re-generating a game list for EmulationStation: favorite, hidden, kidgame, lastplayed, playcount, sortname.

beudbeud

@sleve_mcdichael thanks nice, I always build from another computer and after copy it. So I will see for change my way to generate gamelist

Lolonois

Version 3.9.2 is out.

Two enhancements:

A configuration option to tidy some of the most common description quirks.
If you use a definition.dat in XML style for custom import data you don't have to fit the definitions.dat whitespacing outside the XML elements as your import file is parsed as DOM/XML.

Full release notes here.

sleve_mcdichael

On my build, I keep a separate genesis system, not symlinked with megadrive. In earlier versions of Skyscraper (certainly prior to 3.8.1.2309, and I think prior to 3.9.0), using platform -p genesis worked, out the box with no modification.

Today (in 3.9.2) I tried Skyscraper -p genesis -s screenscraper and got:

Please set a valid platform with '-p <PLATFORM>'
Check '--help' for a list of supported platforms. Qutting.

I added a genesis section to peas.json like:

"genesis": {
    "aliases": [
        "megadrive",
        "mega drive",
        "sega mega drive",
        "sega mega drive/genesis"
    ],
    "formats": [
        "*.bin",
        "*.gen",
        "*.md",
        "*.sg",
        "*.smd"
    ],
    "scrapers": [
        "openretro",
        "screenscraper"
    ]
},

Now Skyscraper -p genesis works, but when I include -s screenscraper I am met with:

Platform not supported by ScreenScraper or it hasn't yet been included in Skyscraper for this module...

I understand screenscraper_platforms.json is a reference file not meant to be edited, plus already has this section anyway:

"1": {
    "nom_eu": "Megadrive",
    "nom_us": "Genesis",
    "nom_recalbox": "megadrive",
    "nom_retropie": "genesis,megadrive",
    "nom_launchbox": "Sega Genesis",
    "nom_hyperspin": "Sega Genesis",
    "noms_commun": "Sega Megadrive,Sega Genesis,Megadrive,Genesis,Super Aladdin Boy"
},

Not sure what else is needed to make -p genesis -s screenscraper work like before?

Lolonois

@sleve_mcdichael this can be solved configuration-wise. You are almost there: Last step should be to copy the line megadrive,1,16,36 from platforms_idmap.csv and add it to that file but change the platform to Genesis, thus it reads genesis,1,16,36.

sleve_mcdichael

@Lolonois said in Skyscraper is back! (v3.9.2):

platforms_idmap.csv

Thanks. Apparently I had already known this at one point since I've already got a tg16 line added, but I guess I've forgotten it in the meantime.

Cheers, thanks!

windg

I remember in the past, when i was scrapping psx roms (bin/cue), to saw the metadata, i have to enter in the game folder in the ES menu. Did this behavior change in the recent releases ? Now i can see them without enter in the folder. It's a very welcome change!

Edit: I just saw the changelog.Thank you very much to add this feature!!

windg

I am reporting something that maybe is an issue of the current setup (v3.9.2).

With the multidisk psx games, if the .m3u text contents are :

Game_A (Disk1).cue
Game_A (Disk2).cue

Then, metadata, aren't visible without entering the folder. But they are visible if the .m3u file is :

Game_A (Disk1).CD1
Game_A (Disk2).CD2

Lolonois

@windg how is are directory/files structured?

I have tried with this layout inside ~/RetroPie/roms/psx:

$ find Alone\ in\ the\ Dark\:\ The\ New\ Nightmare*
Alone in the Dark: The New Nightmare/
Alone in the Dark: The New Nightmare/Game (Disk1).CD1
Alone in the Dark: The New Nightmare/Game (Disk1).CD2
Alone in the Dark: The New Nightmare.m3u

and this m3u content (also with cue extension for the disks):

$ cat Alone\ in\ the\ Dark\:\ The\ New\ Nightmare.m3u 
Alone in the Dark: The New Nightmare/Game (Disk1).CD1
Alone in the Dark: The New Nightmare/Game (Disk2).CD2

But in all four cases (Skyscraper 3.9.2 and current dev version and disc extension either cue or cd1/2 the metadata showed up instantly when in System psx (no need to navigate to subfolder).

I ran these commands:

Skyscraper -p psx --refresh -s screenscraper '/home/pi/RetroPie/roms/psx/Alone in the Dark: The New Nightmare.m3u'
Skyscraper -p psx

Could you verify on your end with this setup outlined above?

If it does not remediate the situation:
What is the setting of subdirs in your config.ini (in [main] as in [psx])?
which frontend are your using (see config.ini)?
what is the section "psx" showing in your peas.json?
I am running the emulationstation-dev version, what version do you use?
Can you share the part of your gamelist.xml for that section of <game> in question?

windg

@Lolonois

I am using Skyscraper v3.9.2 installed from binary.
Emulationstation frontend
Folder Structure :

Wild Arms 2 (USA)/
Wild Arms 2 (USA) (Disc 1).bin
Wild Arms 2 (USA) (Disc 1).cue
Wild Arms 2 (USA) (Disc 2).bin
Wild Arms 2 (USA) (Disc 2).cue
Wild Arms 2 (USA).m3u

Wild Arms 2 (USA).m3u :

Wild Arms 2 (USA) (Disc 1).cue
Wild Arms 2 (USA) (Disc 2).cue

config.ini file
peas.json :

},
    "psx": {
        "aliases": [
            "playstation",
            "sony playstation"
        ],
        "formats": [
            "*.cbn",
            "*.chd",
            "*.cue",
            "*.img",
            "*.iso",
            "*.m3u",
            "*.mdf",
            "*.pbp",
            "*.toc",
            "*.z",
            "*.znx"
        ],
        "scrapers": [
            "screenscraper",
            "thegamesdb"
        ]
    },

artwork.xml :

<?xml version="1.0" encoding="UTF-8"?>
<artwork>
  <output type="screenshot" resource="cover"/>
  <output type="marquee" resource="wheel"/>
</artwork>

gamelist.xml

Indeed, if i use .CDx extenstion instead of .cue, it will show the metadata without move into subfolder. But i haven't test multidisk games for other emulators that don't support the .CDx or if the user is using .chd files. Thanks to look into this !

Lolonois

@windg Thanks for the info. To narrow down the context, a few more questions: Which ES version are you running? What is the setting of Parse Gamelists Only in the ES UI Settings menu or in your ~/.emulationstation/es_settings.cfg ?

If I understand correctly, you have highlighted the folder in ES which is shown in your game list of the system PSX and then there is no metadata shown.

However, when you navigate into the folder you get metadata presented
a) regardless of the extension of the files in that folder? -or-
b) only when extension is CD1 / CD2?

Independently of this: Do you get metadata presented when you highlight the entry in ES which is the m3u file on your storage/filesystem?

windg

The Emulationstation version is v2.11.2RP, RetroPie Buster v4.8.6 .

Gamelist view is the default 'Automatic' . I am using the default theme .

"ParseGamelistOnly" value="false"

es_settings.cfg file

If the .m3u have .cue extentions, then :

No Metadata shown when i highlighted the folder in ES.
Metadata shown when navigate into the folder, for the .m3u, for disk1 (bin/cue) and for the disk2 (bin/cue), 3 times .

If the .m3u have .CDx extentions, then :

-Metadata shown when i highlighted the folder in ES.
-No metadata shown when i navigate into the folder. (only one entry is visible here(.m3u), not three like before)

gamelist.xml file with the both games that i test. The first is the Countdown Vampires, a two disks game - m3u file with .CDx extentions. And the other is Wild Arms 2, a two disk game - m3u file with .cue extentions.

To be more clear, this is happening only with the multi-disk games. Single disk shown correctly the metadata when i highlighted the folder in ES. Thanks again !!

(As a workaround, i am hiding the game folder and i have put the .m3u file in the psx rom folder .)

Lolonois

@windg Ah, now I can reproduce it.

The code line where the usage of "folder metadata" is decided is this one. I have inherited the logic from Lars, it is around since quite a while. Even when Lars was the maintainer.

While it is comprehensible that you put the m3u file into the game folder it is also unusual.

It is coincidence that Countdown Vampires shows the data at folder level.
The fact that inside the folder only one file is shown is most likely due to that the extension CD1/2 is not added in es_systems.cfg.
For the same reason (read: CD1/CD2 not in peas.json) the metadata is shown on folder level. The number of known files counted by Skyscraper is 1 in that folder (the file with the m3u extension).

The Wild Arms 2 does not show on folder level as the logic in Skyscraper only sets "folder metadata" when exactly one file with the extensions of peas.json is found in this folder. As cue and m3u are valid extensions it sums up to 3 files, thus no "folder metadata".

I am not sold if I should add another exception (see his comment more above at the code line) for m3u files and having the m3u file inside the game folder.
If there are more users demanding it I might consider it.

Additionally to your workaround I would also set subdirs=false in the config.ini (for [psx]). And set "Parse Gamelists Only" to true.

windg

I will use your suggetion for the multi-disk games. Thank you very much for your time and the detail expanation!

Lolonois

@windg Nvm. Thanks for tabling and challenging it.

One more thing (without the workaround): You still enter the folder of Countdown Vampires for example and then launch the m3u file or do you somehow manage to launch when you focus the folder Countdown Vampires?

windg

@Lolonois Yes, i have to enter into the folder, hit the m3u file to launch the game. I can avoid this if i throw the content of the game folder into the psx folder.

windg

New ppsspp v1.17.0 it's adding support for .chd files.
The current peas.json is missing this format. I add it, tested it and it's working.

},
    "psp": {
        "aliases": [
            "playstation portable",
            "sony playstation portable"
        ],
        "formats": [
            "*.cso",
            "*.iso",
            "*.pbp",
            "*.chd"
        ],
        "scrapers": [
            "screenscraper"
        ]
    }