Skyscraper: How to get gamelist.xml pointing to .m3u playlist?

yserra

I use RetroPie 4.7.1 on a Raspberry Pi 4 with 4GB RAM and a USB harddrive.

I would like to get only one scrapped entry in EmulationStation per multi-disc games.

To start, I converted all my multi-disc Amstrad CPC (lr-caprice32) games this way:

for each multi-disc game (the ones with "(Disk...)" in the name), moved every set of files into a sub-folder with the same basename,
created in this folder a .m3u file with the list of the files,
scraped everything with Skyscraper 3.6.1

I did this because I read in the Skyscraper 3.0.0 announcement :

"Now converts ES game list entries to <folder> entries if a .cue or .m3u file is found in a subfolder. This causes the information and artwork to be shown when selecting the folder in ES rather than the .cue or .m3u file inside the subfolder. (...) And for users who scrape .cue or .m3u files in subfolders, Skyscraper will now autodetect this and convert the game list entry to a <folder> entry instead of a <game> entry. This means that the game data and artwork will show when selecting the folder in EmulationStation."

-> So, in theory, I find a game thanks to the scraped Information, select it, open a folder, and select the first entry (the .m3u). This would have been ok.

But this is NOT what happened. Skyscraper created no <folder> entry at all. It created an entry for every .dsk files in each subfolders, as it used to do since its first releases... So now, I have some titles in ES that are folders with no image, I have to open them blind and, upon all the files that are in there, the only one I would have to run is the .m3u one, but this is the only one that have no scraping information... It's a bit disappointing.

Here is the simple-script I wrote to setup the .m3u/folder structures (I run it from the /home/pi/RetroPie/roms/amstradcpc folder):

#!/bin/bash

for filename in *.dsk
do
if [[ $filename == *"Disk"* ]]; then
croppedfilename=${filename//\(Disk*\).dsk/}
mkdir -p "$croppedfilename"
mv "$filename" "$croppedfilename/"
echo "$filename" >> "$croppedfilename/$croppedfilename.m3u"
fi
done

And here is the two steps I used to scrap the Amstrad CPC games with Skyscraper:

# get the information
skyscraper -p amstradcpc -s screenscraper -u mylogin:mypassword -i /home/pi/RetroPie/roms/amstradcpc --lang fr,en --flags nomarquees 
# create the gamelist.xml
skyscraper -p amstradcpc -f emulationstation -i /home/pi/RetroPie/roms/amstradcpc -g /home/pi/RetroPie/roms/amstradcpc -o /home/pi/RetroPie/roms/amstradcpc/images --lang fr,en --flags forcefilename,relative,skipped

here is an example for the multi-disc game "5Estrellas". I now have on my system those files:

~/RetroPie/roms/amstradcpc/5Estrellas/5Estrellas.m3u
~/RetroPie/roms/amstradcpc/5Estrellas/5Estrellas(Disk1of2).dsk
~/RetroPie/roms/amstradcpc/5Estrellas/5Estrellas(Disk2of2).dsk

the content of the 5Estrellas.m3u is:

5Estrellas(Disk1of2).dsk
5Estrellas(Disk2of2).dsk

The content of the gamelist.xml is:

<game>
    <path>./5Estrellas/5Estrellas(Disk1of2).dsk</path>
    <name>5Estrellas (Disk1of2)</name>
    <thumbnail />
    <image>./images/screenshots/5Estrellas(Disk1of2).png</image>
...
  <game>
    <path>./5Estrellas/5Estrellas(Disk2of2).dsk</path>
    <name>5Estrellas (Disk2of2)</name>
    <thumbnail />
    <image>./images/screenshots/5Estrellas(Disk2of2).png</image>
...

And there is no <folder> entry nowhere in the gamelist.xml.

Where is my mistake?

I did all this on my Mac, since Skyscraper compiles on a Mac.

mitu

For amstradcpc, the supported extensions list does not include m3u.
You can ask @muldjord here to add it by default or modify the config.ini file to include .m3u for the amstradcpc platform:

[amstradcpc]
extensions="*.cpc *.dsk *.m3u"

and then re-run the scraping step.

yserra

thank you @mitu for your answer.

So I added the lines you mentioned in the config.ini file. Then I ran again the 2 steps of the scraping process (adding the flag "onlymissing" in step #1).

This added the scraped information to the .m3u files. That's better.

But there is still no entry <folder> in the gamelist.xml.

mitu

@yserra said in Skyscraper: How to get gamelist.xml pointing to .m3u playlist?:

But there is still no entry <folder> in the gamelist.xml.

3.0.0 is almost 2 years ago, things might have changed since then. Why do you need a 'folder' entry anyway ?

yserra

@mitu
As i said, my goal would be to get only one entry in EmulationStation for a multi-disc game. And since multi-disc games are only handled by Retroarch when you run a .m3u file, my understanding is that I should have a scraped information for the .m3u file only and hide all the other files.

At first, I wanted to create a script that automatically creates a .m3u file per multi-disc game and renames the extension of the other files to hide them from ES (so the ".dsk" here, for instance renaming them to, let say, ".dsm" or whatever) .

But when I looked for information on this forum (here, here, here, here...) , I realized most people put the multi-disc games in a folder. My understanding was it should be the best practice.

But doing so, I need the scraping information to apply to this folder also, because without this information, I'm getting in ES just a name (pointing to the folder) with no information at all. And I don't know what the game behind that name looks like (that's the purpose of scraping, isn't it?). Now, I have a name with no information, I have to select it, then it shows all the files in this folder and, at last, I can see what this game is about. That's not very elegant.

So, now, if I understand well, when you want to get a .m3u playlist, the main goal would be to hide the other files (as I though at first) and everything concerning a subfolder dedicated to that game would be just out of scope, is that correct?

mitu

Why not rename the extension for the files in the .m3u so they're not displayed in the gamelist ? You have game.m3u which can have any filename:

game-1.disk1
game-2.disk2

Another option would be to place the files in a hidden sub-folder and configure EmulationStation to ignore hidden files (not sure if it's default or not). Place the files in a subfolder .game, then the m3u file would contain

.game/file1.dsk
.game/file2.dsk

muldjord

@yserra Skyscraper will only convert it to a <folder> entry in cases where only one single supported file is contained in the subfolder. The reason being that subfolders are also used by some users simply to alphabetize their collections, in which case it should of course not create a <folder> entry for the subfolder a, b and so on, which potentially has hundreds of supported roms inside each of them.
So you need to only have extensions="*.m3u" in config.ini and that should do it. But it would of course require that you have m3u files even for the single-disk games.

This also requires the m3u files to be located inside the subfolders. You can also simply place the m3u files outside the subfolders, so you have:

game.m3u
|-game folder
| |-disk1.dsk
| |-disk2.dsk

where game.m3u contains:

./game folder/disk1.dsk
./game folder/disk2.dsk

Doing that it won't need to be converted to a <folder> entry, but it will still show the folders and scrape the dsk files inside of them and add them as individual entries.

yserra

@muldjord
Thank you for your answer. And congratulations on your fantastic work on the Skyscraper, the best multi-platform scraping tool, in my opinion.

Actually, I'm not very comfortable with the process of adding a .m3u file to each of my 2500+ individual Amstrad CPC games (or to every game in my other retro-collections).

And, @mitu, I'm not comfortable neither with the idea of adding a non-standard extension to my games, just to have ES showing the .m3u playlists only. I understand that ".cd1", ".cd2", etc. are like a standard now in PSX multi-disc games, but I suspect a home-made ".dsm", ".disk1" or whatever I could invent in my room for Amstrad CPC/MSX/other Libretro would drive me to a lot of issues when I update/scrape/configure something in the future. The same applies to hidden folders: if I want to update the scraping info, I would have to rename the folder before, re-hide the folder after... Even the idea to manage ".disk1" or ".disk2" sounds laborious when you have hundreds of folders, thousands of files...

But, I still think one scrapped entry in ES for a multi-disc game sounds like the most elegant solution.

So, I share here the solution I took:
1- As mentioned before, I create a .m3u playlist for each multi-disc game and put all the files, including the .m3u, inside a folder with the same basename (I do this with the script in my first post)

2- As @mitu told me, I edit the Skyscraper's config.ini to add the .m3u extension at the platform I want to scrap.

3- I scrap everything. So regular files, regular files in subfolders, and .m3u files get an image+description in gamelist.xml.

4- I duplicate the information related to the .m3u files in the gamelist.xml to create new <folder> sections with the same scraped information. Here is the command lines I used to do this:

# extract the XML sections about .m3u files to a new tmp.xml file
perl -ne 'BEGIN{$/="</game>\n";} print m|(<game>.*m3u.*$/)|ms' gamelist.xml >> tmp.xml
# change any <game> tag to a <folder> tag
sed -i .bak "s/<game>/<folder>/g" tmp.xml 
# change any </game> tag to a </folder> tag
sed -i .bak "s/<\/game>/<\/folder>/g" tmp.xml 
# crop any .m3u filename to keep the path only
sed -i .bak 's/\/[[:alnum:][:blank:]\.\+\&,\(\);_-]*.m3u//g' tmp.xml
# delete the closing tag at the end of gamelist.xml
sed -i .bak '/^<\/gameList>/d' gamelist.xml
# add the content of tmp.xml to the end of gamelist.xml
cat tmp.xml >> gamelist.xml
# add the closing tag at the end of gamelist.xml
echo "</gameList>" >> gamelist.xml

And I must say, I'm quite happy with the result : now I have in ES only one scraped entry for any mono-disc game as I have one scraped entry for any subfolder (multi-disc game). When I select a game that is a multi-disc game, It opens the sublist - that is also scraped with the same information - and I just have to select the first entry to run the game.

I hope all this sounds useful.