Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu Sure thing. I've been working on another project of mine already in the mean time.

Keep in mind a few things about the file structure of what you're downloading.

It's different than the Pi filestructure in a few ways.
Our system had a Media folder, then the Systems were inside of that. Pi file structure made me have to change that to be /{system_name}/Media/
Roms in our structure were a folder inside of the media folder. Pi file structure made that completely different.

Please reference the folder structure I had above to see the changes made for Pi.

Used2BeRX

@UDb23 Hey UDb23. I was wondering if you could still make the program put all of the individual files into a global gamelist.xml for each file. That way I would only have to copy 1 single file to the Pi for the conversion (that was what meleu originally thought we were doing too).

Also, I think we should talk about how to make the folder synopsis.txt files for conversion as well. It would be nice to have the entire gamelist.xml file made and then just have the media/roms altered on the pi after the fact.

What do you think?

UDb23

@Used2BeRX when @meleu said:

Go play some games! :D

I'm pretty sure he meant he's looking to make also the txt to xml conversion on the Pi.

Nevertheless I can finalize MetaX with the bugfixes we spoke about and also generate the global gamelist.xml file (on top of single files).

Concerning folder synopsis.txt; I can add this too to MetaX.
I would suggest a structure basically the same as the game txt files:

path=
image=
name=
releasedate=
developer=
publisher=

and description should be at the end of the file and have this line before:
=======

Folder txt files should be named Fold_yyyy.txt, where yyyy should recall the folder name (it's not relevant to the program as the real folder name is in the path inside the txt).

Used2BeRX

@UDb23 Sounds great.

I've been talking to Nemesis and I wanted both his and Meleu's input on one more possible change. I think Nemesis has come up with a way to verify exactly what media is in which folders (waiting on his confirmation on that).

I'm thinking if this is the case, it might be a good idea if MetaX creates empty media fields, ie: <path></path>, <image></image>.

I would want to hear what Nemesis and Meleu have to say about that first though.

Also, although I already re-cased everything to be all caps, it looks like Nemesis has come up with a way to ignore case sensitivity. That could come in handy for other people with their own sets who don't want to go through the massive pain of having to re-case everything.

I appreciate you finishing MetaX though. If I only had to copy one file to the Pi per system before finalizing it with a script it would be much less of a headache.

and description should be at the end of the file and have this line before:
=======

Folder txt files should be named Fold_yyyy.txt, where yyyy should recall the folder name (it's not relevant to the program as the real folder name is in the path inside the txt).

So the line before description for the folders should be exactly 7 "=", or just any number of them?

Also, is it alright if I have the TXT files cased as "FOLD_YYYYYY.TXT"? After re-casing, everything else is all in Caps now for consistency.

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

So the line before description for the folders should be exactly 7 "=", or just any number of them?

Program just needs to understand that's the line before description and not part of another text; so I would say "====" (4 x "=") should be fine. Unless you know some game or description text that includes "====".

Also, is it alright if I have the TXT files cased as "FOLD_YYYYYY.TXT"? After re-casing, everything else is all in Caps now for consistency.

No problem; you can use whatever case you like.

Used2BeRX

@UDb23 Sweet.

Did you choose the ====== because of what I had in the description? I actually used that because I thought it looked good (it actually displayed in the rom screen). Could we just use the "__________________" like the rom entries? No worries if we can't. Just let me know.

I still haven't heard from Nemesis on whether or not we should leave the media fields blank. I'm going to be going to bed soon, so I might not have an answer on that for a while unless he posts it here.

meleu

@Used2BeRX can you post somewhere a pack with all the Synpsis txt files you have?

No need for any media, only txt files.

UDb23

@Used2BeRX we can use whatever chars you like. if we use the same as rom entries, that's fine.

meleu

Hey guys, can you confirm the correlations below?

xml          : txt

<name>       : the very first line of the txt file
<desc>       : the content starting below "______" to the end of file
<releasedate>: "Release Year"
<developer>  : "Developer"
<publisher>  : "Publisher"
<genre>      : "Genre"
<players>    : "Players"
<rating>     : UNUSED
<path>       : UNUSED (script should find the ROM based on the game name)
<image>      : UNUSED (script should find the image based on the game name)
<video>      : UNUSED (script should find the video based on the game name)
<marquee>    : UNUSED (script should find the marquee based on the game name)

Used2BeRX

@UDb23 SUPER IMPORTANT!!!!

I just realized that we're going to need an extra line created by MetaX. I don't think that meleu will be able to properly match all of the files from the global gamelist.xml without a line that has the "GAME FILE NAME" in it. What I mean is, after the global gamelist.xml file is created, there is nothing inside of it that says that the entry for "Mega Man" should match any filenames "MEGA MAN I.XXX"

Could you add that to the program in a field under <name></name> and call it <xtrasname></xtrasname>?

@meleu Those are all good, but there's a lot more than that.

Here's a breakdown of the XML of Super Mario Bros.

<path>./SUPER MARIO BROS I.zip</path>
Script should find ROM based on <xtrasname></xtrasname> (The field I just asked UDb23 to add to MetaX output)

<image>/home/pi/RetroPie/Media/NES/Artwork/Box Front/Super Mario Bros..jpg</image>

<image> = /Artwork/Box Front/ by default. (Was going to talk later about script to change which image type is named as image. I'm hoping that once these are done that at some point skins can take advantage of multiple images at once or just swap which ones you use like we're able to do on our emulators).

Script should find <image> based on <xtrasname></xtrasname>

<cart>/home/pi/RetroPie/Media/NES/Artwork/Cart/Super Mario Bros..jpg</cart>
Cartridge/CD Image: Script should find <cart> based on <xtrasname></xtrasname>

<title>/home/pi/RetroPie/Media/NES/Artwork/Titles/Super Mario Bros..jpg</title>
In game Title Shot: Script should find <title> based on <xtrasname></xtrasname>

<action>/home/pi/RetroPie/Media/NES/Artwork/Action/Super Mario Bros..jpg</action>
In game Action Shot: Script should find <action> based on <xtrasname></xtrasname>

<threedbox>/home/pi/RetroPie/Media/NES/Artwork/3D Boxart/Super Mario Bros..png</threedbox>
3D Boxart: Script should find ROM based on <xtrasname></xtrasname>

<video>/home/pi/RetroPie/Media/NES/Movies/Super Mario Bros..wmv</video>
Video Preview: Script should find ROM based on <xtrasname></xtrasname>

<gamefaq>/home/pi/RetroPie/Media/NES/GameFAQs/Super Mario Bros..Zip</gamefaq>
GameFAQ Collection in Zip File: Script should find <gamefaq> based on <xtrasname></xtrasname>

<manual>/home/pi/RetroPie/Media/NES/Manuals/Super Mario Bros..zip</manual>
Game Manual: Script should find <manual> based on <xtrasname></xtrasname>

<vgmap>/home/pi/RetroPie/Media/NES/VGMaps/Super Mario Bros..zip</vgmap>
VGMap Collection in Zip File: Script should find <vgmap> based on <xtrasname></xtrasname>

<marquee>/home/pi/RetroPie/Media/NES/Artwork/Marquee/Super Mario Bros..jpg</marquee>
Marquee Image: Script should find <marquee> based on <xtrasname></xtrasname>

OTHER FIELDS:
<name>Super Mario Bros.</name>
<xtrasname></xtrasname> (Synopsis File Name) THIS DOESN'T EXIST FROM METAX YET!
<platform>Nintendo Entertainment System</platform>
<region>USA</region>
<media>Cartridge</media>
<controller>NES Gamepad</controller>
<genre>Platformer</genre>
<gametype>Licensed</gametype>
<releasedate>1985</releasedate>
<developer>Nintendo</developer>
<publisher>Nintendo</publisher>
<players>1 or 2 Alternating</players>
<license></license>
<programmer></programmer>
<musician></musician>

You've got an email with the synopsis entries. NES has a folder for synopsis files that MetaX converted to XML files. Unfortunatley at this point it still put in "dummy" fields, so all of the media is pointing to Atari 2600 folders in those changes.

UDb23 was going to fix that and let you pick a media spot, but I wanted to know if it would just be better for UDb23 to make all of those fields and just leave them blank and let your script find any media that exists and add only what is actually in the folders???

Used2BeRX

@meleu To simplify everything for the Linux scripting since we're going to be populating the Media fields, I will take the time to convert all of the media folders to the same names that the roms folders are on Pi.

For example:

NES will be:
/home/pi/RetroPie/Media/nes/
/home/pi/RetroPie/Media/nes/Artwork/
/home/pi/RetroPie/Media/nes/Artwork/3D Boxart/
/home/pi/RetroPie/Media/nes/Artwork/Action/
/home/pi/RetroPie/Media/nes/Artwork/Box Front/
/home/pi/RetroPie/Media/nes/Artwork/Cart/
/home/pi/RetroPie/Media/nes/Artwork/Titles/
/home/pi/RetroPie/Media/nes/Movies/
/home/pi/RetroPie/Media/nes/GameFAQs/
/home/pi/RetroPie/Media/nes/Manuals/
/home/pi/RetroPie/Media/nes/VGMaps/
/home/pi/RetroPie/Media/nes/Marquees/

Game Boy Advance will be:
/home/pi/RetroPie/Media/gba/
/home/pi/RetroPie/Media/gba/Artwork/
/home/pi/RetroPie/Media/gba/Artwork/3D Boxart/
/home/pi/RetroPie/Media/gba/Artwork/Action/
/home/pi/RetroPie/Media/gba/Artwork/Box Front/
/home/pi/RetroPie/Media/gba/Artwork/Cart/
/home/pi/RetroPie/Media/gba/Artwork/Titles/
/home/pi/RetroPie/Media/gba/Movies/
/home/pi/RetroPie/Media/gba/GameFAQs/
/home/pi/RetroPie/Media/gba/Manuals/
/home/pi/RetroPie/Media/gba/VGMaps/
/home/pi/RetroPie/Media/gba/Marquees/

It will take me a while to make these changes and to separate systems that were combined on our old system, but it should eliminate a lot of possible confusion for your scripting.

meleu

@Used2BeRX man, keep in mind that these are the only xml elements that ES takes into account when reading a gamelist.xml:

name
path
desc
image
video
marquee
thumbnail (according to docs it's ignored)
rating
releasedate
developer
publisher
genre
players
playcount (automatically filled)
lastplayed (automatically filled)

Any extra info you have in your txt files is completely ignored.

Used2BeRX

@meleu I know that. I'm hoping that in the future there will be more use for these files. I just want pointers to them in the files in case. Our emulators make great use of all of those files.

Check my last post. I'm going to be re-doing the media folders to match RetroPie standards to simplify the coding necessary to find all the parts and populate the fields.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu I know that. I'm hoping that in the future there will be more use for these files. I just want pointers to them in the files in case.

As I'm trying to convert those files so we can use them on RetroPie/EmulationStation, I'll focus on the currently useful xml elements and ignore the others. If/when the other info become useful, I can try to update the script to make use of them.

Check my last post. I'm going to be re-doing the media folders to match RetroPie standards to simplify the coding necessary to find all the parts and populate the fields.

I think there's no need for this, since my script will try to find them using the find command. But feel free to name the paths as you wish.

meleu

@UDb23 @Used2BeRX OK guys, the first step was taken. The basic functionality of the script is ready. Currently it does nothing for <path>, <image>, <video>, and <marquee>

You can see it here: https://github.com/meleu/share/blob/master/Used2BeTXT.sh

And get it with this command:

wget https://raw.githubusercontent.com/meleu/share/master/Used2BeTXT.sh

Set the permission to execute:

chmod +x Used2BeTXT.sh

And take a look at the --help message:

[PROMPT]$ ./Used2BeTXT.sh --help

Usage:
./Used2BeTXT.sh [OPTIONS] synopsis1.txt [synopsisN.txt ...]

The OPTIONS are:

-h|--help       print this message and exit.

-u|--update     update the script and exit.

The script gets data from "synopsis1.txt" and adds those data in xml format to
a file named "PLATFORM_gamelist.txt", where PLATFORM is the one indicated in
'Platform:' line in "synopsis.txt".

Here is an example of a synopsis txt file named Ace of Aces.txt

Ace Of Aces
Platform: Atari 7800
Region: USA
Media: Cartridge
Controller: 7800 Pro Line Controller
Genre: Flight Simulator - Action
Gametype: Licensed
Release Year: 1988
Developer: Imagineering Inc/Absolute Entertainment
Publisher: Atari
Players: 1
_________________________

Ace of Aces is a 1st person, 2D flight simulation. You're onboard a British RAF Mosquito, maverick fighter bomber. Your mission ... stop enemy trains, intercept
 the terrible german V-1 buzz bombs, sink the german U-boats and down the Nazi bombers. Are you ready for this challenge?

http://www.mobygames.com/game/atari-7800/ace-of-aces

Here is the command line to convert that content to xml and add the metadata to atari7800_gamelist.xml:

[PROMPT]$ ./Used2BeTXT.sh Ace\ of\ Aces.txt

And here is the resulting atari7800_gamelist.xml file:

<?xml version="1.0"?>
<gameList>
  <game>
    <name>Ace Of Aces</name>
    <path/>
    <image/>
    <video/>
    <marquee/>
    <desc>
Ace of Aces is a 1st person, 2D flight simulation. You're onboard a British RAF Mosquito, maverick fighter bomber. Your mission ... stop enemy trains, intercept the terrible german V-1 buzz bombs, sink the german U-boats and down the Nazi bombers. Are you ready for this challenge?&#xD;
&#xD;
http://www.mobygames.com/game/atari-7800/ace-of-aces</desc>
    <releasedate>19880101T000000</releasedate>
    <developer>Imagineering Inc/Absolute Entertainment</developer>
    <publisher>Atari</publisher>
    <genre>Flight Simulator - Action</genre>
    <players>1</players>
  </game>
</gameList>

Good thing: If later the user run the script with another synopsis text file of an atari7800 game, the metadata will be added to the atari7800_gamelist.xml

meleu

Two notes about the conversion:

<players> in gamelist.xml must be an integer. Then strings like 1 or 2 VS; CO-OP will be 2; and 1 to 8 Alternating will be 8.
<releasedate> in gamelist.xml represents an exact date and time and must be in a specific format. The txt files has only the year and in the conversion I'm making <releasedate> be midnight of January 1st of that year. Example: 1987 will be 19870101T000000

Used2BeRX

@meleu

As I'm trying to convert those files so we can use them on RetroPie/EmulationStation, I'll focus on the currently useful xml elements and ignore the others. If/when the other info become useful, I can try to update the script to make use of them.

That's not going to be useful to me since I want to tie all of this in and start asking around for people who can make changes to emulation station when this is all set up to work. I have some people who want to be able to use these XML files on other platforms now with our media as well. It doesn't hurt anything to add them as far as I know.

I think there's no need for this, since my script will try to find them using the find command. But feel free to name the paths as you wish.

That's cool at least. Tons of work I don't have to do now.

<players> in gamelist.xml must be an integer. Then strings like 1 or 2 VS; CO-OP will be 2; and 1 to 8 Alternating will be 8.

That's unfortunate.

<releasedate> in gamelist.xml represents an exact date and time and must be in a specific format. The txt files has only the year and in the conversion I'm making <releasedate> be midnight of January 1st of that year. Example: 1987 will be 19870101T000000

And that's just downright terrible. Who made these decisions? The percentage of titles where it is actually known the exact release date on retro console system games can't be more than 15-20%.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

Who made these decisions?

Aloshi, the guy who created EmulationStation.

That's not going to be useful to me since I want to tie all of this in and start asking around for people who can make changes to emulation station when this is all set up to work.

Not sure if you are aware about how ES development is going today, but Aloshi doesn't touch the ES code since 2015-March and put very clear that he doesn't want to work on this anymore.

The ES branch we use in RetroPie is receiving some improvements by some C++ hackers from RetroPie community (some of them are in this thread here), and as I said in previous post: "I'll focus on the currently useful xml elements and ignore the others. If/when the other info become useful (I mean, implemented in ES), I can try to update the script to make use of them."

Also, I posted on the other forum. It seems that @NeMesiS is already trying to sort your stuff too, using his own coding style. It's all fine for me. I had a good time writing that script because I love to code, but it seems that @NeMesiS is more used with your file/directory structure and is almost done with what you want to do.

I'm going to have a break on this stuff but feel free to contact me if you guys think my help can be useful.

Cheers!

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I want to tie all of this in and start asking around for people who can make changes to emulation station

My suggestion is to ask for one feature at a time. And you have to be very clear about the use case of the respective data.

Example, I love to consult gamefaqs guides, but I have no idea how those files can be useful in emulationstation.

Other metadata I have no idea how can be used in EmulationStation:

action
cart
title
manual
vgmap
platform (a gamelist.xml is very specific for a platform, there's no need for this entry)
media
controller

Making my script add those fields as xml elements on gamelist.xml would be pretty straightforward to code, but most of metadata in the resulting gamelist.xml would be useless for ES (and I'm not sure if ES ignores unused xml elements or it crashes).

Used2BeRX

@meleu I'd like to hear back from Udb23 about the program he is making. I'd like him to add the <xtrasname></xtrasname> field for easy reference for linux scripting to find all of the files. I'd also like him to finish MetaX to add both the fold_xxxx.txt files for the folders and to put together a global gamelist.xml file from that.

All I need at that point is to have a script run that will add all of the media from the different folders and put them in the fields. I know that some of that isn't usable now in EmulationStation, but somebody got in contact with me the other day that would like to use these XML files for other emulation that is based off of XBMC as well. The XML format is very useful in other places as well, and it is likely that even if nobody ever programs any of our stuff to work on the Pi that it will be implemented elsewhere.

Other metadata I have no idea how can be used in EmulationStation:

action

cart

title

manual

vgmap

platform (a gamelist.xml is very specific for a platform, there's no need for this entry)

media

controller

I know at this stage there would be no real use at all for Manuals, VGMaps and GameFAQs. I would suggest that anybody should have a modded XBox to see what the best emulation available actually looks like and to see how this could be integrated. Of particular value is the GameFAQs which you can pull up easily within a game. It's great, especially when you're playing those old 8-bit and 16-bit RPGs.

Media, Controller and platform are really just extraneous information that we displayed on our in-game synopsis. It was way better using the Release Year instead of the weird way that it's done on Emulation Station too considering that almost zero games have the exact day of release and there is no reason to have a time-stamp associated with that field on top of that.

What could be used almost immediately though is the <cart></cart>, <title></title>, <action></action> , <threedbox></threedbox>. It could be made even easier by a few button presses at some point, but immediately a script could be written that would change the fields in any gamelist.xml file (or all of them at once even). This script could allow you to change the image that is displayed in EmulationStation while scrolling through the games.

If somebody were to prefer having an action shot of all of the games instead of the Box Art, for instance, a simple command in the console could change all of the gamelist.xml files to point <image></image> to the "/artwork/action/" directories instead of the "/artwork/box front/" directories.

Making my script add those fields as xml elements on gamelist.xml would be pretty straightforward to code, but most of metadata in the resulting gamelist.xml would be useless for ES (and I'm not sure if ES ignores unused xml elements or it crashes).

It doesn't hurt to add things that aren't used. I'm not 100% sure about that, but I've asked around in several places and people have told me that it doesn't. I would appreciate it if you would write the script to add all of the Media once UDb23 finished MetaX to do everything else in a global gamelist.xml

You seem to be pretty adamant on not adding this other stuff. If you don't want to do that for me, maybe you could at least share your code with Nemesis so that he could see what you've done so far and write it himself at that point.

EDIT: Some of my buddies are already asking about adding new fields for artwork on another thread here: https://retropie.org.uk/forum/topic/10654/es-add-some-news-artworks-to-skin-like-in-a-vg-museum/7

You should check out that video. The 3 guys working on that project I've known and worked with for over 10-12 years. They're very talented.