Making premade gamelist.xml with xtra media collections

meleu

@UDb23 @Used2BeRX OK guys, the first step was taken. The basic functionality of the script is ready. Currently it does nothing for <path>, <image>, <video>, and <marquee>

You can see it here: https://github.com/meleu/share/blob/master/Used2BeTXT.sh

And get it with this command:

wget https://raw.githubusercontent.com/meleu/share/master/Used2BeTXT.sh

Set the permission to execute:

chmod +x Used2BeTXT.sh

And take a look at the --help message:

[PROMPT]$ ./Used2BeTXT.sh --help

Usage:
./Used2BeTXT.sh [OPTIONS] synopsis1.txt [synopsisN.txt ...]

The OPTIONS are:

-h|--help       print this message and exit.

-u|--update     update the script and exit.

The script gets data from "synopsis1.txt" and adds those data in xml format to
a file named "PLATFORM_gamelist.txt", where PLATFORM is the one indicated in
'Platform:' line in "synopsis.txt".

Here is an example of a synopsis txt file named Ace of Aces.txt

Ace Of Aces
Platform: Atari 7800
Region: USA
Media: Cartridge
Controller: 7800 Pro Line Controller
Genre: Flight Simulator - Action
Gametype: Licensed
Release Year: 1988
Developer: Imagineering Inc/Absolute Entertainment
Publisher: Atari
Players: 1
_________________________

Ace of Aces is a 1st person, 2D flight simulation. You're onboard a British RAF Mosquito, maverick fighter bomber. Your mission ... stop enemy trains, intercept
 the terrible german V-1 buzz bombs, sink the german U-boats and down the Nazi bombers. Are you ready for this challenge?

http://www.mobygames.com/game/atari-7800/ace-of-aces

Here is the command line to convert that content to xml and add the metadata to atari7800_gamelist.xml:

[PROMPT]$ ./Used2BeTXT.sh Ace\ of\ Aces.txt

And here is the resulting atari7800_gamelist.xml file:

<?xml version="1.0"?>
<gameList>
  <game>
    <name>Ace Of Aces</name>
    <path/>
    <image/>
    <video/>
    <marquee/>
    <desc>
Ace of Aces is a 1st person, 2D flight simulation. You're onboard a British RAF Mosquito, maverick fighter bomber. Your mission ... stop enemy trains, intercept the terrible german V-1 buzz bombs, sink the german U-boats and down the Nazi bombers. Are you ready for this challenge?&#xD;
&#xD;
http://www.mobygames.com/game/atari-7800/ace-of-aces</desc>
    <releasedate>19880101T000000</releasedate>
    <developer>Imagineering Inc/Absolute Entertainment</developer>
    <publisher>Atari</publisher>
    <genre>Flight Simulator - Action</genre>
    <players>1</players>
  </game>
</gameList>

Good thing: If later the user run the script with another synopsis text file of an atari7800 game, the metadata will be added to the atari7800_gamelist.xml

meleu

Two notes about the conversion:

<players> in gamelist.xml must be an integer. Then strings like 1 or 2 VS; CO-OP will be 2; and 1 to 8 Alternating will be 8.
<releasedate> in gamelist.xml represents an exact date and time and must be in a specific format. The txt files has only the year and in the conversion I'm making <releasedate> be midnight of January 1st of that year. Example: 1987 will be 19870101T000000

Used2BeRX

@meleu

As I'm trying to convert those files so we can use them on RetroPie/EmulationStation, I'll focus on the currently useful xml elements and ignore the others. If/when the other info become useful, I can try to update the script to make use of them.

That's not going to be useful to me since I want to tie all of this in and start asking around for people who can make changes to emulation station when this is all set up to work. I have some people who want to be able to use these XML files on other platforms now with our media as well. It doesn't hurt anything to add them as far as I know.

I think there's no need for this, since my script will try to find them using the find command. But feel free to name the paths as you wish.

That's cool at least. Tons of work I don't have to do now.

<players> in gamelist.xml must be an integer. Then strings like 1 or 2 VS; CO-OP will be 2; and 1 to 8 Alternating will be 8.

That's unfortunate.

<releasedate> in gamelist.xml represents an exact date and time and must be in a specific format. The txt files has only the year and in the conversion I'm making <releasedate> be midnight of January 1st of that year. Example: 1987 will be 19870101T000000

And that's just downright terrible. Who made these decisions? The percentage of titles where it is actually known the exact release date on retro console system games can't be more than 15-20%.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

Who made these decisions?

Aloshi, the guy who created EmulationStation.

That's not going to be useful to me since I want to tie all of this in and start asking around for people who can make changes to emulation station when this is all set up to work.

Not sure if you are aware about how ES development is going today, but Aloshi doesn't touch the ES code since 2015-March and put very clear that he doesn't want to work on this anymore.

The ES branch we use in RetroPie is receiving some improvements by some C++ hackers from RetroPie community (some of them are in this thread here), and as I said in previous post: "I'll focus on the currently useful xml elements and ignore the others. If/when the other info become useful (I mean, implemented in ES), I can try to update the script to make use of them."

Also, I posted on the other forum. It seems that @NeMesiS is already trying to sort your stuff too, using his own coding style. It's all fine for me. I had a good time writing that script because I love to code, but it seems that @NeMesiS is more used with your file/directory structure and is almost done with what you want to do.

I'm going to have a break on this stuff but feel free to contact me if you guys think my help can be useful.

Cheers!

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I want to tie all of this in and start asking around for people who can make changes to emulation station

My suggestion is to ask for one feature at a time. And you have to be very clear about the use case of the respective data.

Example, I love to consult gamefaqs guides, but I have no idea how those files can be useful in emulationstation.

Other metadata I have no idea how can be used in EmulationStation:

action
cart
title
manual
vgmap
platform (a gamelist.xml is very specific for a platform, there's no need for this entry)
media
controller

Making my script add those fields as xml elements on gamelist.xml would be pretty straightforward to code, but most of metadata in the resulting gamelist.xml would be useless for ES (and I'm not sure if ES ignores unused xml elements or it crashes).

Used2BeRX

@meleu I'd like to hear back from Udb23 about the program he is making. I'd like him to add the <xtrasname></xtrasname> field for easy reference for linux scripting to find all of the files. I'd also like him to finish MetaX to add both the fold_xxxx.txt files for the folders and to put together a global gamelist.xml file from that.

All I need at that point is to have a script run that will add all of the media from the different folders and put them in the fields. I know that some of that isn't usable now in EmulationStation, but somebody got in contact with me the other day that would like to use these XML files for other emulation that is based off of XBMC as well. The XML format is very useful in other places as well, and it is likely that even if nobody ever programs any of our stuff to work on the Pi that it will be implemented elsewhere.

Other metadata I have no idea how can be used in EmulationStation:

action

cart

title

manual

vgmap

platform (a gamelist.xml is very specific for a platform, there's no need for this entry)

media

controller

I know at this stage there would be no real use at all for Manuals, VGMaps and GameFAQs. I would suggest that anybody should have a modded XBox to see what the best emulation available actually looks like and to see how this could be integrated. Of particular value is the GameFAQs which you can pull up easily within a game. It's great, especially when you're playing those old 8-bit and 16-bit RPGs.

Media, Controller and platform are really just extraneous information that we displayed on our in-game synopsis. It was way better using the Release Year instead of the weird way that it's done on Emulation Station too considering that almost zero games have the exact day of release and there is no reason to have a time-stamp associated with that field on top of that.

What could be used almost immediately though is the <cart></cart>, <title></title>, <action></action> , <threedbox></threedbox>. It could be made even easier by a few button presses at some point, but immediately a script could be written that would change the fields in any gamelist.xml file (or all of them at once even). This script could allow you to change the image that is displayed in EmulationStation while scrolling through the games.

If somebody were to prefer having an action shot of all of the games instead of the Box Art, for instance, a simple command in the console could change all of the gamelist.xml files to point <image></image> to the "/artwork/action/" directories instead of the "/artwork/box front/" directories.

Making my script add those fields as xml elements on gamelist.xml would be pretty straightforward to code, but most of metadata in the resulting gamelist.xml would be useless for ES (and I'm not sure if ES ignores unused xml elements or it crashes).

It doesn't hurt to add things that aren't used. I'm not 100% sure about that, but I've asked around in several places and people have told me that it doesn't. I would appreciate it if you would write the script to add all of the Media once UDb23 finished MetaX to do everything else in a global gamelist.xml

You seem to be pretty adamant on not adding this other stuff. If you don't want to do that for me, maybe you could at least share your code with Nemesis so that he could see what you've done so far and write it himself at that point.

EDIT: Some of my buddies are already asking about adding new fields for artwork on another thread here: https://retropie.org.uk/forum/topic/10654/es-add-some-news-artworks-to-skin-like-in-a-vg-museum/7

You should check out that video. The 3 guys working on that project I've known and worked with for over 10-12 years. They're very talented.

meleu

@Used2BeRX

You seem to be pretty adamant on not adding this other stuff.

It's not true, I just wasn't seeing how those data could be useful for our use on RetroPie/EmulationStation. I'll try to add an option to run my script creating a xml file with all the fields you have on your synopsis files.

If you don't want to do that for me, maybe you could at least share your code with Nemesis so that he could see what you've done so far and write it himself at that point.

All the links I posted to my scripts is, of course, a link to the code. :-)
But on the other forum @NeMesiS said
"I really don't... I don't understand he's script at all... I wont be able to help...
I might just continue on with my script since I've already put alot of time into it..."

Looks like we have different "coding styles". :-)

Used2BeRX

@meleu Yeah. I suppose I already have it. DOS/Windows think... Doh! Was thinking like MetaX which I wouldn't be able to break down without source code.

Do you understand where I'm coming from on the different artwork folders at least?

Once the gamelist.xml files are made, there could be 4 or 5 scripts made that could be run from the Linux console that would make changes to any gamelist.xml files to change if displays a Title Shot or an Action Shot or a 3D Boxart, etc. You could have an option to only change one system at a time or to do all of them at once.

They could just be called "art-box.sh, art-title.sh, art-cart.sh, art-3dbox.sh, etc.

meleu

@Used2BeRX by the way, that ugly releasedate format is ugly only on the xml file. In the actual ES screen (what the user sees) the date/time format is up to the theme maker.

Used2BeRX

@meleu Thanks. That's good to know. I'll make sure the guys know that when making their skin. No sense in having a release date of "January 1st, XXXX" for 85% of the games if we can just hide that. :)

meleu

@Used2BeRX I think I've made something that can make you (partially) happy.

My script now can convert all of those ~~crazy~~ xml elements you want, you just have to run it with the --full parameter. If you don't know what I'm talking about, maybe your Linuxer fellows do.

I've created a repository for it: https://github.com/meleu/Used2BeTXT

The script can be found here: https://github.com/meleu/Used2BeTXT/blob/master/Used2BeTXT.sh

The command to download it from command line is:

wget https://raw.githubusercontent.com/meleu/Used2BeTXT/master/Used2BeTXT.sh

The script is still unable to fill the fields that are paths to some file (roms and any other media). But looking the messages from @NeMesiS on the other forums make me believe that he knows how to get the path to those files (I think he's using a bunch of finds). Try to make him read this post and take a look at the code and implement the parts marked with TODO.

Example of what he needs to do:

go to script's line 111
fill the path variable with the path to the rom:

path="$(command used to get the path to the ROM)"

do the same for the others fields marked with TODO.
it's enough. Once those variables are correctly filled, the script will be able to generate the xml file you want (I hope).

Here you can see some gamelist.xml files that my script generated

for NES (EmulationStation friendly): https://raw.githubusercontent.com/meleu/Used2BeTXT/master/nintendoentertainmentsystem_gamelist.xml
for NES with all the fields you want: https://raw.githubusercontent.com/meleu/Used2BeTXT/master/nintendoentertainmentsystem_FULL_gamelist.xml
for Atari 7800 (ES-friendly): https://raw.githubusercontent.com/meleu/Used2BeTXT/master/atari7800_gamelist.xml
for Atari 7800 with all the fields you want: https://raw.githubusercontent.com/meleu/Used2BeTXT/master/atari7800_FULL_gamelist.xml
for Atari 2600 (ES-friendly): https://raw.githubusercontent.com/meleu/Used2BeTXT/master/atari2600_gamelist.xml
for Atari 2600 with all the fields you want: https://raw.githubusercontent.com/meleu/Used2BeTXT/master/atari2600_FULL_gamelist.xml

Used2BeRX

@meleu That looks very promising. :)

Are you unable to do the find for the media for some reason? I will direct Nemesis to look at this post, but I do not understand enough to know what you're having a problem with.

I had two questions for you. I've asked them before but you haven't answered them.

Would we be able to make a script that generated the directories and file names with exact extensions in the media fields already generated in MetaX?
Would it be better for MetaX to generate empty media fields and let a script on the Pi find all of that media and generate the contents of the Media fields?

I'm asking because I want to make a post to UDb23 for what I'm currently asking him to do in MetaX. I think he might be lost at this point since we've been talking so much in here recently. I want to have a short and sweet post with exact directions so he doesn't have to sift through 12 pages here. :)

meleu

@Used2BeRX

Are you unable to do the find for the media for some reason?

Yes. The reason is that I don't have the files in my system. :-)

I want to make the script find the media files and then put the files full path in the respective xml element.

I will direct Nemesis to look at this post, but I do not understand enough to know what you're having a problem with.

It seems that @NeMesiS has those Xtras stuff in his system. Therefore he is able to find the media files in his system.

Would we be able to make a script that generated the directories and file names with exact extensions in the media fields already generated in MetaX?

Yes. We just need a standardization - maybe the standard already exists and I don't know it :)

Would it be better for MetaX to generate empty media fields and let a script on the Pi find all of that media and generate the contents of the Media fields?

I'm afraid that generating xml files with empty media fields is exactly what my script does currently. The next step, find the media, is what I was having problems. But now I've found the files (I emailed you about that) and will try to sort this later (need to sleep now)

I'm asking because I want to make a post to UDb23 for what I'm currently asking him to do in MetaX.

I still think that the more natural way to go is to solve everything in Linux (and @NeMesiS seems to agree), but he is liking to code this stuff... It's up to him. :-)

UDb23

@meleu First of all special thanks for your script. It's great a usual !!

@Used2BeRX I see you're very passionate on your project; please keep in mind that we write code and do other things to support Retropie in our free time...and ,in my case, it's limited time.
That said, I do will complete MetaX (can't commit on timing) with all fields/tags and complete gamelists.xml (including folder info) generation.

phulshof

Might it be useful to agree on a new file format, like the truRIP dat format? I could get that group to add the fields we want, and I could add support to ES just like I did for RetroFE. That way we have a single format supported by a major preservation group that can be used cross front-ends...

UDb23

@phulshof said in Making premade gamelist.xml with xtra media collections:

like the truRIP dat format

What is the database/xml structure for that ?

phulshof

@UDb23 said in Making premade gamelist.xml with xtra media collections:

@phulshof said in Making premade gamelist.xml with xtra media collections:

like the truRIP dat format

What is the database/xml structure for that ?

http://trurip.org/
They've recently released new databases for many optical media systems like the GameCube. A tad too fancy for the current pi perhaps, but they're working on complete databases and corresponding art sets.

meleu

@Used2BeRX can you post here an example of a synopsis text file for folder?

I think it must have the same format as the synopsis you already have:

the very first line of the file is the name of the folder
the description goes below the lines that start with underlines ________ and goes to the end of file
some line must indicate that this is a folder synopsis, my suggestion: a line starting with Folder:.
other fields can go exactly the same way you do for game synopsis.
the actual filename is completely irrelevant, the script only looks at file's contents.

meleu

@Used2BeRX Luckily I'm having fun coding this stuff and having free time to look into it... :-)

The script is now able to create <folder> entries based on the file structure I described in the post above.

For example it is able to convert this folder.txt:

(1) Licensed Atari 7800 ProSystem Games
Folder: (1) Licensed
Platform: Atari 7800
Release Year: 1982
Developer: Atari Inc.
Publisher: Atari Inc.
_________________________

The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986.  Bla bla bla bla...

Bla bla bla bla...

Into this <folder> entry in gamelist.xml:

  <folder>
    <name>(1) Licensed Atari 7800 ProSystem Games</name>
    <path/>
    <image/>
    <video/>
    <marquee/>
    <desc>
The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986.  Bla bla bla bla...

Bla bla bla bla...</desc>
    <releasedate>19820101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>

Of course, it still lacks the roms/media finding feature, but I think I'm on the right way...

Used2BeRX

@meleu looks great meleu. I've been sleeping (and I'm not done yet), but I got up and saw all of this and wanted to post back. I believe you found everything you need at this point, or was there still something you were asking me for?

@UDb23 Thanks. As Meleu keeps plugging away at this it is possible that we won't need MetaX in the end actually. We'll have to see how it goes though. There needs to be certain things that happen and the ability to do a few things consistantly before I'm sure of it. It looks like I'm going to have to have all of the TXT files on the Pi (around 11,000 of them), and I was hoping to avoid that, but in the end this is still way better than writing the rest of these by hand.

@phulshof I sent you an email, and inside of it I told you that we have no intention of changing our file structure because for any single change there would be over a dozen or more other changes that would need to be made. Since you focus mainly at newer optical media at this point, and largely we do the earlier systems, you are welcome to adopt our naming system, otherwise we will remain two separate projects.