Making premade gamelist.xml with xtra media collections

UDb23

@phulshof said in Making premade gamelist.xml with xtra media collections:

like the truRIP dat format

What is the database/xml structure for that ?

phulshof

@UDb23 said in Making premade gamelist.xml with xtra media collections:

@phulshof said in Making premade gamelist.xml with xtra media collections:

like the truRIP dat format

What is the database/xml structure for that ?

http://trurip.org/
They've recently released new databases for many optical media systems like the GameCube. A tad too fancy for the current pi perhaps, but they're working on complete databases and corresponding art sets.

meleu

@Used2BeRX can you post here an example of a synopsis text file for folder?

I think it must have the same format as the synopsis you already have:

the very first line of the file is the name of the folder
the description goes below the lines that start with underlines ________ and goes to the end of file
some line must indicate that this is a folder synopsis, my suggestion: a line starting with Folder:.
other fields can go exactly the same way you do for game synopsis.
the actual filename is completely irrelevant, the script only looks at file's contents.

meleu

@Used2BeRX Luckily I'm having fun coding this stuff and having free time to look into it... :-)

The script is now able to create <folder> entries based on the file structure I described in the post above.

For example it is able to convert this folder.txt:

(1) Licensed Atari 7800 ProSystem Games
Folder: (1) Licensed
Platform: Atari 7800
Release Year: 1982
Developer: Atari Inc.
Publisher: Atari Inc.
_________________________

The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986.  Bla bla bla bla...

Bla bla bla bla...

Into this <folder> entry in gamelist.xml:

  <folder>
    <name>(1) Licensed Atari 7800 ProSystem Games</name>
    <path/>
    <image/>
    <video/>
    <marquee/>
    <desc>
The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986.  Bla bla bla bla...

Bla bla bla bla...</desc>
    <releasedate>19820101T000000</releasedate>
    <developer>Atari Inc.</developer>
    <publisher>Atari Inc.</publisher>
    <genre/>
    <players/>
  </folder>

Of course, it still lacks the roms/media finding feature, but I think I'm on the right way...

Used2BeRX

@meleu looks great meleu. I've been sleeping (and I'm not done yet), but I got up and saw all of this and wanted to post back. I believe you found everything you need at this point, or was there still something you were asking me for?

@UDb23 Thanks. As Meleu keeps plugging away at this it is possible that we won't need MetaX in the end actually. We'll have to see how it goes though. There needs to be certain things that happen and the ability to do a few things consistantly before I'm sure of it. It looks like I'm going to have to have all of the TXT files on the Pi (around 11,000 of them), and I was hoping to avoid that, but in the end this is still way better than writing the rest of these by hand.

@phulshof I sent you an email, and inside of it I told you that we have no intention of changing our file structure because for any single change there would be over a dozen or more other changes that would need to be made. Since you focus mainly at newer optical media at this point, and largely we do the earlier systems, you are welcome to adopt our naming system, otherwise we will remain two separate projects.

meleu

@Used2BeRX

was there still something you were asking me for?

Yes. Folder synopsis text files.

It looks like I'm going to have to have all of the TXT files on the Pi (around 11,000 of them)

The files can be at any Linux system. Putting them on your pi is an option, but not the only one.

You can, for example, install a Linux virtual machine on your Windows computer and the Linux virtual machine (called guest system) can share files with Windows real machine (called host system).

It can be all new to you, but keep in mind that first of all we are a learning community. And if you want to maintain/manage all that huge amount of data you have, you must learn some computer stuff and how to automate things. Otherwise you'll spend most of your time manually editing files and less time actually playing games, and this is really sad...

At least one programming language would help you a lot (if you don't know any, I suggest python).

edit: if you want to try Linux on virtual machine, I suggest VirtualBox to manage virtual machines, and Linux Mint, as it's the most easy to use Linux distrubution I know.

Used2BeRX

@meleu I thought you were able to get those synopsis files when you got back home from work. You're still blocked from mega then?

Hopefully you can get them from the link I sent in email, or you're going to have to tell me how to get them to you.

My computer can't handle virtual box. I'm poor and out of work and I'm running this on a computer that my brother gave me from around 2007. It barely runs Windows 10. Up until last week I had a laptop too, but since don't have air conditioning it died from overheating, so now when this thing slows down on me while it's doing something I just have to sit and wait instead of switching between the two of them. For me, the only option is to have 11,000 txt files on the Pi.

(This is also why I'm still working on a Pi Zero. It's not even my Pi Zero... it's my brothers. I want to buy a Pi 3, but I can't afford one.)

meleu

@Used2BeRX

@meleu I thought you were able to get those synopsis files when you got back home from work. You're still blocked from mega then?

Oh... I was at home only for sleep last night. I left the computer downloading that stuff. I'll take a look at those files this evening. But could you paste here the content of one example of folder synopsis text file?

I'm out of work

Then why are you spending your time with video game stuff?! :-)
(just kidding, please don't take it seriously)

I'm looking forward to get home and finally convert all those damn text files! :-)

Cheers!

Used2BeRX

@meleu I uploaded it to another place and sent it to your email. Maybe you can look at them from there. Here's "SUPER MARIO BROS I.TXT" on NES.

Super Mario Bros.
Platform: Nintendo Entertainment System
Region: USA
Media: Cartridge
Controller: NES Gamepad
Genre: Platformer
Gametype: Licensed
Release Year: 1985
Developer: Nintendo
Publisher: Nintendo
Players: 1 or 2 Alternating
_________________________

One day the kingdom of the peaceful mushroom people was invaded by the Koopaa tribe of turtles famous for their black magic. The quiet, peace-loving Mushroom People were turned into mere stones, bricks and even field horse-hair plants, and the Mushroom Kingdom fell into ruin.

The only one who can undo the magic spell on the Mushroom People and return them to their normal selves is the Princess Toadstool, the daughter of the Mushroom King. Unfortunately, she is presently in the hands of the great Koopa turtle king.

Mario, the hero of this story (maybe) hears about the Mushroom People's plight and sets out on a quest to free the Mushroom Princess from the evil Koopa and restore the fallen kingdom of the Mushroom People.

You are Mario! It's up to you to save the Mushroom People from the black magic of the Koopa!

--From the NES Super Mario Bros. instruction manual.

Overview:

Super Mario Bros. was one of the great classics, and everyone has played it, but we will give you a little insight on the game. Based on the arcade game Mario Bros., Super Mario Bros. became one of the most popular games ever. It had decent graphics and was fairly easy to play. You play either Mario Or Luigi, two plumbers that are trying to save the Princess Toadstool of the Mushroom Kingdom from the dreaded King Koopa. Super Mario Bros. was the start of a great series that has continued to this day.

Directions:

Run, jump, and collect items, but most importantly keep moving. Collect Mushrooms to get bigger and flowers to spit fire. Collect coins to get extra lives.

Nine worlds, containing three levels each, stand between you and your goal. Find secret areas that might let you skip certain levels and save the princess. 

Controls:

A Button: Jump
B Button: Run, shoot fireballs
Start Button: Pause the game/Start the game
Select Button: Select 1 player, 2 player(or Duck Hunt if playing SMB/DH)

Hints:

Continue
If you die and lose all your lives, just hold A+B and press start at the start screen to start at the first level of the world you were in.

Extra Points
When finishing a stage you have to jump onto a flagpole as high as you can. To get the highest score possible, be sure to land on the pole as high as possible and try to land on the pole when the clock's last digit is a multiple of three. If you do this, fireworks will go off that give you extra points.

http://consoleclassix.com/nes/super_mario_bros.html

Then why are you spending your time with video game stuff?! :-)

I haven't been able to get a good job since I was laid off in 2010. I bought a rehab house when the market crashed though so my bills are almost non-existent. Part time bullshit jobs suck my balls so I only work when I need to. Can't afford to buy anything nice, but you get used to it. It's so nice not having a boss most of the time and just doing whatever the hell I want to do.

I'm still rockin' a 720 resolution projection TV. Hopefully my family starts upgrading their flat screens soon and I can get one or two of them. My bro is going to be giving me his XBox 360 since he doesn't use it anymore, so I'll have about 20 new games to play. :)

meleu

@Used2BeRX I meant the FOLDER synopsis you were talking about with @UDb23 in a post above, not the games' synopsis.

I already got the games' synopsis from the link you sent me and generated all those xml files in the Used2BeTXT repo using them.

Used2BeRX

@meleu My mistake.

You're going to have to wait on those. I have to write them. My project for the day. :)

I'll up them when I finish them all.

SOME NOTES: (Tell me if these are good or bad):

Since I'm going to have all of the TXT files on the Pi, I plan on putting all of them in /home/pi/RetroPie/Media/{system_name}/Synopsis/
How is the script going to populate the path field for the folders? Should I have the path in the synopsis description somewhere or will your script find it based off of whatever I put in the "Folder: " field?
I am taking the time to move all artwork/media to folders that the RetroPie use for consoles, so I will be separating the systems that were grouped together as well. This will make everything easier for everybody to not have a hybrid system. This will take me a while to do, but I just wanted to let you know.
I am adding a folder to the /Media/{System_Name}/Artwork/ folder called "Folders". This is where I will put media for the folder entries in the gamelist.xml. Each piece of artwork in this folder will identically match the "FOLDERXXXX.TXT" file.
Because the Folders themselves are not all in UPPER CASE like the Roms and Media, the folder TXT files will not be either. They will match case though. Should the FOLDERXXXXX.TXT files have an upper case or lower case extension (.TXT or .txt) or does it matter?

meleu

@Used2BeRX

Since I'm going to have all of the TXT files on the Pi, I plan on putting all of them in /home/pi/RetroPie/Media/{system_name}/Synopsis/

Do as you wish. The script gets the input files from command line.

How is the script going to populate the path field for the folders? Should I have the path in the synopsis description somewhere or will your script find it based off of whatever I put in the "Folder: " field?

You're right. The script will try to find based on what you put in the "Folder:" field.

I am taking the time to move all artwork/media to folders that the RetroPie use for consoles, so I will be separating the systems that were grouped together as well. This will make everything easier for everybody to not have a hybrid system. This will take me a while to do, but I just wanted to let you know.

Man, I think it's an insane painful work. But go ahead if you want.

I would suggest to let everything in the RetroPie/Media as it seems to be the way the Xtras are organized.

I am adding a folder to the /Media/{System_Name}/Artwork/ folder called "Folders". This is where I will put media for the folder entries in the gamelist.xml. Each piece of artwork in this folder will identically match the "FOLDERXXXX.TXT" file.

OK.

Because the Folders themselves are not all in UPPER CASE like the Roms and Media, the folder TXT files will not be either. They will match case though. Should the FOLDERXXXXX.TXT files have an upper case or lower case extension (.TXT or .txt) or does it matter?

To be honest I don't like the upper case in everything, like you're doing for the synopsis text files. When I list the directory contents it looks like those files are screaming at me :-)

Do as you wish. The script should be smart enough to deal with different cases.

I've just got home and am working in the script.

Used2BeRX

@meleu said

Since I'm going to have all of the TXT files on the Pi, I plan on putting all of them in /home/pi/RetroPie/Media/{system_name}/Synopsis/

Do as you wish. The script gets the input files from command line.

Okay. Makes sense for me to put them here so that's where they'll be.

How is the script going to populate the path field for the folders? Should I have the path in the synopsis description somewhere or will your script find it based off of whatever I put in the "Folder: " field?

You're right. The script will try to find based on what you put in the "Folder:" field.

Okay. Great.

I am taking the time to move all artwork/media to folders that the RetroPie use for consoles, so I will be separating the systems that were grouped together as well. This will make everything easier for everybody to not have a hybrid system. This will take me a while to do, but I just wanted to let you know.

Man, I think it's an insane painful work. But go ahead if you want.

Yeah. It kind of is, especially since I actually have two versions of artwork based off of resolution, but as I was considering the synopsis I realized that they should all be separate too so they don't get added to 2 or 3 different gamelist.txt files by all being grouped together. I'm already half-way through switching up the first set and I'm getting in the groove so it shouldn't be all that difficult.

I would suggest to let everything in the RetroPie/Media as it seems to be the way the Xtras are organized.

It will still be in the RetroPie/Media. I just mean I'm getting rid of the RetroPie/Media/SMSGG and separating that into RetroPie/Media/gamegear, RetroPie/Media/mastersystem and RetroPie/Media/sg-1000 to match the folders the RetroPie already uses for roms. :)

I am adding a folder to the /Media/{System_Name}/Artwork/ folder called "Folders". This is where I will put media for the folder entries in the gamelist.xml. Each piece of artwork in this folder will identically match the "FOLDERXXXX.TXT" file.

OK.

Good. The folder.txt files will match the names of the folders exactly, so the media will as well.

Because the Folders themselves are not all in UPPER CASE like the Roms and Media, the folder TXT files will not be either. They will match case though. Should the FOLDERXXXXX.TXT files have an upper case or lower case extension (.TXT or .txt) or does it matter?

To be honest I don't like the upper case in everything, like you're doing for the synopsis text files. When I list the directory contents it looks like those files are screaming at me :-)

Do as you wish. The script should be smart enough to deal with different cases.

Yeah... sorry about that. Initially with MetaX it looked as if it wasn't capable of dealing with cases properly. There were slight differences between cases in quite a few of the Media/Roms files too since case didn't matter on the XBox. Converting them to all CAPS solved those issues. Maybe I should have converted them to all lower-case instead, but that would take me a lot of time to change now and I'd have to delete everything on the pi and re-upload it when I was done so I'm not messing with it again. :)

I've just got home and am working in the script.

Sweet.

I'm going to make some "generic" folder.txt files for you sometime in the next few hours. The data in them won't be accurate other than the first line and the folder name and the file name itself. I figure you don't care what else is in there now and it will probably take me a day or two to actually write them up. I send you an email with the preliminary folder.txt files when they're ready.

meleu

@Used2BeRX

I'm getting rid of the RetroPie/Media/SMSGG and separating that into RetroPie/Media/gamegear, RetroPie/Media/mastersystem and RetroPie/Media/sg-1000 to match the folders the RetroPie already uses for roms. :)

please, focus on that and we are almost done!

Used2BeRX

@meleu That's the part I said I was doing before and you said was crazy work and that I didn't need to do it lol.... I don't think I was explaining it right before is all. Yeah, I'm separating all of them, and making those generic folder notepad files while I go along.

BTW... I don't have folders on my Pi for the following systems. I was hoping you might know what they would be and if the Pi even supports them so I can properly name all the folders.

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)
Watara Supervision (I don't see anything about this emulator at all)

THANKS!!!

meleu

@Used2BeRX

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)

I saw somebody requesting to add this on RetroPie, but I think that it wasn't added yet...

Watara Supervision (I don't see anything about this emulator at all)

I don't know this system

Also, there are some synopsis files referring to a platform named "Sega NET", I don't know what emulator/core runs this kind of game on RetroPie...

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)

I saw somebody requesting to add this on RetroPie, but I think that it wasn't added yet...

Watara Supervision (I don't see anything about this emulator at all)

I don't know this system

Supervision was a pathetic attempt to compete with Gameboy. The games weren't great, but there were a couple cool ones.

I'll put both of these systems to the side for now.

Also, there are some synopsis files referring to a platform named "Sega NET", I don't know what emulator/core runs this kind of game on RetroPie...

SegaNET was an online service offered by SEGA. https://en.wikipedia.org/wiki/SegaNet

The roms in our collection run in our Genesis emulator. I'm assuming they run fine on the RetroPie too, although I haven't tested them. Consider these Genesis games since that's where they're going to be.

Here's the first pack of folders. Atari 2600: https://www.sendspace.com/file/ar9r3c

All the data inside is generic except for the first line description and the "Folder: ". The file names match the actual rom folders exactly.

I will try to work fast and get the rest of them to you soon. For now they will all have the Atari 2600 info for everything besides those first two lines, but should be perfect for your testing. I will be working on adding the actual info over the next few days.

Used2BeRX

@meleu Here you go: https://www.sendspace.com/file/jhwy6f

All of the folder.txt files, perfect matches for the folders.

The only systems I'm missing from that are SegaCD, PSX, Virtualboy, Wonderswan and Colecovision. This should be more than enough to make sure the script is working though.

meleu

Hey man, sit down, relax, take a deep breath, and see if it's the kind of file you want:
https://raw.githubusercontent.com/meleu/Used2BeTXT/master/megadrive_FULL_gamelist.xml

:D

Keep in mind that the conversion is very CPU intensive, because the script tries to find all those files you want (and that are useless for ES).

On my intel Core i7 it took almost one hour to process all the synopsis in the genesis folder (all synopsis for megadrive, sega32x and segacd), I think it will take much more time on raspi zero. But it still much more faster than human :-)

I think I can optimize some steps to make it faster, but I'm going to have a break now. :-)

Cheers!

Used2BeRX

@meleu LOOKS AWESOME!!!

I can't wait to see it with the folders I sent you. No worries on the time it takes to make them. I can just leave it running till it's done. Way better than writing that by hand and doing it until the year 2020. :)

I decided to try it out and none of them showed up. I found a bug. :(

Well... it's probably not really a bug, but if you're using the setup as it was, the roms are in the media folder so that's why it's finding them there. Roms for Genesis would be in: /home/pi/RetroPie/roms/megadrive/

The gamelist that you sent has them in /home/pi/RetroPie/Media/Genesis/roms:)

Like I said, probably not a bug. It just found them where they were.

Without knowing how to code anything, I can guess about a way to make the speed faster just based off of that. When it looks for "Roms" have it only look in the "/Roms/{system_name}/ Folder and sub-folders, and when it is looking for specific media, have it look in the /Media/{system_name}/{media_type_folder}.

I think it's probably looking everywhere now from what I'm seeing in the results, and you could probably cut down a lot of time by pointing it in the right directioin before each search.

Freakin' awesome though. We're so close now!

I'm going to do a find/replace on the doc and get back to you in a few. I'm also going to have to change the .png files to .jpg because the old set you have used .png and the re-sized set I made for Pi Zero has smaller .jpg files.