Making premade gamelist.xml with xtra media collections

meleu

Since I'm going to have all of the TXT files on the Pi, I plan on putting all of them in /home/pi/RetroPie/Media/{system_name}/Synopsis/

Do as you wish. The script gets the input files from command line.

How is the script going to populate the path field for the folders? Should I have the path in the synopsis description somewhere or will your script find it based off of whatever I put in the "Folder: " field?

You're right. The script will try to find based on what you put in the "Folder:" field.

I am taking the time to move all artwork/media to folders that the RetroPie use for consoles, so I will be separating the systems that were grouped together as well. This will make everything easier for everybody to not have a hybrid system. This will take me a while to do, but I just wanted to let you know.

Man, I think it's an insane painful work. But go ahead if you want.

I would suggest to let everything in the RetroPie/Media as it seems to be the way the Xtras are organized.

I am adding a folder to the /Media/{System_Name}/Artwork/ folder called "Folders". This is where I will put media for the folder entries in the gamelist.xml. Each piece of artwork in this folder will identically match the "FOLDERXXXX.TXT" file.

OK.

Because the Folders themselves are not all in UPPER CASE like the Roms and Media, the folder TXT files will not be either. They will match case though. Should the FOLDERXXXXX.TXT files have an upper case or lower case extension (.TXT or .txt) or does it matter?

To be honest I don't like the upper case in everything, like you're doing for the synopsis text files. When I list the directory contents it looks like those files are screaming at me :-)

Do as you wish. The script should be smart enough to deal with different cases.

I've just got home and am working in the script.

Used2BeRX

@meleu said

Since I'm going to have all of the TXT files on the Pi, I plan on putting all of them in /home/pi/RetroPie/Media/{system_name}/Synopsis/

Do as you wish. The script gets the input files from command line.

Okay. Makes sense for me to put them here so that's where they'll be.

How is the script going to populate the path field for the folders? Should I have the path in the synopsis description somewhere or will your script find it based off of whatever I put in the "Folder: " field?

You're right. The script will try to find based on what you put in the "Folder:" field.

Okay. Great.

I am taking the time to move all artwork/media to folders that the RetroPie use for consoles, so I will be separating the systems that were grouped together as well. This will make everything easier for everybody to not have a hybrid system. This will take me a while to do, but I just wanted to let you know.

Man, I think it's an insane painful work. But go ahead if you want.

Yeah. It kind of is, especially since I actually have two versions of artwork based off of resolution, but as I was considering the synopsis I realized that they should all be separate too so they don't get added to 2 or 3 different gamelist.txt files by all being grouped together. I'm already half-way through switching up the first set and I'm getting in the groove so it shouldn't be all that difficult.

I would suggest to let everything in the RetroPie/Media as it seems to be the way the Xtras are organized.

It will still be in the RetroPie/Media. I just mean I'm getting rid of the RetroPie/Media/SMSGG and separating that into RetroPie/Media/gamegear, RetroPie/Media/mastersystem and RetroPie/Media/sg-1000 to match the folders the RetroPie already uses for roms. :)

I am adding a folder to the /Media/{System_Name}/Artwork/ folder called "Folders". This is where I will put media for the folder entries in the gamelist.xml. Each piece of artwork in this folder will identically match the "FOLDERXXXX.TXT" file.

OK.

Good. The folder.txt files will match the names of the folders exactly, so the media will as well.

Because the Folders themselves are not all in UPPER CASE like the Roms and Media, the folder TXT files will not be either. They will match case though. Should the FOLDERXXXXX.TXT files have an upper case or lower case extension (.TXT or .txt) or does it matter?

To be honest I don't like the upper case in everything, like you're doing for the synopsis text files. When I list the directory contents it looks like those files are screaming at me :-)

Do as you wish. The script should be smart enough to deal with different cases.

Yeah... sorry about that. Initially with MetaX it looked as if it wasn't capable of dealing with cases properly. There were slight differences between cases in quite a few of the Media/Roms files too since case didn't matter on the XBox. Converting them to all CAPS solved those issues. Maybe I should have converted them to all lower-case instead, but that would take me a lot of time to change now and I'd have to delete everything on the pi and re-upload it when I was done so I'm not messing with it again. :)

I've just got home and am working in the script.

Sweet.

I'm going to make some "generic" folder.txt files for you sometime in the next few hours. The data in them won't be accurate other than the first line and the folder name and the file name itself. I figure you don't care what else is in there now and it will probably take me a day or two to actually write them up. I send you an email with the preliminary folder.txt files when they're ready.

meleu

@Used2BeRX

I'm getting rid of the RetroPie/Media/SMSGG and separating that into RetroPie/Media/gamegear, RetroPie/Media/mastersystem and RetroPie/Media/sg-1000 to match the folders the RetroPie already uses for roms. :)

please, focus on that and we are almost done!

Used2BeRX

@meleu That's the part I said I was doing before and you said was crazy work and that I didn't need to do it lol.... I don't think I was explaining it right before is all. Yeah, I'm separating all of them, and making those generic folder notepad files while I go along.

BTW... I don't have folders on my Pi for the following systems. I was hoping you might know what they would be and if the Pi even supports them so I can properly name all the folders.

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)
Watara Supervision (I don't see anything about this emulator at all)

THANKS!!!

meleu

@Used2BeRX

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)

I saw somebody requesting to add this on RetroPie, but I think that it wasn't added yet...

Watara Supervision (I don't see anything about this emulator at all)

I don't know this system

Also, there are some synopsis files referring to a platform named "Sega NET", I don't know what emulator/core runs this kind of game on RetroPie...

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX

Pokemon Mini (https://github.com/libretro/PokeMini, don't know how to install this. It's not in the RetroPie packages install menus.)

I saw somebody requesting to add this on RetroPie, but I think that it wasn't added yet...

Watara Supervision (I don't see anything about this emulator at all)

I don't know this system

Supervision was a pathetic attempt to compete with Gameboy. The games weren't great, but there were a couple cool ones.

I'll put both of these systems to the side for now.

Also, there are some synopsis files referring to a platform named "Sega NET", I don't know what emulator/core runs this kind of game on RetroPie...

SegaNET was an online service offered by SEGA. https://en.wikipedia.org/wiki/SegaNet

The roms in our collection run in our Genesis emulator. I'm assuming they run fine on the RetroPie too, although I haven't tested them. Consider these Genesis games since that's where they're going to be.

Here's the first pack of folders. Atari 2600: https://www.sendspace.com/file/ar9r3c

All the data inside is generic except for the first line description and the "Folder: ". The file names match the actual rom folders exactly.

I will try to work fast and get the rest of them to you soon. For now they will all have the Atari 2600 info for everything besides those first two lines, but should be perfect for your testing. I will be working on adding the actual info over the next few days.

Used2BeRX

@meleu Here you go: https://www.sendspace.com/file/jhwy6f

All of the folder.txt files, perfect matches for the folders.

The only systems I'm missing from that are SegaCD, PSX, Virtualboy, Wonderswan and Colecovision. This should be more than enough to make sure the script is working though.

meleu

Hey man, sit down, relax, take a deep breath, and see if it's the kind of file you want:
https://raw.githubusercontent.com/meleu/Used2BeTXT/master/megadrive_FULL_gamelist.xml

:D

Keep in mind that the conversion is very CPU intensive, because the script tries to find all those files you want (and that are useless for ES).

On my intel Core i7 it took almost one hour to process all the synopsis in the genesis folder (all synopsis for megadrive, sega32x and segacd), I think it will take much more time on raspi zero. But it still much more faster than human :-)

I think I can optimize some steps to make it faster, but I'm going to have a break now. :-)

Cheers!

Used2BeRX

@meleu LOOKS AWESOME!!!

I can't wait to see it with the folders I sent you. No worries on the time it takes to make them. I can just leave it running till it's done. Way better than writing that by hand and doing it until the year 2020. :)

I decided to try it out and none of them showed up. I found a bug. :(

Well... it's probably not really a bug, but if you're using the setup as it was, the roms are in the media folder so that's why it's finding them there. Roms for Genesis would be in: /home/pi/RetroPie/roms/megadrive/

The gamelist that you sent has them in /home/pi/RetroPie/Media/Genesis/roms:)

Like I said, probably not a bug. It just found them where they were.

Without knowing how to code anything, I can guess about a way to make the speed faster just based off of that. When it looks for "Roms" have it only look in the "/Roms/{system_name}/ Folder and sub-folders, and when it is looking for specific media, have it look in the /Media/{system_name}/{media_type_folder}.

I think it's probably looking everywhere now from what I'm seeing in the results, and you could probably cut down a lot of time by pointing it in the right directioin before each search.

Freakin' awesome though. We're so close now!

I'm going to do a find/replace on the doc and get back to you in a few. I'm also going to have to change the .png files to .jpg because the old set you have used .png and the re-sized set I made for Pi Zero has smaller .jpg files.

Used2BeRX

@meleu I forgot that I can't use that now since I cased everything in CAPS :)

I'm having a problem though. I've removed everything except for the first entry for "3 Ninjas Kick Back". I'm positive I have the path and image correct, but it won't display in EmulationStation.

I've never done a gamelist.xml for Genesis. Is there someplace special I have to have the gamelist.xml? I altered EmulationStation to display the "Sega Genesis" artwork instead of the Megadrive artwork. I'm not sure if that changes anything.

EDIT: I FIGURED IT OUT!!!!

Not sure if you are aware of this or not... actually, I'm not sure if it's because I changed the artwork or not...

I had to create a shortcut called "genesis" in /opt/retropie/configs/all/emulationstation/gamelists that points to /opt/retropie/configs/all/emulationstation/gamelists/megadrive/

I can't wait to try it out on my end. This is going to be great. Thanks again meleu. :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I decided to try it out and none of them showed up. I found a bug. :(

Man, that file is not supposed to be used on your system. I'm just showing you that my script is able to generate that kind of file. Those paths you see in that file were automatically detected by the script on my system.

You have to run the script with your synopsis files on your own system in order to get a nice gamelist.xml with the right paths.

I have some ideas for optimizations, but I really need to sleep now.

See you later.

Used2BeRX

@meleu Yeah. I figured that out. The only thing that I couldn't figure was why I couldn't get anything to work at all and with genesis you need to have that shortcut to the megadrive folder and then the gamelist.xml works.

Have a good sleep man. You've had a busy day. :)

I'm going to keep working on getting everything prepared for this. I've got a lot of file comparing to do now since I changed so much in the media folders to make it comply to the retropie file system.

Used2BeRX

@meleu It took me almost a full day, but I got every single piece where it needs to be and everything is properly named. I also did this for the pack of things that cannot be used currently and/or cannot be used on a Pi Zero as well. (The only exception is CD based games which I think will bring their own challenges when we get to that point).

Nemesis is working on a way to auto convert names of videos so we can have upgraded MP4 versions above the WMV8 ones we have now. Let me know when you've got something to test out.

I had another idea about a 3rd option when running the script that would probably benefit a Pi Zero. I was thinking maybe a 3rd option would find and include ONLY the first line name from the game.txt files, the rom location/name and the "Box Front" location/name for every game, and it would leave out all other data, even the synopsis data.

We may run into problems on a Pi Zero with all of this data in a large amount of gamelist.xml files, and perhaps with the smaller artwork if everything else was stripped out and a skin that didn't display metadata were used, you could still have a boxart or action screen displayed for every game with the improved names that are in the current game.txt files.

meleu

@Used2BeRX The script already does what you're suggesting (create a gamelist.xml only with metadata useful for ES).

Are you able to test the script?

Please, see if @NeMesiS or other Linux bud from your crew can test my tool and give feedback/suggestions. No need to analyze the code if it seems obscure, just test it with all the options I provided (look the --help) and see if I can improve something.

Used2BeRX

@meleu I wasn't aware that you had uploaded the new one. I thought you were still testing things and trying to tweak it so it didn't take so long to put a list together.

I was talking about making an option that is completely bare bones. One that only has the following 3 things:

First-Line game name from the game.txt
Path to rom
Path to Box Front for <image>

EDIT: I just checked here: https://github.com/meleu/Used2BeTXT

Unless you uploaded the script somewhere else, that's still the one you uploaded a few days ago. Can you tell me where the newest script is so I can test it out?

meleu

@Used2BeRX

@meleu I wasn't aware that you had uploaded the new one.

Ooops! My bad, sorry. I thought I uploaded it... I'll do it when I get home (in ~10 hours).

And then you will be able to get the latest version using the --update option.

I was talking about making an option that is completely bare bones. One that only has the following 3 things:

First-Line game name from the game.txt

Path to rom

Path to Box Front for <image>

Yep. The new (and not yet uploaded) script does it plus description in <desc>. Are you suggesting to add an option to omit the <desc>?

Used2BeRX

@meleu Sounds good. I'll test it out as soon as I can.

Are you suggesting to add an option to omit the <desc>?

Yes. The Pi Zero is very low on performance, and some of those <desc> are reaaaaaaaaly long and make the gamelist.xml files really big.

I'm hoping that if we had a script that only did those 3 things and nothing else that combined with the shrunk jpg box images that I made that I can actually use this with all the games on the Pi Zero without crashing it.

meleu

@Used2BeRX

I'm hoping that if we had a script that only did those 3 things and nothing else that combined with the shrunk jpg box images that I made that I can actually use this with all the games on the Pi Zero without crashing it.

Looks like you're very very proud of the images you have. Otherwise I would suggest the use of the Sseph's scraper. It's a very powerful tool with many options, as you can see in its wiki

EDIT
You can even run it on Windows: https://github.com/sselph/scraper/releases

EDIT 2
It has an option similar to what you talked about some posts above:

-gdb_img="b": Comma seperated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo.

Used2BeRX

@meleu I tried sselph's scraper and was never able to get it to work.

Our images are superior I'm guessing. I touched up over a thousand of them by hand myself and for every game that didn't have a boxart like prototypes and pirates and homebrew we had a talented graphic artist that made boxarts for them.

lol... why would I use a scraper after we've done all of this work?

meleu

@Used2BeRX

lol... why would I use a scraper after we've done all of this work?

haha. I'm doing this because I take it as a diversion. If a scraper can give you the results you want, please, use it! :-)

I'm a kind of data manipulation enthusiast. I love shell scripting, regular expressions, parsing files... With your request in this thread I saw a possiblity to improve my hability to parse files and manage xml files and this is already happening :-)