Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu Hey Meleu...

You don't know of any program or an easy way I could input the gamelist.xmlfile and be able to sort the values inside like a spreadsheet or like in MS Access, do you?

Maybe I'm making this more complicated than it needs to be.

EmulationStation itself already does this for any of the fields that are displayed in the romlist/skins. Maybe there's another simple program out there that can just display all of the info if I tell it what fields to look for without actually launching any roms or anything?

meleu

@used2berx sorry, I didn't get what you want exactly. Could you give an example?

Used2BeRX

@meleu I don't have the Pi available right now, but I recall that somewhere in the romlists you can choose to sort by different fields, such as Genre or # of players.

I'd be looking for some sort of way to be able to do this for all of the fields... even the ones not currently supported by ES.

It would be ideal if I could actually edit the XML from wherever I was seeing this sorted data because I'd be able to convert it back to synopsis.txt files if you made that script. But even if I can't still edit this info from here I'd at least know which synopsis.txt files I needed to go in and edit.

Used2BeRX

@meleu Hey man. I've been all over the map with this project, but my focus is now back to the synopsis. I've still got quite a few more dozens of hours work to put into it before I'm ready to start messing with the script again, but I was wondering if you'd have some time to work with me on a few things in the future. I realize the holidays are coming up soon and that might not be possible.

I'll try to organize my thoughts here and get everything I was thinking of in the same post.

Just so I don't forget, another thing I was hoping that we could do is use the Players: field in the synopsis.txt files to create two tags in the gamelist.xml. The first would be our standard <players> tag that uses your code to only show the maximum amount of players in the game, but also a <players-full> tag that would show the text string that I put on that line. Personally I prefer the way that we've displayed that on the XBox and I'm hoping that if this is built on my end I might one day be able to get somebody here to add that possible tag to the setup here. I know it wouldn't be usable for anything right now. Maybe we could just throw that in the "full" option and leave it out when we don't use that.

Used2BeRX

@meleu I had a Linux question for you Meleu.

I think what I want to do would be much easier in Linux than in Notepad++

In re-doing all of the synopsis entries, I have been using the / character to separate the genre types, if more than one, in an individual game. Just the other day though, I thought using that character might cause errors in the gamelist.xml (I'm not sure about that). But even if it doesn't, I think I'd rather use the | character instead for separating the genres since it looks better. (Not sure if that would cause any errors either).

So...... Two questions, actually. Would / or | cause any problems in the gamelist.xml file?

And, is there a command or simple script I can run that would only change the / to | in instances where / is located in the Genre: line?

For example, changing any instances of the following:

Genre: Arcade / Action / Platform

To:

Genre: Arcade | Action | Platform

Thanks!

meleu

@used2berx hey bro. The tool to do this magic is sed. I know it isn't the most intuitive tool, but I'm sure that it's the perfect tool to do the kind of work you do.

Oh! Here are the magic words you need:

sed -i '/^Genre:/ s#/#|#g' file.txt

If you want to apply this to all your .txt files you can use *.txt rather than file.txt.

meleu

@Used2BeRX I think I'll have some free time to play with bash code in the next couple of days. Can you update your "wishlist"?

Here is the last one I've found in a post above, and I think I already implemented many of those things:

Create the backup folder if it doesn't exist, and backup any gamelist.xml file before creating a new one if one of the append options isn't used on an exisiting gamelist.

Done in this commit.

Make option to change which artwork tag is set as <image> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made.

This item still is in the ToDo list. :)

Platform: Nintendo Entertainment System and Platform: Famicom Disk System entries in the synopsis.txt files located in /Media/nes/Synopsis/ folder will treat these entries as if they were NES and not FDS games and look for roms and all media in the /nes/ folders instead.

Done in this commit.

Add <marquee> tag to both regular and FULL operations, looking for the media in /Media/[System]/Marquee/ folders. (Search for both .JPG and .PNG).

Done in this commit.

IIRC another thing you asked was to remove any line with only an URL in the description. ~~It's in the ToDo list also.~~ EDIT: Done in this commit.

If you have more to request, add comments here:
https://github.com/meleu/Used2BeTXT/issues/1

I can't assure I will do them, but from time to time I like spend my free time with coding rather than playing videogames. :D

Used2BeRX

@meleu Thanks for the command above :)

I'm going to probably be trying that out by sometime next week. I'll be sure to do it on a backup set of files just so I don't mess anything up. I bet there is a lot of other cool stuff that I can do with that command that can really help me in the future as well. Notepad++ can actually do a fairly decent job of bulk "find and replace", but it's still too limited for what I want to do and I remembered you said something about this command before.

I should have some time in the next 2 days to sit down and figure out what I want to do and update my wishlist. I know you hit a lot of the important things there, but there's so many posts on this thread now and so much time has gone by while I was doing other things that I don't even remember what all I wanted to see happen.

The only thing I would add to what you put above was an extension of Numbers 2 and 4 above:

Make option to change which artwork tag is set as <image> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made.

Add <marquee> tag to both regular and FULL operations, looking for the media in /Media/[System]/Marquee/ folders. (Search for both .JPG and .PNG).

If you've already done both of these, it would be sweet to be able to do the same thing we did to the <image> tag in number 2, to the <marquee> tag in number 4.

Currently, we are able to display two images per game since the introduction of the <marquee> tag. This doesn't necessarily have to be a Marquee image though.

If we are able to quickly change which type of images we want to see on the fly with your script, this would open up a lot of different options. Maybe somebody would like to display an image of the Box and the Cart? Maybe somebody else would like to display the Box and a title shot? Or the Box and an action shot?

Personally, I'm probably going to go with two setups, one for the Pi Zero and one for the Pi3. Because I'll have videos that will work with the Pi3, it would be nice to also have a Box and Cart image for all of the games with it. But the videos won't work on the Pi Zero, so it would be nice to have either the Box and the Title or the Box and the Action shot.

Thanks so much man. I'll get an updated wishlist to you if there was anything else missing and I'll add them all to the link you sent me. :)

Used2BeRX

@meleu Hey bud :)

I believe I've found everything I requested before that you haven't put in there and everything that I can currently think of at the moment. I might have more to add to the wishlist after I finally get to running this thing again and kicking the tires a bit, but for now I've added the stuff to the wishlist page you provided. I've bookmarked it too so I can put the stuff up there and we could keep track of it. I should probably at some point add all of the things you've already done there so we can have a list of everything the script is capable of.

I put everything in my order of personal importance. I'll post them again here for ya now too. Please let me know if you have any questions about anything I'm requesting, and if you want me to re-word or simplify any of it on your checklist page.

HUGE THANKS AGAIN!!!

This is going to be awesome. :)

Synopsis script wishlist:

Platform: Nintendo Entertainment System and Platform: Famicom Disk System entries in the synopsis.txt files located in /Media/nes/Synopsis/ folder will treat these entries as if they were NES and not FDS games and look for roms and all media in the /nes/ folders instead. (This should not occur if the synopsis, media and roms are located in the /Media/fds/ directory though).
Make option to change which artwork tag is set as <marquee> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made. (Same as what I believe you already did for the <image> tag). This way we can easily change what type of image displays for two image types after the gamelist.xml is created.

I'm going to add "dummy" *.jpg files in the Marquee folder since I don't actually have any marquee images, unless you have a better solution for this. I think right now the script just wouldn't create <marquee> tags at all, so this request wouldn't actually do anything unless the original run of the script created those lines in the gamelist.xml

Add the following fields to the Full option:

Translation Release Year:
Translated By:
Hack Release Year:
Hacked By:
Original Title:
Alternate Title:

These aren't always used in every synopsis, so it should only create these in the entries that have them in the synopsis.txt file

Use the Players: field in the synopsis.txt files to create two tags in the gamelist.xml. The first would be our standard <players> tag that uses your code to only show the maximum amount of players in the game, but also a <players-full> tag that would show the text string that I put on that line. Personally I prefer the way that we've displayed that on the XBox and I'm hoping that if this is built on my end I might one day be able to get somebody here to add that possible tag to the setup here. I know it wouldn't be usable for anything right now. Maybe we could just throw that in the "full" option and leave it out when we don't use that.
gamelist.xml to synopsis.txt reverse-order feature.

Used2BeRX

Oh.... BTW....

I've been in the middle of re-writing the synopsis files so they work better with RetroPie. (Not only because of memory constraints, but nobody wants to read a slowly scrolling novel for a game description either). You've probably read that I'm doing anything necessary to reduce the file size for each game to only 1kb or less. Sometimes, this requires me to re-word some of the descriptions, but I'm taking extra care to make them still be great descriptions for each game.

The original file size for all NES synopsis files was somewhere around 3.40MB for only 1,700 or so games. Right now, the file size is 1.79MB for 2,020 games and shrinking. :)

Can't wait to see what this all looks like after re-running your script when the edits are done!

Used2BeRX

@used2berx Well I finally got through the rewrites, and the file size is now 1.57MB. So even though there is 300-400+ more games than used to be in the collection, the file size of the gamelist.xml is going to be about 2.1MB less than it used to be.

Used2BeRX

@meleu Hey man.

I've begun my testing on the Pi Zero for the NES games, and so far so good with that. 100% of them play even on a Pi Zero with at least one of the 3 emulators. (Default being quicknes of course.... and only about 100 of the US Licensed games. I'm assuming I'm going to run into a lot of problems when we start getting into hacks, translations, pirates, etc.).

Before I go to bed, I'm going to re-run the script with all the new synopsis files with the FULL command and see what happens with the script as is. I'll let you know if anything that we think should be working has any problems.

I haven't re-sized a lot of the artwork for new games or anything yet, so it's going to be a very incomplete gamelist.xml for now. I'm just re-running it at this point for testing purposes as well as "fixing" the names of all of the rom entries to display the full name instead of just the rom-file names.

I'll let you know how it goes hopefully in a few days. :)

Used2BeRX

@meleu Hey man. If you wouldn't mind, could you update this page to have all of the current commands and features in the script?

https://github.com/meleu/Used2BeTXT/blob/master/README.md

I have some questions about how things work and I'm seeing some odd behaviors that I wasn't expecting, but I'm not sure if I'm just not using the program right and I don't want to bother you with them yet. I'm specifically speaking about the backup process and if a script should be overwritten, appended, or written from scratch.

I know you might have that in the --help command, but I don't want to break the program now to check at the moment. Was hoping if it's in the help, you could just copy and paste that to the readme file.

Thanks :)

meleu

@used2berx the README has been updated.

Used2BeRX

@meleu Thanks. I should have some time this weekend to kick the tires a bit. Don't know how much time I'll have before then though, but if I figure anything out I'll let you know.

Used2BeRX

@meleu Hey bud,

I was wondering if it is possible to not make tags if the media/field isn't available for the game? I wanted to ask you before I put it on the wishlist.

I was really surprised after running a FULL command that we had a 3.00MB file. Granted there were 300-400 more games this time around, I spent a lot of time paring the individual game synopsis files to 1kb or under and ended up with a total file size of around 1.6MB, so I was shocked to see we're still at 3.00MB for the NES gamelist.xml. (But then I did remember that we've got a LOT of long file names/paths on each entry that weren't in the synopsis).

I know it probably doesn't sound like we'd shave much off of that size by eliminating empty tags, but remember that at least for the NES, there are over 2,020 games included, as well as around 30 folders.

Maybe this won't be an issue when I finally get a Pi3, but it seems like having massive gamelist.xml files impacts performance on the Pi Zero. Fortunately, there aren't any problems so far with just this NES gamelist, but if I were to have FULL gamelists for all of the systems one day, I'm thinking they wouldn't run on the Pi Zero at all.

Also, removing the website address at the bottom of each entry should clear up some space too. :)

Used2BeRX

@meleu Sorting by Genre in RetroPie is pretty awesome man. It's going to help me fine-tune the final genres for the sets. I actually had no idea how many different ones I was using until I tried this feature out. There's so many combinations of them. (198!)

At least with this feature I can see if there are any that I put in a different order on two separate games, or if I called it something slightly different.

For example, VS Fighter / VS Fighting, Board Game / Board Games, Card Game / Card Games.

lol... just noticed I have "Actoin / Platformer". Might have a few misspellings in there. :)

Well... scrolling through them, I think I should be able to eliminate 30 of the genres pretty easily. 15-20 of them were probably going to be gone by next re-run of the script anyhow after my Pi testing since I've been heavily editing the Genre and Players fields as I've been going through the testing process.

BTW... I've tested over half of the NES games on the Pi Zero so far, and the results are really good. Getting into the more obscure territory right now, but we're still rocking over 99% compatibility. :)

Used2BeRX

@meleu Hey Meleu,

Is there a command I could run that would create an output text file for me that has every different line used in the Genre: line of all of the synopsis.txt files? (Only once though, I don't need to see all 300 times that "Action / Platformer" shows up.)

It's helpful to have that information to scroll through on the RetroPie screen, but I'd like to actually look at a text file of it so I can "sort" some things and better re-name the Genre fields that way.

I thought you'd know a quicker way to do this than me typing all of them out looking at my TV screen. :)

mitu

@used2berx Assuming your files are all .txt in a single folder, execute the following command from within that folder:

for i in *.txt; do sed -ne '/Genre:/ s/Genre:\(\s\)//p' "$i"  >> /tmp/genres.txt; done

to get all Genres in a single /tmp/genres.txt. To sort and eliminate duplicates, execute:

cat /tmp/genres.txt | sort | unix > sorted.txt

and the sorted.txt file should have a sorted and no-duplicate list of all the genres.

EDIT: corrected the for loop, it always looked inside 1 file.

Used2BeRX

@mitu Thanks man. I'm going to give that a try and see how it works out. :)