Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu Hey man. Sorry I haven't gotten back to you with a re-test yet. Things have been really busy for a while in RL. I've had a little time this week to get some major things done and I'm doing a lot of updates to the synopsis files for the NES.

I was wondering if I could ask for another request before I get around to a re-test. I was hoping that we could add an option to leave out the website information in the gamelist.xml entries. You probably know that every synopsis entry contains the link where the info came from. (Even better would probably be to leave the website out by default and only add it if you use an option to add them).

I don't believe that there is any way that this information could be beneficial for the end user. Maybe if there was some sort of brower functionality added to EmulationStation one day it would be cool to be able to open the links, but until then I think it would be a better presentation to leave this information out.

I like having the info in all of the files, so I was hoping that we could add some code that would eliminate the links. Maybe the best way to do it would be to finish copying the body of each file once a link has been encountered?

Thanks again man. I can't wait to try this out again once I get all of these files up to date and resize the artwork and give it a shot.

meleu

@used2berx can you provide an example?

Used2BeRX

@meleu Sure thing. :)

Murder on the Mississippi.txt

Murder on the Mississippi
Original Title: Mississippi Satsujin Jiken
Platform: Nintendo Entertainment System
Region: Translated (Japan)
Media: Cartridge
Controller: NES Gamepad
Genre: Text Adventure
Gametype: Translated
Release Year: 2017
Translated By: GAFF Translations
Version: 1.00
Players: 1
_________________________

The famous detective Sir Charles Foxworth is taking a holiday on the Mississippi with his servant Regis on the Delta Princess. He's just taking a stroll around the upper deck of the ship when he discovers an open cabin door. Behind it: a dead man. As it turns out, the body of Raleigh Cartwright III. Sir Charles has three days to find the murderer, because then, the ship will reach New Orleans, giving the culprit a safe chance to escape.

The joystick controls the player's sprite to move into the respective direction, and pressing the button opens a menu with appropriate commands for the current situation. Walk around, Inspect an item, or examine evidence.

You will meet some characters who also happen to be suspects. Use these encounters as a fact-finding session. Take notes on what he or she has said. Share these notes with other passengers and crew members. But your first job is to find the body. Until you do, any information you gather will be of little value. Once you find the body, the other passengers will prove to be much more talkative.

Collect as many notes and evidence as you can to solve the mystery. Once you think that you have it solved, make an accusation by going to the character that you want to accuse and by using the joystick, point to Talk to [character's name], and press the joystick button. Then point to Accuse [character's name] and press the joystick button. Depending on the amount of evidence collected, you will know if your hunch was right or wrong.

Murder on the Mississippi has four possible endings with many variations on each one. Don't let the Delta Princess arrive to New Orleans before you solve the mystery, or the murderer will get away.

http://www.romhacking.net/translations/3133/
http://www.mobygames.com/game/nes/murder-on-the-mississippi

meleu

@used2berx removing any line with only an URL is OK for you?

Used2BeRX

@meleu That would be perfect.

At one point my synopsis files would contain links on occasion inside of the body. This was generally for hacks and translations that would have addendum patches to them. I'm in the process of updating all of these files and that type of information won't be in the synopsis anymore. If somebody wants to read any of that, that's what the links are for ;)

The new synopsis files I'm working on will be short, sweet and to the point. They will talk about the game and not about patch information.

That being said, there will probably be weird issues on some of the files for the non-NES systems that haven't been updated, but I'll get to everything eventually. :)

Used2BeRX

@meleu Hey buddy.

I was wondering if there was any way to revert the gamelist.xml back into the individual synopsis.txt files?

I was thinking that it would be cool to be able to take the gamelist.xml file and put it all into a spreadsheet so I could do some serious editing of individual cells, so I could have uniform names for all of the different game genres, for instance.

I haven't figured out any good way to convert my synopsis.txt files into a spreadsheet, but I'm thinking that I should have much better luck doing that with the XML file. I thought it would be a lot easier for me to do that and edit them that way rather than individually going through about 2,000 games. I was also thinking that this would make my work MUCH easier when I finally start on other systems to have a way to easily edit and convert these things to the 3 different formats. (synopsis.txt, gamelist.xml and a spreadsheet).

I'd, of course, have to figure out how to convert a spreadsheet back to a gamelist.xml file before converting them back to synopsis.txt as well.

I'm just kind of spit-balling ideas here since I'm not really good at this type of stuff at all. I've spent a ridiculous amount of hours doing most of this work by hand individually for all the games, but a system like this could eventually save me thousands of hours once I get around to all of the other systems. I'm kind of in the process of learning "what not to do" in the future.

I'm open to any ideas you have, or anybody else who has a good idea.

BTW... I've also added a few more categories for the synopsis files. For instance, the Translated Games now have a Release Year: (Original Game Release Year) and a Translation Release Year:. I now have synopsis information on the Translations that reflects both the translation itself as well as the original game information. They also show the Developer: and Publisher: of the original game as well as the Translated By: field.

I'm in the process of "upgrading" around 300-400 hacks to do the same. They'll have a Hack Release Year: on them.

I've also added Players: information for every Translation and Hack in the set by verifying the players on each game individually as I was testing them.

Lot's of work being done behind the scenes man. I still don't want to make the spreadsheet public, but if you have the link for it still you should check it out. Lots of big updates reflected there, and it should be a lot more self-explanatory than it was the last time you looked.

Take it easy. ;)

meleu

@used2berx gamelist.xml to synopsis.txt can be done. But converting to/from a spreadsheet is not that easy. But if you are able to edit xml files on Excel or other spreadsheet program, then probably you're also able to save your modifications on a xml file. Aren't you?

When I find some time I'll try to make that gamelist.xml to synopsis.txt feature.

Cheers!

Used2BeRX

@meleu Well, the reason I wanted to do it in a spreadsheet is because then for any different field I could alphabetaize them and go through and make any fixes for any typos and stuff. I keep using the Genre as an example because it really needs the work. Sometimes it's RPG and other times it's Role Playing Game. Sometimes it's Sports - Basketball, other times it's Sports / Basketball.

If I was somehow able to get them all into a spreadsheet to edit, and then somehow revert them, it would be much easier to do this editing.

Looking at them in the romlist on the Pi will be able to help a little, but the problem there is that about 1/2 of the fields in my synopsis files aren't yet implemented at all in EmulationStation so I can't check them in there.

meleu

@used2berx sometimes I feel like if you were my neighboor we would solve all this editing stuff that takes hundreds of hours of your time in an afternoon. :-)

Basic regular expression and sed knowledge would help you a lot!

Used2BeRX

@meleu You're probably right.

I was never very good at studying. I tried to get into computer programming but it was such a drag.

Funny thing is I can literally spend hundreds or even thousands of hours doing stuff by hand rather than learn how to code things. It soothes my OCD itch. There's not many things I'd less rather do than try to learn from a book. Maybe if I had a good teacher it would be a different story.

I've got to do something about this problem here though. There's only 2,000 games in the NES compared to all of them in the other systems. I'm getting to the age where I don't know if I have enough time left on earth to get it all done if I don't start doing a few things differently.

Used2BeRX

@meleu Hey Meleu...

You don't know of any program or an easy way I could input the gamelist.xmlfile and be able to sort the values inside like a spreadsheet or like in MS Access, do you?

Maybe I'm making this more complicated than it needs to be.

EmulationStation itself already does this for any of the fields that are displayed in the romlist/skins. Maybe there's another simple program out there that can just display all of the info if I tell it what fields to look for without actually launching any roms or anything?

meleu

@used2berx sorry, I didn't get what you want exactly. Could you give an example?

Used2BeRX

@meleu I don't have the Pi available right now, but I recall that somewhere in the romlists you can choose to sort by different fields, such as Genre or # of players.

I'd be looking for some sort of way to be able to do this for all of the fields... even the ones not currently supported by ES.

It would be ideal if I could actually edit the XML from wherever I was seeing this sorted data because I'd be able to convert it back to synopsis.txt files if you made that script. But even if I can't still edit this info from here I'd at least know which synopsis.txt files I needed to go in and edit.

Used2BeRX

@meleu Hey man. I've been all over the map with this project, but my focus is now back to the synopsis. I've still got quite a few more dozens of hours work to put into it before I'm ready to start messing with the script again, but I was wondering if you'd have some time to work with me on a few things in the future. I realize the holidays are coming up soon and that might not be possible.

I'll try to organize my thoughts here and get everything I was thinking of in the same post.

Just so I don't forget, another thing I was hoping that we could do is use the Players: field in the synopsis.txt files to create two tags in the gamelist.xml. The first would be our standard <players> tag that uses your code to only show the maximum amount of players in the game, but also a <players-full> tag that would show the text string that I put on that line. Personally I prefer the way that we've displayed that on the XBox and I'm hoping that if this is built on my end I might one day be able to get somebody here to add that possible tag to the setup here. I know it wouldn't be usable for anything right now. Maybe we could just throw that in the "full" option and leave it out when we don't use that.

Used2BeRX

@meleu I had a Linux question for you Meleu.

I think what I want to do would be much easier in Linux than in Notepad++

In re-doing all of the synopsis entries, I have been using the / character to separate the genre types, if more than one, in an individual game. Just the other day though, I thought using that character might cause errors in the gamelist.xml (I'm not sure about that). But even if it doesn't, I think I'd rather use the | character instead for separating the genres since it looks better. (Not sure if that would cause any errors either).

So...... Two questions, actually. Would / or | cause any problems in the gamelist.xml file?

And, is there a command or simple script I can run that would only change the / to | in instances where / is located in the Genre: line?

For example, changing any instances of the following:

Genre: Arcade / Action / Platform

To:

Genre: Arcade | Action | Platform

Thanks!

meleu

@used2berx hey bro. The tool to do this magic is sed. I know it isn't the most intuitive tool, but I'm sure that it's the perfect tool to do the kind of work you do.

Oh! Here are the magic words you need:

sed -i '/^Genre:/ s#/#|#g' file.txt

If you want to apply this to all your .txt files you can use *.txt rather than file.txt.

meleu

@Used2BeRX I think I'll have some free time to play with bash code in the next couple of days. Can you update your "wishlist"?

Here is the last one I've found in a post above, and I think I already implemented many of those things:

Create the backup folder if it doesn't exist, and backup any gamelist.xml file before creating a new one if one of the append options isn't used on an exisiting gamelist.

Done in this commit.

Make option to change which artwork tag is set as <image> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made.

This item still is in the ToDo list. :)

Platform: Nintendo Entertainment System and Platform: Famicom Disk System entries in the synopsis.txt files located in /Media/nes/Synopsis/ folder will treat these entries as if they were NES and not FDS games and look for roms and all media in the /nes/ folders instead.

Done in this commit.

Add <marquee> tag to both regular and FULL operations, looking for the media in /Media/[System]/Marquee/ folders. (Search for both .JPG and .PNG).

Done in this commit.

IIRC another thing you asked was to remove any line with only an URL in the description. ~~It's in the ToDo list also.~~ EDIT: Done in this commit.

If you have more to request, add comments here:
https://github.com/meleu/Used2BeTXT/issues/1

I can't assure I will do them, but from time to time I like spend my free time with coding rather than playing videogames. :D

Used2BeRX

@meleu Thanks for the command above :)

I'm going to probably be trying that out by sometime next week. I'll be sure to do it on a backup set of files just so I don't mess anything up. I bet there is a lot of other cool stuff that I can do with that command that can really help me in the future as well. Notepad++ can actually do a fairly decent job of bulk "find and replace", but it's still too limited for what I want to do and I remembered you said something about this command before.

I should have some time in the next 2 days to sit down and figure out what I want to do and update my wishlist. I know you hit a lot of the important things there, but there's so many posts on this thread now and so much time has gone by while I was doing other things that I don't even remember what all I wanted to see happen.

The only thing I would add to what you put above was an extension of Numbers 2 and 4 above:

Make option to change which artwork tag is set as <image> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made.

Add <marquee> tag to both regular and FULL operations, looking for the media in /Media/[System]/Marquee/ folders. (Search for both .JPG and .PNG).

If you've already done both of these, it would be sweet to be able to do the same thing we did to the <image> tag in number 2, to the <marquee> tag in number 4.

Currently, we are able to display two images per game since the introduction of the <marquee> tag. This doesn't necessarily have to be a Marquee image though.

If we are able to quickly change which type of images we want to see on the fly with your script, this would open up a lot of different options. Maybe somebody would like to display an image of the Box and the Cart? Maybe somebody else would like to display the Box and a title shot? Or the Box and an action shot?

Personally, I'm probably going to go with two setups, one for the Pi Zero and one for the Pi3. Because I'll have videos that will work with the Pi3, it would be nice to also have a Box and Cart image for all of the games with it. But the videos won't work on the Pi Zero, so it would be nice to have either the Box and the Title or the Box and the Action shot.

Thanks so much man. I'll get an updated wishlist to you if there was anything else missing and I'll add them all to the link you sent me. :)

Used2BeRX

@meleu Hey bud :)

I believe I've found everything I requested before that you haven't put in there and everything that I can currently think of at the moment. I might have more to add to the wishlist after I finally get to running this thing again and kicking the tires a bit, but for now I've added the stuff to the wishlist page you provided. I've bookmarked it too so I can put the stuff up there and we could keep track of it. I should probably at some point add all of the things you've already done there so we can have a list of everything the script is capable of.

I put everything in my order of personal importance. I'll post them again here for ya now too. Please let me know if you have any questions about anything I'm requesting, and if you want me to re-word or simplify any of it on your checklist page.

HUGE THANKS AGAIN!!!

This is going to be awesome. :)

Synopsis script wishlist:

Platform: Nintendo Entertainment System and Platform: Famicom Disk System entries in the synopsis.txt files located in /Media/nes/Synopsis/ folder will treat these entries as if they were NES and not FDS games and look for roms and all media in the /nes/ folders instead. (This should not occur if the synopsis, media and roms are located in the /Media/fds/ directory though).
Make option to change which artwork tag is set as <marquee> so you can switch between Box Front, Cart, Titles, Action and 3D Boxarts types after the gamelists are already made. (Same as what I believe you already did for the <image> tag). This way we can easily change what type of image displays for two image types after the gamelist.xml is created.

I'm going to add "dummy" *.jpg files in the Marquee folder since I don't actually have any marquee images, unless you have a better solution for this. I think right now the script just wouldn't create <marquee> tags at all, so this request wouldn't actually do anything unless the original run of the script created those lines in the gamelist.xml

Add the following fields to the Full option:

Translation Release Year:
Translated By:
Hack Release Year:
Hacked By:
Original Title:
Alternate Title:

These aren't always used in every synopsis, so it should only create these in the entries that have them in the synopsis.txt file

Use the Players: field in the synopsis.txt files to create two tags in the gamelist.xml. The first would be our standard <players> tag that uses your code to only show the maximum amount of players in the game, but also a <players-full> tag that would show the text string that I put on that line. Personally I prefer the way that we've displayed that on the XBox and I'm hoping that if this is built on my end I might one day be able to get somebody here to add that possible tag to the setup here. I know it wouldn't be usable for anything right now. Maybe we could just throw that in the "full" option and leave it out when we don't use that.
gamelist.xml to synopsis.txt reverse-order feature.

Used2BeRX

Oh.... BTW....

I've been in the middle of re-writing the synopsis files so they work better with RetroPie. (Not only because of memory constraints, but nobody wants to read a slowly scrolling novel for a game description either). You've probably read that I'm doing anything necessary to reduce the file size for each game to only 1kb or less. Sometimes, this requires me to re-word some of the descriptions, but I'm taking extra care to make them still be great descriptions for each game.

The original file size for all NES synopsis files was somewhere around 3.40MB for only 1,700 or so games. Right now, the file size is 1.79MB for 2,020 games and shrinking. :)

Can't wait to see what this all looks like after re-running your script when the edits are done!