Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu Sure. These are only the ones that were done years ago. My hope is to add more console/handheld systems in time. I don't think I'll be adding any computer based systems though since they require a lot of individual controller configurations and would be a monsterous project on its own. Although I have done some work with DOSBox myself and I know another guy who is also working on that, and there has been an amazing Commodore 64 collection with the controller configs, but that would be a LONG way down the road.

Here's the current list:
Atari 2600
Atari 5200
Atari 7800
Atari 800 * See note below
Atari Lynx
Bandai Wonderswan
ColecoVision
Magnavox Odyssey2
Neo Geo Pocket
Neo Geo Pocket Color
Nintendo 64
Nintendo Entertainment System
Nintendo Game Boy
Nintendo Game Boy Advance
Nintendo Game Boy Color
Nintendo Virtual Boy
TurboGrafx-16 / PC Engine * I didn't separate the CD games for some reason... I will in the future.
Sega 32X
Sega CD / Mega CD
Sega Game Gear
Sega Genesis / Mega Drive
Sega Master System
Sega SG-1000
Sony Playstation
Super Nintendo Entertainment System
Watara Supervision

I believe that's all of them that I had done in the past. There were a few other meaningless ones like the Dreamcast VMU unit and Pokemon Mini games that aren't even emulated on the Pi that I probably wouldn't bother with.

BTW... With the Atari 800, could you do the same thing we're doing with the NES/FDS systems, so that if the Atari 800 stuff is located where the Atari 5200 stuff is it will work with the 5200? I have no interest in doing all the work on the Atari 800 computer since I never owned one and it would be a ton of controller configs to make, but I figured I'd probably put a smaller collection of 800 games in there that I found that were quite enjoyable that I've added to the XBox set years back. This way if anybody were to do the Atari 800 themselves someday this script could still work for them.

Used2BeRX

Also, I was wondering if we could add something else to the to-do list. I wanted to know if we could add a tag that is the synopsis.txt file name without the extension (txt). We could just make the tag <filename></filename>.

This might have several uses down the road since this is the exact file name of all of the associated media for every game. Right now it would be great for when I'm working on the spreadsheet as a way of alphabetizing stuff. When we get all the tags in our script with the FULL command, I hope to talk to @mitu about an option in his own script that would transfer over all of the tags that we're using. Otherwise, when we go through the process of turning the spreadsheets all the way back to text files there will be a lot of missing information.

BTW... I'm also going to try to see if somebody on the team here could add more image tags for us to use. And even possibly implement something where you could press a certain button on the controller to scroll through the existing image types. I think it would be awesome if we could have a video preview and an image box that you could scroll through the Boxart, Cart art and Title screen of all of the games.

Right now we only have the <image> and <marquee> tags. If we were to have tags for the tags you've already added like <boxart> to your script, it would eliminate the need to be able to change which image type is the <image> and not require a re-run of the script just to change the image type in the romlist.

Thanks again man! Excited to see you're back in any capacity. :)

meleu

@used2berx When a tag in gamelist.xml is empty or inexistent, what do you want to put on the synopsis.txt file?

Example: if there's no <originaltitle> in the gamelist.xml, do you want an empty Original Title: on synopsis.txt or simply skip this info?

Used2BeRX

@meleu Good question. Thanks for asking since I didn't think of that before. :)

Any empty tags should just not be put in the synopsis.txt files. This is because on the XBox they are displayed "as-is", so if there were a bunch of empty fields it would look messy and would take more scrolling to get through the whole thing.

You probably also thought of it, but any media tags we make for the locations in the gamelist.xml file should also not make their way back into the synopsis files. (No artwork, videos, gamefaqs, etc.)

Essentially, the only tags that should make their way back into the synopsis are the ones that were there originally. The only exception would be data/information tags that had information that I added in the spreadsheet before converting back to the gamelist.xml. For instance, if I realized in the spreadsheet that a game was missing the Publisher: information and I filled it in the spreadsheet, then that information would then make it back into the synopsis file.

Now that I think about it, I'm wondering if we should make a separate tag for the synopsis cited website at the end of the <desc> tag? This way we could still have that information in the synopsis.txt, gamelist.xml and the spreadsheet, but we don't have to take extra steps to remove it so it doesn't show up in the RetroPie romlists. The only thing we'd need to do is make sure that they get placed back at the end of the <desc> after two line breaks when it's covnerted back to the synopsis.txt.

Maybe call it <citation>?

meleu

@used2berx

Maybe call it <citation>?

Do as you prefer.

On the code side it would be easy to consider that "if the very last line of a synopsis.txt is an URL, then it's a <citation>".

meleu

@used2berx said in Making premade gamelist.xml with xtra media collections:

You probably also thought of it, but any media tags we make for the locations in the gamelist.xml file should also not make their way back into the synopsis files. (No artwork, videos, gamefaqs, etc.)

Please, confirm if these are the ones to be ignored:

path
image
video
marquee
xtrasname
cart
title
action
threedbox
gamefaq
manual
vgmap

Used2BeRX

@meleu

On the code side it would be easy to consider that "if the very last line of a synopsis.txt is an URL, then it's a <citation>".

That will work. :)

Please, confirm if these are the ones to be ignored:

path
image
video
marquee
xtrasname
cart
title
action
threedbox
gamefaq
manual
vgmap

Yes. That is all correct.

Used2BeRX

@meleu

Hey man. I'm not getting any traction on my thread about trying to get more image types supported in RetroPie. Do you know who I should hit up for that idea?

The thread is here: https://retropie.org.uk/forum/topic/16535/multiple-image-types-tags/2

Thanks :)

Used2BeRX

@meleu

I'm not sure how linux scripting works, but I'm assuming after we run our desired command to start the script that we could have it ask a few questions for our image setup? I just found out from the other thread today that we currently have access to 3 image types.

<image>
<marquee>
<thumbnail>

The thumbnail tag is used to show a screenshot before a video plays if you set a delay. @EctoOne actually uses this in such a way that the video isn't played at all and it acts as a third image type. This would be perfect for the Pi Zero setup that I'm building that won't have videos.

With this, there is actually currently coverage for Box Front, Cart and Action images.

So what I'm asking if we could do is two things...

Make a tag for every image type. (The default script now doesn't make <boxfront> because it makes the box <image> instead. I would like to see this become it's own tag... I'll explain why later.)
Ask a few questions about image types after desired script command is run, but before the script is executed. These following questions:

Image type for <image> ?

Box Front
Cart
Title
Action
3D Box

Image type for <marquee> ?

Box Front
Cart
Title
Action
3D Box

Image type for <thumbnail> ?

Box Front
Cart
Title
Action
3D Box

If we do it this way and leave the regular image tags we decided on for the image type, this should be very easy to change them in the script (or a separate script) in the future. That way we could run it and just swap out the locations for different image types without having to rebuild the entire gamelist from scratch. The image changing script could ask the same 3 questions and then just replace those tag lines with the image location of the desired art type. :)

Thanks bud... sorry to change this on you like that, but I think this is the best way to go forward with this new image information we just got.

Used2BeRX

Also... Below is an example of one of the game entries the script currently makes. I've highlighted where in the entries the new tags should be generated for ease of viewing/usage.

Along with the image tags, there are a few other ones that we've discussed from time to time in here along with some notes about them.

<game>
<name>American Gladiators</name>
<path>/home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/American Gladiators.zip</path>
<image>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</image>
<video/>
<marquee/>  `(This was already in the script)`
<thumbnail/> (**New addition**)
<desc> [Body text removed to simplify this post]</desc>
<link>https://www.gamefaqs.com/nes/563382-american-gladiators</link>  (**Our new link code**)
<releasedate>19910101T000000</releasedate>
<releasetext>1991</releasetext> (**Our text based release year from the synopsis `Release Year: ` line**)
<hackreleasedate/> (**New addition**)
<transreleasedate/> (**New addition**)
<developer>Incredible Technologies, Inc.</developer>
<publisher>GameTek, Inc.</publisher>
<genre>Game Show / Competitive Events</genre>
<players>2</players>
<playerstext>1 to 2 VS</playerstext> (**Need this to properly convert back to synopsis after edits.  Just a pure rip of the synopsis information without your code for converting it to an ES friendly field**)
<region>USA</region>
<platform>nes</platform>
<media>Cartridge</media>
<controller>NES Gamepad</controller>
<gametype>Licensed</gametype>
<xtrasname>American Gladiators</xtrasname>
<originaltitle/>
<alternatetitle/>
<hackedby/>
<translatedby/>
<version/>
<boxfront>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</boxfront>  (**Same as <image>.  Our code defaults <boxfront> to <image>, but this should also be a separate line for scripting quick changes to image types on a script re-run**)
<cart>/home/pi/RetroPie/Media/nes/Artwork/Cart/American Gladiators.jpg</cart>
<title>/home/pi/RetroPie/Media/nes/Artwork/Titles/American Gladiators.jpg</title>
<action>/home/pi/RetroPie/Media/nes/Artwork/Action/American Gladiators.jpg</action>
<threedbox>/home/pi/RetroPie/Media/nes/Artwork/3D Boxart/American Gladiators.png</threedbox>
<gamefaq/>
<manual/>
<vgmap/>
<license/>
<programmer/>
<musician/>
</game>

I can't think of anything more right now that we were talking about. I hope this all makes sense. Please let me know if I need to clarify anything. :)

Used2BeRX

@meleu I forgot about the "real date" tag. I added that to the example I gave above. :)

EDIT: I also forgot about two tags we haven't even discussed yet.

In the synopsis for both hacks and translations I have lines Translation Release Year: and Hack Release Year: as well as the original game release year.

I've added both tags to the post above: <hackreleasedate/> and <transreleasedate/>

Thanks :)

Used2BeRX

@meleu

Hey man. I mis-remembered our prior conversations and just found out that the FDS games aren't being built with the NES gamelist.xml. I was hoping you might have some time to work on that because now that I'm done with the cart artwork I've re-run the gamelist script and I can't verify any of the FDS stuff is working and the artwork is good. I'm considering going back into the FDS synopsis.txt files and changing the system name back to Nintendo Entertainment System just so I can finally get these things working if it's too difficult to make the code do this part.

Let me know.

Used2BeRX

@meleu

Thanks for adding the option to remove the FULL out of the gamelist.xml files man. That feature works flawlessly and will save me a lot of time renaming files down the road. :)

Used2BeRX

@meleu Hey bud. You alright man?

I think I found a bug in the ointment. It seems that when scraping the synopsis files, the script is not grabbing the body text for all of the games.

I found this out a few weeks ago, but I haven't brought it up since you haven't replied for so long. Just making sure that you're still around is all. :)

meleu

@used2berx hello friend.

I'm sorry to say I'm moving from this code. I started to work on this to work out my skills to manage data on text and xml files (basically improving my sed and xmlstarlet skills) and it was a great exercise. I'm glad to make something useful for someone during this exercise, but now I'm quite busy with other projects and don't have enough free time to maintain this script.

If you find another shell programmer, tell him/her that the script is always available at that github repo. And if they have questions about the code I can try to answer at some time. I can also try to review others code if they submit Pull Requests to that repo.

Cheers, and take care!

Used2BeRX

@meleu Well... that's going to put a huge damper on any future Rx releases for the RetroPie scene....

Bummer.

Maybe somebody else will help out down the road.

Good luck with whatever you've got going on man. Thanks for the work so far.

Used2BeRX

@meleu I had some time to sit down with the synopsis files and I believe that I figured out the common theme to any of the entries where the body text wasn't filled out with your script. In Notepad++, the ones that don't work right are listed as an "ANSI" file. All of the ones that work that I've checked seem to be listed as "UTF-8".

I don't know much about this other than they're two forms of character encoding and that UTF-8 is far more advanced than ANSI was. I have no clue how some of these files were saved differently. I'm going to try re-saving all of the non-working files and see if I can somehow save them as UTF-8.

Just wanted to let you know what that bug was.

EDIT: Yep. I believe that's what the problem is. In Notepad++ I can go to the "Encoding" menu and then select the second option "Encode in UTF-8". Some characters, like alternate apostrophes and quotes then change to weird things like "x92" and are highlighted. I can then just select them and put in a regular character and save it.

I'm going to do this to about a dozen of the bad ones and re-run the script before I sleep. They should all be working correctly with the new gamelist.xml.

Used2BeRX

@used2berx Awesome. That worked. Now I just need to go through and figure out which files had the ANSI format and convert them all to UTF-8 and the game descriptions will all be good. :)

Used2BeRX

Okay. Some weird things were happening. Lot's of characters that were in the synopsis files had bad characters that showed up as a floating box in both the XBox and the RetroPie. RP seems to have been able to display more of them though. What I can't figure out is why some files were ANSI when I opened them in Notepad++ and others were UTF-8, even though they had the same bad characters. Once they were fixed and saved though, they were automatically UTF-8 and I couldn't even manually save them as ANSI if I wanted to (which I didn't, but I was just testing it out).

I'm glad this issue came up now, actually. For years it has always bothered me that the strange characters looked so lousy on the XBox. The fact that they were actually breaking the script meant that I had to do something about them finally. I'll know for sure if I got them all after the new gamelist.xml is generated, but I think it's pretty safe to say that all 2,116 NES/FDS synopsis files are clean now. :)

I'm working with somebody to hopefully make a script to automate this process for future systems. I don't like automation unless I know exactly what it's doing, but after having spent 8 hours with this last night I feel pretty comfortable with it as long as it does exactly what I need it to do... which is the following:

Change all characters on the left side to the ones on the right side of the equal sign:

à = a
ā = a
é = e
ñ = n
ṇ = n
ø = o
ō = o
ǚ = u
ū = u

™ = Nothing (remove it entirely)

– = -
— = -
‘ = '
’ = '
´ = '
… = ...
“ = "
” = "

That's it.

There may be a few different characters that I find here and there as I move on to other systems down the line, but they could be added to the automation process as I discover them. This list should take care of 99% of any future issues. :)

Oh... and as a bonus, this also makes the gamelist.xml slightly smaller since every one of these characters were a long series of numbers in the XML files such as &#x2013 and &#x201D. It won't really make much of a difference on an XML file that is over 3.3MB, but every little bit helps... especially on the Pi Zero.

Those numbers are actually how I found all of these problems. I just did a search in Notepad++ on the last generated gamelist.xml for &#x and I found all of those one-off accented letters that I never would have found using my eyeballs. :)

Used2BeRX

Thanks to my bud Nemesis I've learned a lot about the code in this script. I'm no coder, but using existing code I was able to make a few changes so far that I wanted in this script. I'm re-running it now, and all of the desired changes I made this time around are taking effect. :)

I was able to add code to add and find all data for Famicom Disk System games that are in NES subfolders without having to have manually changed the "Platform: " line to erroneously read "Nintendo Entertainment System" instead of the legitimate tag of "Famicom Disk System".

I was also able to add a <boxfront> tag as a placeholder for all Box Front images, so now we have one for Box Front, Cart, Titles, Action and 3D Boxart images.

I was also able to make the code put the Cart images in the <marquee> tag, so now a skin that uses both tags can display both the Box art and the Cartridge art simultaneously.

The next thing I want to do is a bit trickier. I've heard from somebody here that if you don't have a video for a game, the <video> tag can be used as a third image type. I'd like to add some code in there that will search for the video as it is already doing, but if it doesn't, it will add either the Title image or the Action image shots in that tag.

Once all this is taken care of, I'm hoping Nemesis and I can come up with a quick script that will change which images populate the <image>, <marquee> and <video> tags on the fly after the gamelist.xml is created. Right now, on the Pi Zero it takes about 3 hours for the script to run on the NES alone. But after it is created it should only take a few seconds to run a script that will change which folder each image tag points to for every game.

Another thing on the list now is to add every single tag from the synopsis.txt files to the gamelist.xml file so when I convert them and then use @mitu 's script to convert it to a spreadsheet, I can make all the edits I want in the spreadsheet and then convert it back to a gamelist.xml file and finally back to the synopsis.txt files without losing any of the original data.

This is exciting. If we can get this working it's going to make editing game information for both the XBox and the RetroPie easier than it ever has been before.

Thanks again for all that hard work @meleu