Making premade gamelist.xml with xtra media collections

Used2BeRX

Hey man. I'm not getting any traction on my thread about trying to get more image types supported in RetroPie. Do you know who I should hit up for that idea?

The thread is here: https://retropie.org.uk/forum/topic/16535/multiple-image-types-tags/2

Thanks :)

Used2BeRX

@meleu

I'm not sure how linux scripting works, but I'm assuming after we run our desired command to start the script that we could have it ask a few questions for our image setup? I just found out from the other thread today that we currently have access to 3 image types.

<image>
<marquee>
<thumbnail>

The thumbnail tag is used to show a screenshot before a video plays if you set a delay. @EctoOne actually uses this in such a way that the video isn't played at all and it acts as a third image type. This would be perfect for the Pi Zero setup that I'm building that won't have videos.

With this, there is actually currently coverage for Box Front, Cart and Action images.

So what I'm asking if we could do is two things...

Make a tag for every image type. (The default script now doesn't make <boxfront> because it makes the box <image> instead. I would like to see this become it's own tag... I'll explain why later.)
Ask a few questions about image types after desired script command is run, but before the script is executed. These following questions:

Image type for <image> ?

Box Front
Cart
Title
Action
3D Box

Image type for <marquee> ?

Box Front
Cart
Title
Action
3D Box

Image type for <thumbnail> ?

Box Front
Cart
Title
Action
3D Box

If we do it this way and leave the regular image tags we decided on for the image type, this should be very easy to change them in the script (or a separate script) in the future. That way we could run it and just swap out the locations for different image types without having to rebuild the entire gamelist from scratch. The image changing script could ask the same 3 questions and then just replace those tag lines with the image location of the desired art type. :)

Thanks bud... sorry to change this on you like that, but I think this is the best way to go forward with this new image information we just got.

Used2BeRX

Also... Below is an example of one of the game entries the script currently makes. I've highlighted where in the entries the new tags should be generated for ease of viewing/usage.

Along with the image tags, there are a few other ones that we've discussed from time to time in here along with some notes about them.

<game>
<name>American Gladiators</name>
<path>/home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/American Gladiators.zip</path>
<image>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</image>
<video/>
<marquee/>  `(This was already in the script)`
<thumbnail/> (**New addition**)
<desc> [Body text removed to simplify this post]</desc>
<link>https://www.gamefaqs.com/nes/563382-american-gladiators</link>  (**Our new link code**)
<releasedate>19910101T000000</releasedate>
<releasetext>1991</releasetext> (**Our text based release year from the synopsis `Release Year: ` line**)
<hackreleasedate/> (**New addition**)
<transreleasedate/> (**New addition**)
<developer>Incredible Technologies, Inc.</developer>
<publisher>GameTek, Inc.</publisher>
<genre>Game Show / Competitive Events</genre>
<players>2</players>
<playerstext>1 to 2 VS</playerstext> (**Need this to properly convert back to synopsis after edits.  Just a pure rip of the synopsis information without your code for converting it to an ES friendly field**)
<region>USA</region>
<platform>nes</platform>
<media>Cartridge</media>
<controller>NES Gamepad</controller>
<gametype>Licensed</gametype>
<xtrasname>American Gladiators</xtrasname>
<originaltitle/>
<alternatetitle/>
<hackedby/>
<translatedby/>
<version/>
<boxfront>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</boxfront>  (**Same as <image>.  Our code defaults <boxfront> to <image>, but this should also be a separate line for scripting quick changes to image types on a script re-run**)
<cart>/home/pi/RetroPie/Media/nes/Artwork/Cart/American Gladiators.jpg</cart>
<title>/home/pi/RetroPie/Media/nes/Artwork/Titles/American Gladiators.jpg</title>
<action>/home/pi/RetroPie/Media/nes/Artwork/Action/American Gladiators.jpg</action>
<threedbox>/home/pi/RetroPie/Media/nes/Artwork/3D Boxart/American Gladiators.png</threedbox>
<gamefaq/>
<manual/>
<vgmap/>
<license/>
<programmer/>
<musician/>
</game>

I can't think of anything more right now that we were talking about. I hope this all makes sense. Please let me know if I need to clarify anything. :)

Used2BeRX

@meleu I forgot about the "real date" tag. I added that to the example I gave above. :)

EDIT: I also forgot about two tags we haven't even discussed yet.

In the synopsis for both hacks and translations I have lines Translation Release Year: and Hack Release Year: as well as the original game release year.

I've added both tags to the post above: <hackreleasedate/> and <transreleasedate/>

Thanks :)

Used2BeRX

@meleu

Hey man. I mis-remembered our prior conversations and just found out that the FDS games aren't being built with the NES gamelist.xml. I was hoping you might have some time to work on that because now that I'm done with the cart artwork I've re-run the gamelist script and I can't verify any of the FDS stuff is working and the artwork is good. I'm considering going back into the FDS synopsis.txt files and changing the system name back to Nintendo Entertainment System just so I can finally get these things working if it's too difficult to make the code do this part.

Let me know.

Used2BeRX

@meleu

Thanks for adding the option to remove the FULL out of the gamelist.xml files man. That feature works flawlessly and will save me a lot of time renaming files down the road. :)

Used2BeRX

@meleu Hey bud. You alright man?

I think I found a bug in the ointment. It seems that when scraping the synopsis files, the script is not grabbing the body text for all of the games.

I found this out a few weeks ago, but I haven't brought it up since you haven't replied for so long. Just making sure that you're still around is all. :)

meleu

@used2berx hello friend.

I'm sorry to say I'm moving from this code. I started to work on this to work out my skills to manage data on text and xml files (basically improving my sed and xmlstarlet skills) and it was a great exercise. I'm glad to make something useful for someone during this exercise, but now I'm quite busy with other projects and don't have enough free time to maintain this script.

If you find another shell programmer, tell him/her that the script is always available at that github repo. And if they have questions about the code I can try to answer at some time. I can also try to review others code if they submit Pull Requests to that repo.

Cheers, and take care!

Used2BeRX

@meleu Well... that's going to put a huge damper on any future Rx releases for the RetroPie scene....

Bummer.

Maybe somebody else will help out down the road.

Good luck with whatever you've got going on man. Thanks for the work so far.

Used2BeRX

@meleu I had some time to sit down with the synopsis files and I believe that I figured out the common theme to any of the entries where the body text wasn't filled out with your script. In Notepad++, the ones that don't work right are listed as an "ANSI" file. All of the ones that work that I've checked seem to be listed as "UTF-8".

I don't know much about this other than they're two forms of character encoding and that UTF-8 is far more advanced than ANSI was. I have no clue how some of these files were saved differently. I'm going to try re-saving all of the non-working files and see if I can somehow save them as UTF-8.

Just wanted to let you know what that bug was.

EDIT: Yep. I believe that's what the problem is. In Notepad++ I can go to the "Encoding" menu and then select the second option "Encode in UTF-8". Some characters, like alternate apostrophes and quotes then change to weird things like "x92" and are highlighted. I can then just select them and put in a regular character and save it.

I'm going to do this to about a dozen of the bad ones and re-run the script before I sleep. They should all be working correctly with the new gamelist.xml.

Used2BeRX

@used2berx Awesome. That worked. Now I just need to go through and figure out which files had the ANSI format and convert them all to UTF-8 and the game descriptions will all be good. :)

Used2BeRX

Okay. Some weird things were happening. Lot's of characters that were in the synopsis files had bad characters that showed up as a floating box in both the XBox and the RetroPie. RP seems to have been able to display more of them though. What I can't figure out is why some files were ANSI when I opened them in Notepad++ and others were UTF-8, even though they had the same bad characters. Once they were fixed and saved though, they were automatically UTF-8 and I couldn't even manually save them as ANSI if I wanted to (which I didn't, but I was just testing it out).

I'm glad this issue came up now, actually. For years it has always bothered me that the strange characters looked so lousy on the XBox. The fact that they were actually breaking the script meant that I had to do something about them finally. I'll know for sure if I got them all after the new gamelist.xml is generated, but I think it's pretty safe to say that all 2,116 NES/FDS synopsis files are clean now. :)

I'm working with somebody to hopefully make a script to automate this process for future systems. I don't like automation unless I know exactly what it's doing, but after having spent 8 hours with this last night I feel pretty comfortable with it as long as it does exactly what I need it to do... which is the following:

Change all characters on the left side to the ones on the right side of the equal sign:

à = a
ā = a
é = e
ñ = n
ṇ = n
ø = o
ō = o
ǚ = u
ū = u

™ = Nothing (remove it entirely)

– = -
— = -
‘ = '
’ = '
´ = '
… = ...
“ = "
” = "

That's it.

There may be a few different characters that I find here and there as I move on to other systems down the line, but they could be added to the automation process as I discover them. This list should take care of 99% of any future issues. :)

Oh... and as a bonus, this also makes the gamelist.xml slightly smaller since every one of these characters were a long series of numbers in the XML files such as &#x2013 and &#x201D. It won't really make much of a difference on an XML file that is over 3.3MB, but every little bit helps... especially on the Pi Zero.

Those numbers are actually how I found all of these problems. I just did a search in Notepad++ on the last generated gamelist.xml for &#x and I found all of those one-off accented letters that I never would have found using my eyeballs. :)

Used2BeRX

Thanks to my bud Nemesis I've learned a lot about the code in this script. I'm no coder, but using existing code I was able to make a few changes so far that I wanted in this script. I'm re-running it now, and all of the desired changes I made this time around are taking effect. :)

I was able to add code to add and find all data for Famicom Disk System games that are in NES subfolders without having to have manually changed the "Platform: " line to erroneously read "Nintendo Entertainment System" instead of the legitimate tag of "Famicom Disk System".

I was also able to add a <boxfront> tag as a placeholder for all Box Front images, so now we have one for Box Front, Cart, Titles, Action and 3D Boxart images.

I was also able to make the code put the Cart images in the <marquee> tag, so now a skin that uses both tags can display both the Box art and the Cartridge art simultaneously.

The next thing I want to do is a bit trickier. I've heard from somebody here that if you don't have a video for a game, the <video> tag can be used as a third image type. I'd like to add some code in there that will search for the video as it is already doing, but if it doesn't, it will add either the Title image or the Action image shots in that tag.

Once all this is taken care of, I'm hoping Nemesis and I can come up with a quick script that will change which images populate the <image>, <marquee> and <video> tags on the fly after the gamelist.xml is created. Right now, on the Pi Zero it takes about 3 hours for the script to run on the NES alone. But after it is created it should only take a few seconds to run a script that will change which folder each image tag points to for every game.

Another thing on the list now is to add every single tag from the synopsis.txt files to the gamelist.xml file so when I convert them and then use @mitu 's script to convert it to a spreadsheet, I can make all the edits I want in the spreadsheet and then convert it back to a gamelist.xml file and finally back to the synopsis.txt files without losing any of the original data.

This is exciting. If we can get this working it's going to make editing game information for both the XBox and the RetroPie easier than it ever has been before.

Thanks again for all that hard work @meleu

Used2BeRX

Awesome.

I don't really know crap about coding, but I'm pretty good at pattern recognition and logic puzzles. Even though I don't know exactly what @meleu 's code is doing, I'm slowly figuring out ways to manipulate it.

So far I've been able to add a line for <boxfront/>that's separate from <image/>. At least temporarily, I've pointed the secondary image type <marquee/> to the Cart location as well. This will make it much easier to switch between which image types are in the "hot spots" and are actually usable in RetroPie in an alternate script. The reason you'd want to do this is because it would literally take a few seconds to switch the file locations in the completed gamelist.xml instead of generating an entirely new one from scratch with different image parameters (which on a Pi Zero right now takes over 3 hours for just the NES gamelist).

That being said, I'm going to want to figure out a better way to get the <image/> and <marquee/> tags now that I've done all of this. I should be making it simply "copy" the file locations from <boxfront> for the image tag, and from <cart> for the marquee tag. I believe right now it's looking in the same location twice now, adding a lot of code run time to a process that already takes hours to finish. I'm sure this is a really simple thing to do. I'll figure it out eventually. It doesn't make any sense to do it the way that I'm doing it now.

Meleu had to do some coding wizardry to make my Players: and Release Year: fields work in RetroPie as well. This was great, but in order to preserve all data when transferring the gamelist.xml to a spreadsheet, doing massive editing on the spreadsheet, then converting back to a gamelist and further back into individual synopsis.txt files for the XBox, I need to preserve the data exactly as it is found in those fields.

By copying and pasting some existing code that populated other fields, I have managed to add an <xtrasplayers> and <xtrasreleaseyear> tag to the gamelist. Though not useful at all in RetroPie, these tags will preserve the data in its entirety when converting from format to format. :)

I've got some other various tags to add as well that I'm going to get around to sometime today hopefully. It's taking me a pretty long time since I'm documenting everything pretty heavily as I go just so I remember what I've been doing and I don't get lost.

No wonder why @meleu wanted out. This is a lot of stuff to keep straight. :)

Thanks again bud. Sorry it took me almost a year to get around to actually learning anything about the code. Probably better that you stopped so I would be forced to learn some of this on my own.

Used2BeRX

Does anybody know how to copy a location put in an XML tag into a 2nd tag?

I want to change @meleu's script to search for all image location tags first (<boxfront>, <cart>, <titles>, <action>, <threedbox>, <xtrasmarquee>).

Then, based off of which image types are selected when running the command, I would want the actual tags used by RetroPie to be populated by these image locations without having to re-scan for them. (Default locations pasted here would be the "Box Front" folder for the <image> tag, and "Marquee" folder for the <marquee> tag.

In this example, it would fill <image> with whatever info was just scanned and put in the <boxfront> tag for the game. And it would fill the <marquee> tag with whatever info was just scanned and put in the <xtrasmarquee> tag.

This will accomplish 3 things.

It will cut down the time needed to run the script quite a lot by not having to scan for the same image twice for two separate tags.
This will allow me to create a script that will (hopefully) easily be able to change which images are considered <image> and <marquee> after the gamelist is made. It currently takes 3 to 4 hours to run this script on the Pi Zero for my NES/FDS collection of 2,116 games, so running it several different times to change these image types is foolish, and creating a script to do this on the fly would be much easier than what I've been doing, which is constantly going into Notepad++ after creation and manually running a few "find and replace all" commands to change the directories after the fact.
This would allow me to hijack the <marquee> tag for whatever secondary image I want to use in a particular skin easily. Ideally, we'll see more possible image types in RetroPie in the future, but with the addition of the Marquee tag at least a skin could display both the Box art and the Cart art at the same time.

I've managed to do quite a bit of things I wanted to do with this script now, but because I don't really know what the code is actually doing I'm lost at this point. I'm going to keep trying to Google the answer I'm looking for, but since I'm not even that familiar with Linux jargon and terminology, I'm having a difficult time finding anything relevant.

Thanks to anyone who reads this and has any ideas! :)

Used2BeRX

Interesting...

I finally put some random stuff over to the Pi in all of the media folders (including new ones that I added). All of them were filled out with the proper file in the location except for my new tag <xtrasmarquee>. I didn't figure it out right away since the media was there and all of the code for the new tag seemed to be in order.

That's when I figured out that I had improperly set the following code:

            # xtrasmarquee : find it
            boxfront="$(find_file "Artwork/Marquee" "$file_name" png jpg)"

It should have read like this:

            # xtrasmarquee : find it
            xtrasmarquee="$(find_file "Artwork/Marquee" "$file_name" png jpg)"

The "interesting" part to me was that rather than create a 2nd entry, it simply re-wrote the <boxfront> image location that it had previously just found with the Marquee image instead.

This:

            <boxfront>/home/pi/RetroPie/Media/nes/Artwork/Marquee/8 Eyes.png</boxfront>

Instead of This:

            <boxfront>/home/pi/RetroPie/Media/nes/Artwork/Box Front/8 Eyes.png</boxfront>

I'm actually happy to see this happen, and I'm learning things. Now I'm pretty sure if I can figure out how to code it correctly that it shouldn't be hard to copy the existing <boxfront> tag into the <image> tag for each game since it seems to wrap up what it's doing for each game entry before moving on. I was worried this was going to be a pretty complex solution.

Also, I fogot to mention that I added "PDF" images to the search for game Manuals as well as zip files. Last time I checked there is no way for RetroPie to actually view manuals, but in case it ever gets added, the script will look for either PDF files or Zip files that house PNG or JPG images of the manuals.

So the existing code will successfully output all media for a game now like this.

<boxfront>/home/pi/RetroPie/Media/nes/Artwork/Box Front/8 Eyes.jpg</boxfront>
<cart>/home/pi/RetroPie/Media/nes/Artwork/Cart/8 Eyes.png</cart>
<title>/home/pi/RetroPie/Media/nes/Artwork/Titles/8 Eyes.png</title>
<action>/home/pi/RetroPie/Media/nes/Artwork/Action/8 Eyes.png</action>
<threedbox>/home/pi/RetroPie/Media/nes/Artwork/3D Boxart/8 Eyes.png</threedbox>
<xtrasmarquee>/home/pi/RetroPie/Media/nes/Artwork/Marquee/8 Eyes.png</xtrasmarquee>
<gamefaq>/home/pi/RetroPie/Media/nes/GameFAQs/8 Eyes.zip</gamefaq>
<manual>/home/pi/RetroPie/Media/nes/Manuals/8 Eyes.pdf</manual>
<vgmap>/home/pi/RetroPie/Media/nes/VGMaps/8 Eyes.zip</vgmap>

What I'm looking to do now is find a way to create the <image> tag by simply copying the image location in the <boxfront> tag. And then create the <marquee> tag by copying the image location in the <marquee> tag. These would be the defaults, and would ultimately be able to be changed upon gamelist.xml generation and/or with the yet to be created script that could change all of the locations for these tags on the fly after it was generated.

Though this isn't absolutely necessary to figure out, it would save a lot of time during the creation of the gamelist.xml file by not having to find the same image twice for two tags. This might be a moot issue on a Pi 3, but with how long it takes to make these files on a Pi Zero every little bit helps.

Used2BeRX

Got it! Finally....

Although, I'm not sure if I actually accomplished what I was trying to do, which is simply copy the output from one element <xtrasmarquee> and paste it into another element <marquee>. It's quite possible that it's actually still looking for the actual file name and path again instead of copying the data. Anybody know how to verify this or know firsthand if it's doing it right?

Here's the lines in question:

            # xtrasmarquee : find it
            xtrasmarquee="$(find_file "Artwork/Marquee" "$file_name" png jpg)"
	    marquee=$xtrasmarquee

So it's first finding the actual file when generating <xtrasmarquee>.

Then the second line is either copying the xtrasmarquee output data and pasting it into the <marquee> element (which is what I want), or it is entirely re-running the first line, searching for the file and location, and then putting the data in the <marquee> element (which is exactly what I don't want).

It's definitely doing one of the two things though. Earlier in the code I populate the <marquee> tag with the file/location for the "Cart" image. If I run the command above it is properly showing the "Marquee" image file/location. If I comment out that line, then it will show the "Cart" image file/location instead.

Progress..... I think. :)

Used2BeRX

Well.... looking at my last post here, I can tell just about exactly how long it took me to figure out how to do the next thing I wanted to do.

The script was writing in the first line as a line break for all the <desc/> entries. Not a huge deal at the end of the day and not super noticible while scrolling through games since they all did it, but with screen real estate at a premium, having that useless blank line at the beginning of every game description was dumb.

For 3 hours I tried so many different things and re-ran the script only to fail. I was totally on the wrong track. I was trying to get the code that recognized the first line of the Body text in the synopsis.txt files to also recognize line breaks as part of what was to be removed, basically by changing the highlighted text below to about 100 different combinations of crap that was never going to work in the first place....

desc="$(sed '/^__________/,$!d' "$file" | tail -n +2 | tr -d '\r' | sed 's/&/&/g')"

The secret was in tail

All I had to do was change tail -n +2 to tail -n +3 and it got rid of that extra line.

(What I believe) this command does, is starts writing from the line specified. (With "-" numbers actually writing everything up until those lines at the end, and "+" starting the writing at the line specified.

The beginning of every "Body" text for the descriptions starts with ____________________ and then has a blank line before the text starts. This is because the entire file shows up when displayed on the XBox as simple text, and that line break looks better than not having one. In the case of tail here in this line of code, the ____________________ line is considered line +1

So with tail -n +2, the script was starting at that blank line when it began writing. By changing it to tail -n +3, it begins writing at the correct spot.

Although that fixes the problem now, I would like to find a way to just start writing at the first Alpha-numeric character instead. I did a lot of heavy editing for the NES synopsis and know for a fact every single one is formatted this way. They weren't before I did all this work though. If that space is missing in any future entries on future systems then this script will cut off the first line of text. Now that I think about it, it would actually cut off the first paragraph since the paragraphs are considered a single line without any breaks.....

Used2BeRX

Well that worked, but if I was going to remove the URL links at the bottom of the body so I could put them in their own tags, that was going to be another story altogether. I thought I had it with a simple head -n -2 stuck in that line as well. I'd only been testing game synopsis files with it, and didn't even think what would happen when I tried converting the folder synopsis files that don't have urls at the bottom. Turns out I lose the last paragraph when I do it. :)

Well here's what my <desc> code looks like now. Quite a bit more complex than it was before...

if [[ "$NO_DESC_FLAG" == 0 ]]; then
		desc="$(sed '/^__________/,$!d' "$file" | tail -n +3 | tr -d '\r' | sed 's/&/&amp;/g' | sed -n '/http/{N;s/.*//;x;d;};x;p;${x;p;}' | sed '/^$/d' | sed '/^$/d;G')"

fi

sed -n '/http/{N;s/.*//;x;d;};x;p;${x;p;}' | sed '/^$/d' --- This code removes any line that begins with "http" as well as the line directly above it and below it. This way it knocks out both URLs if there are two as well as the blank line above the first one.

sed '/^$/d;G' After that, I'm not exactly sure what happened with my code. All of the entries I had with more than one paragraph lost the space in between them. Adding this on the end of the line of code brought my spaces back. :)

Here's the code to separate the URLs and give them their own tags:

# url : Synopsis cited link #1 - Citation link for info in all synopsis.txt files / Not usable in RetroPie (05/29/2018)
if [[ "$NO_DESC_FLAG" == 0 ]]; then
    url="$(sed '/^http/,$!d' "$file" | head -1 | tr -d '\r' | sed 's/&/&amp;/g')"
    url="$(grep -Ev '^(https?|ftp)://[^\s/$.?#].[^\s]*$' <<< "$url")"
fi

# urlalt: Synopsis cited link #2 (Typically only for Hacks and Translations) / Not usable in RetroPie (05/29/2018)
if [[ "$NO_DESC_FLAG" == 0 ]]; then
    urlalt="$(sed '/^http/,$!d' "$file" | tail -n +2 | tr -d '\r' | sed 's/&/&amp;/g')"
    urlalt="$(grep -Ev '^(https?|ftp)://[^\s/$.?#].[^\s]*$' <<< "$urlalt")"
fi

@meleu

You wouldn't hardly even recognize this code anymore man. I've been up about 30 hours straight messing with it. I'm going to have to let it run when I finally go to sleep so I can test the results out, but I think I might be able to actually get this thing to do everything I want to do with enough Google searching.

I still don't know what 3/4 of this code even means, but I've never let my complete ignorance on a subject keep me from being an expert on it before. Ain't about to start now. :)

Used2BeRX

So... I'm going to have to change some of that above code. When checking through all of the games, I noticed that there were 3 games that now didn't have any text to the body at all. It turns out that this was because I hadn't put a space between the single paragraph entry and the citation URL, and the code is telling the script to not copy over the URL line as well as the line above and below it.

So this pointed out all 3 entries I had out of 2,116 that had this problem, right? Wrong. :(

This only told me which ones had this typo that were single paragraph entries. Without opening every document and verifying this is correct on all files, I really have no way of knowing if this was a problem on other entries that had more than one paragraph. I only caught these because they were single paragraph entries and there was no body text at all in EmulationStation.

So I'm just going to change that code to delete the URL citation line and the line below it. It doesn't hurt anything to have a kilobyte or two of extra blank lines at the end of all the collective game entries in the gamelist.xml just to be on the safe side in case I made this typo in other spots. :)