Making premade gamelist.xml with xtra media collections

Used2BeRX

@edmaul69 I don't know what you're seeing on your end, but it looks to me like 100% of the time so far the only games that it will convert are 64kb games like "All Night Nippon SMB" and Clu Clu Land (Think the black-box era of very early NES titles).

These seem to be the only games that have a 32k PRG rom. The others vary wildly in many cases but most seem to have a 16k PRG rom.

All in all, I'd say that about 90% of the games in the collection are 128kb and will not be able to be converted with this tool. :(

Used2BeRX

@edmaul69 I just talked with somebody who is very knowledgable with FDS hacking and they told me that this would be a major task and nobody would likely ever try to convert the FDS roms to NES since pretty much every NES emulator will play them. This issue is going to have to be corrected on the emulation side.

A consolidated version of the reply I recieved:

The way the FDS works, it has a BIOS for loading from disks, unlike the NES which doesn't need one. Some games will just load the entire game into RAM at the beginning. Others use both sides and have data for each level in different files, which are all transferred into RAM at the same point, overwriting each other. The files can be any size, and can be written to wherever you like in RAM.

This is very different from a ROM using bank switching. MMCs switch banks in certain specific amounts, and it happens instantly. Whereas a file could only be under a kilobyte, a bank has to be at least 4kb, usually more, and is a set size, so often you'll get wasted space.

Not to mention in all this that instructions in the games will point to different locations if you changed to a ROM. It could take a lot of reprogramming to make a larger FDS game work in ROM format, and require one of the more advanced mappers.

At which point you realize that every emulator can play them so you wonder what's the point.

meleu

@used2berx yo! I think I'll have some free time today. Can you take a look at our discussion here?

Cheers.

Used2BeRX

@meleu Hey buddy. Awesome!

Sorry... I was napping a bit after work, but I got up and saw you had replied. :)

The "gamelist-to-synopsis" feature will be pretty easy to implement.

Great! Man... if @mitu can reverse the direction of his gamelist.xml to spreadsheet and you can do this, that will make future synopsis work about 10 times easier for me in the future. Maybe even 100 times easier! No more having to edit individual text files, and I'll be able to finally sort by any tag for super easy editing of things like Genre.

Platform: Nintendo Entertainment System and Platform: Famicom Disk System entries in the synopsis.txt files located in /Media/nes/Synopsis/ folder will treat these entries as if they were NES and not FDS games and look for roms and all media in the /nes/ folders instead. (This should not occur if the synopsis, media and roms are located in the /Media/fds/ directory though).

Can you confirm if you still want exactly what you're describing above?

Yes. This is what I want. All of my FDS roms are in the NES roms folders, in the following directories (and all of the media is just in the corrosponding nes media folders):

/home/pi/RetroPie/roms/nes/(4) Famicom Disk System/(4_1) Translated FDS/
/home/pi/RetroPie/roms/nes/(4) Famicom Disk System/(4_2) Untranslated FDS/
/home/pi/RetroPie/roms/nes/(4) Famicom Disk System/(4_3) Update Hacks FDS/

The thing is, maybe somebody wants to use your script down the line that actually wants to separate the FDS system from the NES system. I just didn't want to "break" our script for them by simply not allowing this. So that's why I suggested that this only work for us when the FDS media/roms are in the /nes/ folders, and for somebody who was running the script from the /fds/ folder, it would work normally.

PLEASE let me know if that's still not clear. I'm doing my best to describe the situation, but I know that I tend to over-explain things and make them more complicated than they need to be. :)

meleu

@used2berx can you say all the possible Platforms you have on your Synopsis files? This info will be useful to implement the gamelist.xml to synopsis.txt feature.

Used2BeRX

@meleu Sure. These are only the ones that were done years ago. My hope is to add more console/handheld systems in time. I don't think I'll be adding any computer based systems though since they require a lot of individual controller configurations and would be a monsterous project on its own. Although I have done some work with DOSBox myself and I know another guy who is also working on that, and there has been an amazing Commodore 64 collection with the controller configs, but that would be a LONG way down the road.

Here's the current list:
Atari 2600
Atari 5200
Atari 7800
Atari 800 * See note below
Atari Lynx
Bandai Wonderswan
ColecoVision
Magnavox Odyssey2
Neo Geo Pocket
Neo Geo Pocket Color
Nintendo 64
Nintendo Entertainment System
Nintendo Game Boy
Nintendo Game Boy Advance
Nintendo Game Boy Color
Nintendo Virtual Boy
TurboGrafx-16 / PC Engine * I didn't separate the CD games for some reason... I will in the future.
Sega 32X
Sega CD / Mega CD
Sega Game Gear
Sega Genesis / Mega Drive
Sega Master System
Sega SG-1000
Sony Playstation
Super Nintendo Entertainment System
Watara Supervision

I believe that's all of them that I had done in the past. There were a few other meaningless ones like the Dreamcast VMU unit and Pokemon Mini games that aren't even emulated on the Pi that I probably wouldn't bother with.

BTW... With the Atari 800, could you do the same thing we're doing with the NES/FDS systems, so that if the Atari 800 stuff is located where the Atari 5200 stuff is it will work with the 5200? I have no interest in doing all the work on the Atari 800 computer since I never owned one and it would be a ton of controller configs to make, but I figured I'd probably put a smaller collection of 800 games in there that I found that were quite enjoyable that I've added to the XBox set years back. This way if anybody were to do the Atari 800 themselves someday this script could still work for them.

Used2BeRX

Also, I was wondering if we could add something else to the to-do list. I wanted to know if we could add a tag that is the synopsis.txt file name without the extension (txt). We could just make the tag <filename></filename>.

This might have several uses down the road since this is the exact file name of all of the associated media for every game. Right now it would be great for when I'm working on the spreadsheet as a way of alphabetizing stuff. When we get all the tags in our script with the FULL command, I hope to talk to @mitu about an option in his own script that would transfer over all of the tags that we're using. Otherwise, when we go through the process of turning the spreadsheets all the way back to text files there will be a lot of missing information.

BTW... I'm also going to try to see if somebody on the team here could add more image tags for us to use. And even possibly implement something where you could press a certain button on the controller to scroll through the existing image types. I think it would be awesome if we could have a video preview and an image box that you could scroll through the Boxart, Cart art and Title screen of all of the games.

Right now we only have the <image> and <marquee> tags. If we were to have tags for the tags you've already added like <boxart> to your script, it would eliminate the need to be able to change which image type is the <image> and not require a re-run of the script just to change the image type in the romlist.

Thanks again man! Excited to see you're back in any capacity. :)

meleu

@used2berx When a tag in gamelist.xml is empty or inexistent, what do you want to put on the synopsis.txt file?

Example: if there's no <originaltitle> in the gamelist.xml, do you want an empty Original Title: on synopsis.txt or simply skip this info?

Used2BeRX

@meleu Good question. Thanks for asking since I didn't think of that before. :)

Any empty tags should just not be put in the synopsis.txt files. This is because on the XBox they are displayed "as-is", so if there were a bunch of empty fields it would look messy and would take more scrolling to get through the whole thing.

You probably also thought of it, but any media tags we make for the locations in the gamelist.xml file should also not make their way back into the synopsis files. (No artwork, videos, gamefaqs, etc.)

Essentially, the only tags that should make their way back into the synopsis are the ones that were there originally. The only exception would be data/information tags that had information that I added in the spreadsheet before converting back to the gamelist.xml. For instance, if I realized in the spreadsheet that a game was missing the Publisher: information and I filled it in the spreadsheet, then that information would then make it back into the synopsis file.

Now that I think about it, I'm wondering if we should make a separate tag for the synopsis cited website at the end of the <desc> tag? This way we could still have that information in the synopsis.txt, gamelist.xml and the spreadsheet, but we don't have to take extra steps to remove it so it doesn't show up in the RetroPie romlists. The only thing we'd need to do is make sure that they get placed back at the end of the <desc> after two line breaks when it's covnerted back to the synopsis.txt.

Maybe call it <citation>?

meleu

@used2berx

Maybe call it <citation>?

Do as you prefer.

On the code side it would be easy to consider that "if the very last line of a synopsis.txt is an URL, then it's a <citation>".

meleu

@used2berx said in Making premade gamelist.xml with xtra media collections:

You probably also thought of it, but any media tags we make for the locations in the gamelist.xml file should also not make their way back into the synopsis files. (No artwork, videos, gamefaqs, etc.)

Please, confirm if these are the ones to be ignored:

path
image
video
marquee
xtrasname
cart
title
action
threedbox
gamefaq
manual
vgmap

Used2BeRX

@meleu

On the code side it would be easy to consider that "if the very last line of a synopsis.txt is an URL, then it's a <citation>".

That will work. :)

Please, confirm if these are the ones to be ignored:

path
image
video
marquee
xtrasname
cart
title
action
threedbox
gamefaq
manual
vgmap

Yes. That is all correct.

Used2BeRX

@meleu

Hey man. I'm not getting any traction on my thread about trying to get more image types supported in RetroPie. Do you know who I should hit up for that idea?

The thread is here: https://retropie.org.uk/forum/topic/16535/multiple-image-types-tags/2

Thanks :)

Used2BeRX

@meleu

I'm not sure how linux scripting works, but I'm assuming after we run our desired command to start the script that we could have it ask a few questions for our image setup? I just found out from the other thread today that we currently have access to 3 image types.

<image>
<marquee>
<thumbnail>

The thumbnail tag is used to show a screenshot before a video plays if you set a delay. @EctoOne actually uses this in such a way that the video isn't played at all and it acts as a third image type. This would be perfect for the Pi Zero setup that I'm building that won't have videos.

With this, there is actually currently coverage for Box Front, Cart and Action images.

So what I'm asking if we could do is two things...

Make a tag for every image type. (The default script now doesn't make <boxfront> because it makes the box <image> instead. I would like to see this become it's own tag... I'll explain why later.)
Ask a few questions about image types after desired script command is run, but before the script is executed. These following questions:

Image type for <image> ?

Box Front
Cart
Title
Action
3D Box

Image type for <marquee> ?

Box Front
Cart
Title
Action
3D Box

Image type for <thumbnail> ?

Box Front
Cart
Title
Action
3D Box

If we do it this way and leave the regular image tags we decided on for the image type, this should be very easy to change them in the script (or a separate script) in the future. That way we could run it and just swap out the locations for different image types without having to rebuild the entire gamelist from scratch. The image changing script could ask the same 3 questions and then just replace those tag lines with the image location of the desired art type. :)

Thanks bud... sorry to change this on you like that, but I think this is the best way to go forward with this new image information we just got.

Used2BeRX

Also... Below is an example of one of the game entries the script currently makes. I've highlighted where in the entries the new tags should be generated for ease of viewing/usage.

Along with the image tags, there are a few other ones that we've discussed from time to time in here along with some notes about them.

<game>
<name>American Gladiators</name>
<path>/home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/American Gladiators.zip</path>
<image>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</image>
<video/>
<marquee/>  `(This was already in the script)`
<thumbnail/> (**New addition**)
<desc> [Body text removed to simplify this post]</desc>
<link>https://www.gamefaqs.com/nes/563382-american-gladiators</link>  (**Our new link code**)
<releasedate>19910101T000000</releasedate>
<releasetext>1991</releasetext> (**Our text based release year from the synopsis `Release Year: ` line**)
<hackreleasedate/> (**New addition**)
<transreleasedate/> (**New addition**)
<developer>Incredible Technologies, Inc.</developer>
<publisher>GameTek, Inc.</publisher>
<genre>Game Show / Competitive Events</genre>
<players>2</players>
<playerstext>1 to 2 VS</playerstext> (**Need this to properly convert back to synopsis after edits.  Just a pure rip of the synopsis information without your code for converting it to an ES friendly field**)
<region>USA</region>
<platform>nes</platform>
<media>Cartridge</media>
<controller>NES Gamepad</controller>
<gametype>Licensed</gametype>
<xtrasname>American Gladiators</xtrasname>
<originaltitle/>
<alternatetitle/>
<hackedby/>
<translatedby/>
<version/>
<boxfront>/home/pi/RetroPie/Media/nes/Artwork/Box Front/American Gladiators.jpg</boxfront>  (**Same as <image>.  Our code defaults <boxfront> to <image>, but this should also be a separate line for scripting quick changes to image types on a script re-run**)
<cart>/home/pi/RetroPie/Media/nes/Artwork/Cart/American Gladiators.jpg</cart>
<title>/home/pi/RetroPie/Media/nes/Artwork/Titles/American Gladiators.jpg</title>
<action>/home/pi/RetroPie/Media/nes/Artwork/Action/American Gladiators.jpg</action>
<threedbox>/home/pi/RetroPie/Media/nes/Artwork/3D Boxart/American Gladiators.png</threedbox>
<gamefaq/>
<manual/>
<vgmap/>
<license/>
<programmer/>
<musician/>
</game>

I can't think of anything more right now that we were talking about. I hope this all makes sense. Please let me know if I need to clarify anything. :)

Used2BeRX

@meleu I forgot about the "real date" tag. I added that to the example I gave above. :)

EDIT: I also forgot about two tags we haven't even discussed yet.

In the synopsis for both hacks and translations I have lines Translation Release Year: and Hack Release Year: as well as the original game release year.

I've added both tags to the post above: <hackreleasedate/> and <transreleasedate/>

Thanks :)

Used2BeRX

@meleu

Hey man. I mis-remembered our prior conversations and just found out that the FDS games aren't being built with the NES gamelist.xml. I was hoping you might have some time to work on that because now that I'm done with the cart artwork I've re-run the gamelist script and I can't verify any of the FDS stuff is working and the artwork is good. I'm considering going back into the FDS synopsis.txt files and changing the system name back to Nintendo Entertainment System just so I can finally get these things working if it's too difficult to make the code do this part.

Let me know.

Used2BeRX

@meleu

Thanks for adding the option to remove the FULL out of the gamelist.xml files man. That feature works flawlessly and will save me a lot of time renaming files down the road. :)

Used2BeRX

@meleu Hey bud. You alright man?

I think I found a bug in the ointment. It seems that when scraping the synopsis files, the script is not grabbing the body text for all of the games.

I found this out a few weeks ago, but I haven't brought it up since you haven't replied for so long. Just making sure that you're still around is all. :)

meleu

@used2berx hello friend.

I'm sorry to say I'm moving from this code. I started to work on this to work out my skills to manage data on text and xml files (basically improving my sed and xmlstarlet skills) and it was a great exercise. I'm glad to make something useful for someone during this exercise, but now I'm quite busy with other projects and don't have enough free time to maintain this script.

If you find another shell programmer, tell him/her that the script is always available at that github repo. And if they have questions about the code I can try to answer at some time. I can also try to review others code if they submit Pull Requests to that repo.

Cheers, and take care!