Making premade gamelist.xml with xtra media collections

UDb23

@Used2BeRX if you have the single text files available I can create a program that reads the txt and generates corresponding XML.
Did something similar in the past; ideas is it would convert all txt files it finds in a specific folder.
Would make that in VBA meaning you need Excel to run it ( just to run, no need to do any data import in excel).
Could also do it in VB and compile it into an executable; this would require more time as I don't have VB installed and haven't been using it for years..
Let me know if excel VBA could be ok.

Concerning global gameslist XML referencing to single game (sub) xmls I don't know if this is supported by ES.

UDb23

@Used2BeRX Really interested in your work on Atari 2600 gameslist especially if you can provide high quality cover images.
Also interested in building Vic20 gameslist.
Wrote my own mobygames scraper for that and downloaded all metadata from the site. Have found separate hq cover images to be used in the gamelist XML. Will combine them sometime.
Currently priority is on building my first control panel. Correctly drilling holes in plywood board isn't as simple as I thought :-)

Used2BeRX

@UDb23 Here's a download for the synopsis files for the individual 2600 games. If you can do something with this, I'd be happy to zip up a package with all the other systems. These are all named exactly the same as the roms are, except of course the extention is .txt instead of .zip.

Synopsis 2600: https://www.sendspace.com/file/mgf38h

And here's an XML file for the 2600 I'm currently working on. It has the 10 files for each game already made, as well as the <name></name> field filled in (although I will have to go back and edit some of them since I'm not actually ripping that from the txt file and I'm just doing a copy/paste job.

2600 gamelist.xml wip: https://www.sendspace.com/file/km2sf2

As you can see from the parts of the XML that are completed, the file names for the Extras for the system are named exactly the same as the rom itself, so everything should work right... unfortunately, the system we did it on originally didn't care about case, so chances are some things in there are broke because the case doesn't match. That's the stuff that I would have to fix by hand after the files were created.

Excel isn't a problem for me. I don't have it installed right now, but I do have a copy of 2007 lying around here somewhere. Hopefully a version that old will still allow me to use your script.

No need to make a VB program. It's not that I don't have time to wait, but I'm really OCD-ing out here and I'm manually doing things right now I said that I wasn't going to do because I'm waiting for somebody to automate this and patience isn't one of my virtues ;)

Looking forward to see what you can come up with. Chances are the xml I'm working on won't be usable with what you make, but that still isn't enough to make me stop working on it.

I've got good Atari 2600 artwork. There is at least a Box Front and Action shot for every game. Not a lot of title shots since a lot of games didn't have title screens and probably 100-150 carts missing.

For the boxes that we couldn't find or were never made, we actually had a very talented graphic artist on "staff" that created alternate boxarts for prototypes, hacks, translations, unlicensed and other categories.

Looking forward to seeing what you can do man. You're the only hope for my sanity at this point. :)

EDITED TO ADD: Now that I think of it, it might be good if there were two different scripts. One script that would create all of the fields/tags like the XML example I gave you, and then a second one that would "rip" the information from the TXT files and fill in the empty tags made by the first file.

That would make sure that work I've already done on the first part would still be usable by the time you make this, but it would also allow me to make sure the files are all working (because of the case issue) before ripping all the data. Any chance we could do that? Maybe that's easier to make it a two step process anyway?

You may have noticed that the path is not just "./" There are many sub folders for a lot of the systems, so the script should be able differentiate that in the path folder based off of which folder the roms are in. I'm making them in alphabetical order, grouped together in the folders. (Maybe you should see this XML when it's completed to fully understand what I'm talking about). I'll upload it as soon as it's done. I'm hopped up on coffee right now so it shouldn't be long :)

Used2BeRX

@UDb23 Well..... that was horrible. Just the 1st part took me about 5 1/2 hours, so hopefully that's the last system I'm doing that for with your help man.

https://www.sendspace.com/file/modd8g

I'll probably still be editing that a bit, but I did verify that all the boxart works with it. There were about 4 lines in there that I had to change because the artwork was a different case than the rom. Not too bad when it was over 700 entries. I didn't check any of the other artwork or videos yet.

Let me know what questions you've got.

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm really OCD-ing out here and I'm manually doing things right now

You shouldn't invest time in manually editing those files ! ;-)

Excel 2007 is fine to run my VBA code.
Let's proceed in steps with development:

Create program to parse existing txt files and generate separate "per game" xml files + global gamelist.xml.
Parsing requires source txt to have a defined structure.
Looking at those TXTs the first part is ok (assigning specific fields/tags value).
Then, after the"----------" there's the description. Last line always contains info source reference. Is that correct ? if so I will use these "rules" to parse the files.
Add customizable folder paths (preference settings) that will be used during xml generation. In this way people can have supporting files in the folder they prefer.

If I understood correctly for each game the rom name has to be added as it is not contained in the TXT file. All extras (boxart, video etc) are then named the same as the rom but, of course, with different extension and path.
This can be generated too by the program.

For existing XMLs (work you have already done) the program would read-in just the rom names and combine them with corresponding TXT data and generate XML.

I suppose some games will not have some of the extras, so generating ALL EXTRA tags would actually create some "broken links" to missing data. Not sure if that would make ES "unhappy" and generate an error or if ES just ignores this.

Concerning the case issue the program could simply generate all lower or all upper case.
In this way you just need to "case convert" all actual extra files accordingly. I'm pretty sure free windows utilities exist to do this.

I'll be away over the next days but can start working on it this weekend.

Used2BeRX

@UDb23 Hey man. Thanks again.

ES doesn't freak about links to files that aren't there. Many of the games don't have all of the artwork, especially the title shots on systems like the 2600 that didn't have many titles. I don't actually put any of the other media on the Pi Zero I'm working on either, so there have been no errors popping up for missing videos or manuals, etc. Now is there memory being burned up using tags with no file? That's above my pay grade. :)

Haha! Yeah, I know I shouldn't be manually making them. I thought that the first part was a lot quicker than it actually was. I couldn't believe I put 5 1/2 hours more into the 2600 until I checked the time when I was done.

As for your "steps with development"

A per game xml is great, even if we can't use it now. Hopefully somebody could change the code of ES somehow to make use of it. We had to break the synopsis down from one master file to individual games because on large systems the file was insanely huge and was causing memory issues. Hell... if you look at my 2600 xml it's almost 900kb and that's without any of the game data in them. My guess is just for 700 games that file is easily going to be 3MB when its done.

Not every game is going to have every tag. My original synopsis file years ago had every tag for every game, but they made an online editor that the community could use and the individual game files that it creates spit out only the fields that had info in them. Sadly, the people who developed it are gone now so I can't change that.

The last line "usually" contains an info source reference. I made that a big rule of mine when I was creating the system that if we took info from a website to have a link to the page it came from. I can't say for sure it was always followed, and sometimes there was no info found online and people wrote up their own stuff with no reference link.

customizable folder paths would be good. Not only for me, but for people using this for their own stuff in the future too. For me it's good because the media/roms folders are going to be different from system to system and it would not require you to build alternate versions for different systems or me to try to go in and change things and probably break it.

Yes, the rom name is not in the TXT file. The way our system worked was that it would find the matching "filename.txt" when you pulled up the game information.

The problem with the Case Issue is that some games are named, for example "Attack of the Mutant Camels", but the artwork is named "Attack Of The Mutant Camels". It's usually not an either/or situation. If we can't correct for that, I think I'd rather just have it use the exact case of the game and I'll check that everything works by hand afterward and change the file accordingly. If possible, I want to maintain the existing file structure as is, so people don't have to re-download the entire thing or have two copies of it. I might change my mind on this part. I'll have to think about it while you're gone.

Thanks for the help. Looking forward to seeing it in action. Let me know if I can clear anything else up.

herb_fargus

Camel casing articles is a very simple process

UDb23

@herb_fargus said in Making premade gamelist.xml with xtra media collections:

Camel casing

Didn't know that it's called that way. :-))

UDb23

@Used2BeRX Everything clear so far. Will keep you updated.

Used2BeRX

@UDb23 Sweet.

There was a little miscommunication with the other guy that was helping me work something out and he created a python script for Linux that will make individual "game".xml files with tags for all of the game info and put the 10 file tags above it. It actually looks for the media if it's in the proper folders so it puts the proper extension too. Pretty cool, but I'm a Linux dummy and the only way I could really use it is in the Linux console with the Pi. The problem is I've got the Pi Zero with a small SD card so I couldn't really make use of it that way.

I've got really high hopes for what you're talking about doing here. Maybe if we iron out all of the kinks it could be actually turned into that executable you had mentioned as well. I'm thinking that what you're doing could be a valuable tool for anybody with Extra media for their games, even if they're not using my setup.

I've taken your advice and stepped away from this for a while, until you come back with something. I probably could have gotten the first half of the NES gamelist.xml done yesterday if I worked on it all day, but I'd be kicking myself for it once I had your program to work with I'm sure.

Let me know! Thanks.

cafarellidigital

I just wanted to show my support for this project, for all of the work that you've done so far, and I look forward to see where it goes in the future!

Used2BeRX

@cafarellidigital Thanks man. I think this is going to be pretty sweet when it's finished.

Used2BeRX

@UDb23 hey man. Was just wondering if you had any time to work on this at all? I'm pretty busy with some other stuff at the moment, so I'm not rushing you or anything. Just don't want to lose touch with the only guy who seems to have a plan on how to do this.

UDb23

@Used2BeRX Don't worry. ;-)
As promised I started working on it. In the end I decided to go directly for making an executable program (no need for excel..).
Downloaded and installed latest Visual Studio community 2017; took me some time to get acquainted with it.
So far I made a "skeleton" main program with functions to access file system and to do string manipulation. Tomorrow I hope to find time to write the code for the "conversion" from txt to xml (should be easy at this point).

Used2BeRX

@UDb23 Sweet. Just let me know if you need any more input before making the first compile. Otherwise I should be around to take it for a spin whenever you've got it ready.

Used2BeRX

@UDb23 Since I haven't heard back from you yet I figure you're still putting it together. I was hoping you could add a field in the first 10 <marquee></marquee> I don't have any marquee images myself, but some guys I know that are making a skin are using that field and making marquee style images by cropping from title screens.

UDb23

@Used2BeRX Sure I can add this additional tag.
Program development is progressing and I'll provide you a first working version over the weekend.

Used2BeRX

@UDb23 Nice.

I'm going to ask around here and see if I can get some answers on memory issues. I actually "broke" the Pi Zero with just the hand made XML files for only the 4 Atari systems. EmulationStation was taking 2-3 minutes to load and some emulators stopped functioning altogether. I'm just wondering if loading pointers to data that doesn't exist is draining memory resources or not. And I'm also wondering if every single picture you load is draining memory at one time. Seems terribly inefficient to me. I'll be able to test our work here one system at a time, but then I have to rename the XML file so it doesn't get loaded when I go on to the next system.

Anyways, maybe down the road there can be options made in your program to expand or limit exactly what fields the program adds to the XML file depending on what I find out about how things are loaded.

UDb23

@Used2BeRX First beta of the Application to convert your .txt to .xml is ready.
You can find it here.
Just unzip the MetaX.zip and launch Setup.exe.

Use the "Browse" button in the App to select the folder that contains the txt files.
Currently only one by one conversion works: select the file and click "convert" button.

I found that the game metadata source txt files are not consistent meaning some info may or may not be included; also same info (e.g. "developer" can be in different line in one txt and in another line in a different txt).

The application tries to find the specific "tags" regardless of line order; it also identifies the "description" by selecting the lines after the line with "________".

Could you let me know the complete list of "items/tags" the App should:

look for in the source txt files
write to the .xml files (on top of items found in the .txt)

Just want to first make sure everything is converted correctly as single files, and then implement "convert all" and "create global gamelist.xml" functionality.

App requires .net libraries installed (created in VB with Visual Studio community 2017); I suppose it will ask during installation (can't test on another PC).

note: if you convert the same txt multiple times, it will append lines to already existing corresponding .xml (we could call it a "known bug" ;-) ).

Note 2: App does not do anything else as converting txt to xml so it should be totally harmless. In any case keep a backup of your original txt files.

Used2BeRX

@UDb23 Awesome man. Thanks.

I downloaded it, but haven't had time to use it. I should have some feedback for you tomorrow or Tuesday night at the latest. Thanks so much for this again.