Making premade gamelist.xml with xtra media collections

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm really OCD-ing out here and I'm manually doing things right now

You shouldn't invest time in manually editing those files ! ;-)

Excel 2007 is fine to run my VBA code.
Let's proceed in steps with development:

Create program to parse existing txt files and generate separate "per game" xml files + global gamelist.xml.
Parsing requires source txt to have a defined structure.
Looking at those TXTs the first part is ok (assigning specific fields/tags value).
Then, after the"----------" there's the description. Last line always contains info source reference. Is that correct ? if so I will use these "rules" to parse the files.
Add customizable folder paths (preference settings) that will be used during xml generation. In this way people can have supporting files in the folder they prefer.

If I understood correctly for each game the rom name has to be added as it is not contained in the TXT file. All extras (boxart, video etc) are then named the same as the rom but, of course, with different extension and path.
This can be generated too by the program.

For existing XMLs (work you have already done) the program would read-in just the rom names and combine them with corresponding TXT data and generate XML.

I suppose some games will not have some of the extras, so generating ALL EXTRA tags would actually create some "broken links" to missing data. Not sure if that would make ES "unhappy" and generate an error or if ES just ignores this.

Concerning the case issue the program could simply generate all lower or all upper case.
In this way you just need to "case convert" all actual extra files accordingly. I'm pretty sure free windows utilities exist to do this.

I'll be away over the next days but can start working on it this weekend.

Used2BeRX

@UDb23 Hey man. Thanks again.

ES doesn't freak about links to files that aren't there. Many of the games don't have all of the artwork, especially the title shots on systems like the 2600 that didn't have many titles. I don't actually put any of the other media on the Pi Zero I'm working on either, so there have been no errors popping up for missing videos or manuals, etc. Now is there memory being burned up using tags with no file? That's above my pay grade. :)

Haha! Yeah, I know I shouldn't be manually making them. I thought that the first part was a lot quicker than it actually was. I couldn't believe I put 5 1/2 hours more into the 2600 until I checked the time when I was done.

As for your "steps with development"

A per game xml is great, even if we can't use it now. Hopefully somebody could change the code of ES somehow to make use of it. We had to break the synopsis down from one master file to individual games because on large systems the file was insanely huge and was causing memory issues. Hell... if you look at my 2600 xml it's almost 900kb and that's without any of the game data in them. My guess is just for 700 games that file is easily going to be 3MB when its done.

Not every game is going to have every tag. My original synopsis file years ago had every tag for every game, but they made an online editor that the community could use and the individual game files that it creates spit out only the fields that had info in them. Sadly, the people who developed it are gone now so I can't change that.

The last line "usually" contains an info source reference. I made that a big rule of mine when I was creating the system that if we took info from a website to have a link to the page it came from. I can't say for sure it was always followed, and sometimes there was no info found online and people wrote up their own stuff with no reference link.

customizable folder paths would be good. Not only for me, but for people using this for their own stuff in the future too. For me it's good because the media/roms folders are going to be different from system to system and it would not require you to build alternate versions for different systems or me to try to go in and change things and probably break it.

Yes, the rom name is not in the TXT file. The way our system worked was that it would find the matching "filename.txt" when you pulled up the game information.

The problem with the Case Issue is that some games are named, for example "Attack of the Mutant Camels", but the artwork is named "Attack Of The Mutant Camels". It's usually not an either/or situation. If we can't correct for that, I think I'd rather just have it use the exact case of the game and I'll check that everything works by hand afterward and change the file accordingly. If possible, I want to maintain the existing file structure as is, so people don't have to re-download the entire thing or have two copies of it. I might change my mind on this part. I'll have to think about it while you're gone.

Thanks for the help. Looking forward to seeing it in action. Let me know if I can clear anything else up.

herb_fargus

Camel casing articles is a very simple process

UDb23

@herb_fargus said in Making premade gamelist.xml with xtra media collections:

Camel casing

Didn't know that it's called that way. :-))

UDb23

@Used2BeRX Everything clear so far. Will keep you updated.

Used2BeRX

@UDb23 Sweet.

There was a little miscommunication with the other guy that was helping me work something out and he created a python script for Linux that will make individual "game".xml files with tags for all of the game info and put the 10 file tags above it. It actually looks for the media if it's in the proper folders so it puts the proper extension too. Pretty cool, but I'm a Linux dummy and the only way I could really use it is in the Linux console with the Pi. The problem is I've got the Pi Zero with a small SD card so I couldn't really make use of it that way.

I've got really high hopes for what you're talking about doing here. Maybe if we iron out all of the kinks it could be actually turned into that executable you had mentioned as well. I'm thinking that what you're doing could be a valuable tool for anybody with Extra media for their games, even if they're not using my setup.

I've taken your advice and stepped away from this for a while, until you come back with something. I probably could have gotten the first half of the NES gamelist.xml done yesterday if I worked on it all day, but I'd be kicking myself for it once I had your program to work with I'm sure.

Let me know! Thanks.

cafarellidigital

I just wanted to show my support for this project, for all of the work that you've done so far, and I look forward to see where it goes in the future!

Used2BeRX

@cafarellidigital Thanks man. I think this is going to be pretty sweet when it's finished.

Used2BeRX

@UDb23 hey man. Was just wondering if you had any time to work on this at all? I'm pretty busy with some other stuff at the moment, so I'm not rushing you or anything. Just don't want to lose touch with the only guy who seems to have a plan on how to do this.

UDb23

@Used2BeRX Don't worry. ;-)
As promised I started working on it. In the end I decided to go directly for making an executable program (no need for excel..).
Downloaded and installed latest Visual Studio community 2017; took me some time to get acquainted with it.
So far I made a "skeleton" main program with functions to access file system and to do string manipulation. Tomorrow I hope to find time to write the code for the "conversion" from txt to xml (should be easy at this point).

Used2BeRX

@UDb23 Sweet. Just let me know if you need any more input before making the first compile. Otherwise I should be around to take it for a spin whenever you've got it ready.

Used2BeRX

@UDb23 Since I haven't heard back from you yet I figure you're still putting it together. I was hoping you could add a field in the first 10 <marquee></marquee> I don't have any marquee images myself, but some guys I know that are making a skin are using that field and making marquee style images by cropping from title screens.

UDb23

@Used2BeRX Sure I can add this additional tag.
Program development is progressing and I'll provide you a first working version over the weekend.

Used2BeRX

@UDb23 Nice.

I'm going to ask around here and see if I can get some answers on memory issues. I actually "broke" the Pi Zero with just the hand made XML files for only the 4 Atari systems. EmulationStation was taking 2-3 minutes to load and some emulators stopped functioning altogether. I'm just wondering if loading pointers to data that doesn't exist is draining memory resources or not. And I'm also wondering if every single picture you load is draining memory at one time. Seems terribly inefficient to me. I'll be able to test our work here one system at a time, but then I have to rename the XML file so it doesn't get loaded when I go on to the next system.

Anyways, maybe down the road there can be options made in your program to expand or limit exactly what fields the program adds to the XML file depending on what I find out about how things are loaded.

UDb23

@Used2BeRX First beta of the Application to convert your .txt to .xml is ready.
You can find it here.
Just unzip the MetaX.zip and launch Setup.exe.

Use the "Browse" button in the App to select the folder that contains the txt files.
Currently only one by one conversion works: select the file and click "convert" button.

I found that the game metadata source txt files are not consistent meaning some info may or may not be included; also same info (e.g. "developer" can be in different line in one txt and in another line in a different txt).

The application tries to find the specific "tags" regardless of line order; it also identifies the "description" by selecting the lines after the line with "________".

Could you let me know the complete list of "items/tags" the App should:

look for in the source txt files
write to the .xml files (on top of items found in the .txt)

Just want to first make sure everything is converted correctly as single files, and then implement "convert all" and "create global gamelist.xml" functionality.

App requires .net libraries installed (created in VB with Visual Studio community 2017); I suppose it will ask during installation (can't test on another PC).

note: if you convert the same txt multiple times, it will append lines to already existing corresponding .xml (we could call it a "known bug" ;-) ).

Note 2: App does not do anything else as converting txt to xml so it should be totally harmless. In any case keep a backup of your original txt files.

Used2BeRX

@UDb23 Awesome man. Thanks.

I downloaded it, but haven't had time to use it. I should have some feedback for you tomorrow or Tuesday night at the latest. Thanks so much for this again.

genericuserdude

@UDb23 I somehow got banned earlier this morning with no reason. I think it was a mistake. Just wanted to post here that I'm working on trying to get to the bottom of this. If somebody has it out for me for some reason and I get banned with this ID as well, please reach out to me as I think we're on to something really good here and I don't have any way of keeping contact with you right now. I couldn't figure out a way to send a PM, so here's the hail mary.

Please contact me on this online document if the ban doesn't get lifted until we come up with something better: https://docs.google.com/document/d/18cfXi3eO7Gj1piWbzans8zS5CQ2ATaUrxeM0jmdzal8/edit?usp=sharing

UDb23

@genericuserdude ok.
Maybe you should ask @Global-Moderators why this happened.

genericuserdude

@UDb23 said in Making premade gamelist.xml with xtra media collections:

@Global-Moderators

Okay. At least you and the other guy I have been working on another project has seen the link. Please save it in case I get banned again and they remove the posts so we can continue our work.

@Global-Moderators Could somebody please tell me what is going on with the ban?

mediamogul

@genericuserdude

Could somebody please tell me what is going on with the ban?

I just checked the ban history for @Used2BeRX and it says "no reason given" which is not usually done. Also, after a quick look over the recent post history for that account, I don't see anything that should be offensive. I guess it could have been a mistake, but it's a permanent ban and that takes a small amount of extra doing. It can still be reversed, but I'd like to see if there was any purposeful intention from any of the other moderators or admins first.