Making premade gamelist.xml with xtra media collections

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm looking for a way to automate the first 10 lines for the different items for every game.

What exactly should the "automation" code do ?

Then I'm looking for a way to "scrape" the data from individual text files that are named the same as the rom file names to get all of the game synopsis data. Here is an example of the synopsis file called "Astro Chase.txt".

Generating ES gamelist compliant xml file data based on these txt files could be done quite easily.

Still using these complete system gamelists would make sense if you have all roms for that systems (and with the same rom filenames).
Not sure how many people will have them all installed (even if occupied sd space will be limited I imagine).
Otherwise it would require a "scraper" to pick just the needed rom data out of your global XML based on actual roms installed; and that's not that easy to do.

herb_fargus

@Used2BeRX the idea of having preconfigured metadata packs that only work based on specifically named romsets doesn't really sit well with me in the sense that it tacitly implies recommending people go and find the romsets that match.

Rather I still think it would be of more benefit to the community to take the metadata and put it on one of the sources that are being scraped as many people including myself don't use full romsets.

It may be worth discussing with @screech as he maintains the screenscraper site and he may be able to help find a way at automating an import of the metadata you have, then everyone would be able to use it and generate gamelists that match the romsets they have.

Used2BeRX

@UDb23 Hey man. Nice to hear from you.

What exactly should the "automation" code do ?

The automation code, for the first 10 lines of every game, would create the lines you see in the example just with a different filename for each <game></game> entry. Not every single game has every single file associated with it, but it doesn't hurt anything to have the XML call for it when there's nothing in that folder. The location of these files would be exactly the same from system to system except the SYSTEM folder would be different of course.

Our biggest problem with making a code that automatically creates these 10 entries without looking for files matching the rom name first though is that the original setup did not take "case sensitivity" into account since the system it was designed for didn't care about that. Most files will match case, but some won't. If it's just a basic code to make the 10 lines to match the rom file name without actually looking for the data first, everything would need to be verified by hand to make sure that it is working right.

Example: "Astro Chase.bin" was the rom name. The txt file name was "astro chase.txt".

Generating ES gamelist compliant xml file data based on these txt files could be done quite easily.

I LOVE hearing this. :)

Still using these complete system gamelists would make sense if you have all roms for that systems (and with the same rom filenames).
Not sure how many people will have them all installed (even if occupied sd space will be limited I imagine).
Otherwise it would require a "scraper" to pick just the needed rom data out of your global XML based on actual roms installed; and that's not that easy to do.

I can provide data files for Romcenter that would rename your roms, assuming you have the right ones of course. I'm not sure if a scraper would be necessary like you mentioned, but I was wondering if there was a way to have a separate XML file for each game that is referenced in a master XML file instead of putting them all into one file. We had to do that with the TXT files ourselves. Smaller systems like Atari weren't a problem, but when you had systems like the GBA the master information file was getting so big that it was causing a lot of problems.

Perhaps if the games were in separate XML files it would be easier only to use the ones that have matching roms? I don't know.

I assume that not everybody who had these packages would keep everything inside of each one. There are a lot of prototype games, for example, that were included for historical purposes but really provide no entertainment value. They're easily avoided in the setup since everything is cataloged in sub-folders, but for space reasons people might manually remove these.

Please let me know what questions you have, and if I can further clarify anything. Thanks for the reply and I hope to hear back from you soon.

Used2BeRX

@herb_fargus I appreciate what you're saying, but I have no interest in spending any time re-designing this stuff to conform to other peoples stuff. I spent 5 years of my life putting it together with other people online and concluded my own work on it back around 2010. 7 years later people are still working on it. Everything we have is superior to any database that is scrapable online by a factor of 10.

That being said, what we have is very worthwhile to anybody willing to put effort into it. If people don't want to put that effort into it, they can continue to use what they've been using. I'm not getting a commission on this, so I have no stake in what people choose to do.

screech

I'm agree with @herb_fargus ^^

Making a Full pre-scraped folder is too "huge" for most of the user.
That's why "scraper" were invented ;) to catch only the info you need ;)
( It's a green way of thinking ^^ don't download 3Tb Zip if you need only 2 files ^^)

In the other way, offering your work to the comunity is cool ^^

Maybe you can visit : www.screenscraper.fr a lot's of metadata and media are already sorted/cleaned/referenced/... you can use it for your project if you want (it's Creative Comon Licence ;) )
But the best is the reverse way ;) don't hesitate to offer your work on the existing DB so every one can enjoy it with Scraper.

We have some existing process to make "massive upload" ;) so it's possible to use it for your work ;)

herb_fargus

@Used2BeRX I appreciate the effort spent in compiling the data base, and from the pieces I've looked at it looks like it's more or less standardised to some extent.

If there was a GitHub repository or database of sorts sans ROMs. I'm sure we could work some scripting magic out to adapt all the information into a central online database like screenscraper.

Obviously my primary interest is in making the metadata as easily accessible as possible. As I have free time I wouldn't mind taking a look to see what options there are.

Used2BeRX

@herb_fargus I'm okay with the stuff being put up here in a way where it could easily be scraped. I just don't have the time or desire to do that myself. I understand the no roms rule, but as you can imagine that puts a lot of unwanted burden on my end since I cannot just throw out a link and let you guys sort that out the way you want it done.

All I need personally is a way to automate the XML file creation which would save me probably 1,000 worth of manual work and would instantly "convert" the existing work for use on the Pi. I'm happy to share everything, but I put in more than my share of time years ago and pretty much every second of free time in my mid 20's doing it. I can't get that deep into it again all these years later.

As long as credit is given to the appropriate parties, I have no problems with the idea of people using our stuff here to create a superior scraper system.

Used2BeRX

I decided to make a 3rd upload for the 7800 system that is a lot smaller so maybe more people might try it out. I removed the game manuals and gamefaqs from this version since they aren't currently usable, and I also removed the WMV movies. They are still available in the link above if you want them.

This version is only about 25mb instead of over nearly 400mb.

https://mega.nz/#!pepgES5B!ptltqCN9QXaSTyeNbywN8qbOiIYoL5BDwIgtyb1tZaA

Keep in mind that these are the small images. Larger original images are also available for all systems, although some images for more obscure games were just pretty small to begin with, so not all images would be improved with the "HD" packs.

Maybe with such a small download more people will at least look at this example.

Used2BeRX

@UDb23 I've actually changed my mind about what I'm looking for here UDb23. I think I'm going to go ahead and just hand make the gamelist.xml files that have just the files and their locations as well as the title of the game. That way it will display the artwork and videos properly, and I will have to verify by hand that the case sensitivity is addressed on an individual basis anyways.

What would be a HUGE help to me is if there was a way we could make individual game.xml files based off of the txt files. Then, if there was a way that the "master" gamelist.xml could reference these instead of adding them to the master file.

Doing that first part by hand actually doesn't take that long. It's just a TON of copying and pasting and then finding and replacing. It sucks for sure, but at least when I do that I can verify personally that it's all done right.

What DOES take a ton of time on my end is filling in all of the information tags from the txt files. I'm looking at my Atari 2600 collection right now and if I did that I would have to copy information for over 700 games individually, which is more than double of what I had done for 3 other systems combined.

Worst case scenario, if I can't get any help here and the other guy falls through, I can at least have a collection that will work in "no-metadata" skins that will display whatever artwork I choose or a movie.

UDb23

@Used2BeRX if you have the single text files available I can create a program that reads the txt and generates corresponding XML.
Did something similar in the past; ideas is it would convert all txt files it finds in a specific folder.
Would make that in VBA meaning you need Excel to run it ( just to run, no need to do any data import in excel).
Could also do it in VB and compile it into an executable; this would require more time as I don't have VB installed and haven't been using it for years..
Let me know if excel VBA could be ok.

Concerning global gameslist XML referencing to single game (sub) xmls I don't know if this is supported by ES.

UDb23

@Used2BeRX Really interested in your work on Atari 2600 gameslist especially if you can provide high quality cover images.
Also interested in building Vic20 gameslist.
Wrote my own mobygames scraper for that and downloaded all metadata from the site. Have found separate hq cover images to be used in the gamelist XML. Will combine them sometime.
Currently priority is on building my first control panel. Correctly drilling holes in plywood board isn't as simple as I thought :-)

Used2BeRX

@UDb23 Here's a download for the synopsis files for the individual 2600 games. If you can do something with this, I'd be happy to zip up a package with all the other systems. These are all named exactly the same as the roms are, except of course the extention is .txt instead of .zip.

Synopsis 2600: https://www.sendspace.com/file/mgf38h

And here's an XML file for the 2600 I'm currently working on. It has the 10 files for each game already made, as well as the <name></name> field filled in (although I will have to go back and edit some of them since I'm not actually ripping that from the txt file and I'm just doing a copy/paste job.

2600 gamelist.xml wip: https://www.sendspace.com/file/km2sf2

As you can see from the parts of the XML that are completed, the file names for the Extras for the system are named exactly the same as the rom itself, so everything should work right... unfortunately, the system we did it on originally didn't care about case, so chances are some things in there are broke because the case doesn't match. That's the stuff that I would have to fix by hand after the files were created.

Excel isn't a problem for me. I don't have it installed right now, but I do have a copy of 2007 lying around here somewhere. Hopefully a version that old will still allow me to use your script.

No need to make a VB program. It's not that I don't have time to wait, but I'm really OCD-ing out here and I'm manually doing things right now I said that I wasn't going to do because I'm waiting for somebody to automate this and patience isn't one of my virtues ;)

Looking forward to see what you can come up with. Chances are the xml I'm working on won't be usable with what you make, but that still isn't enough to make me stop working on it.

I've got good Atari 2600 artwork. There is at least a Box Front and Action shot for every game. Not a lot of title shots since a lot of games didn't have title screens and probably 100-150 carts missing.

For the boxes that we couldn't find or were never made, we actually had a very talented graphic artist on "staff" that created alternate boxarts for prototypes, hacks, translations, unlicensed and other categories.

Looking forward to seeing what you can do man. You're the only hope for my sanity at this point. :)

EDITED TO ADD: Now that I think of it, it might be good if there were two different scripts. One script that would create all of the fields/tags like the XML example I gave you, and then a second one that would "rip" the information from the TXT files and fill in the empty tags made by the first file.

That would make sure that work I've already done on the first part would still be usable by the time you make this, but it would also allow me to make sure the files are all working (because of the case issue) before ripping all the data. Any chance we could do that? Maybe that's easier to make it a two step process anyway?

You may have noticed that the path is not just "./" There are many sub folders for a lot of the systems, so the script should be able differentiate that in the path folder based off of which folder the roms are in. I'm making them in alphabetical order, grouped together in the folders. (Maybe you should see this XML when it's completed to fully understand what I'm talking about). I'll upload it as soon as it's done. I'm hopped up on coffee right now so it shouldn't be long :)

Used2BeRX

@UDb23 Well..... that was horrible. Just the 1st part took me about 5 1/2 hours, so hopefully that's the last system I'm doing that for with your help man.

https://www.sendspace.com/file/modd8g

I'll probably still be editing that a bit, but I did verify that all the boxart works with it. There were about 4 lines in there that I had to change because the artwork was a different case than the rom. Not too bad when it was over 700 entries. I didn't check any of the other artwork or videos yet.

Let me know what questions you've got.

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm really OCD-ing out here and I'm manually doing things right now

You shouldn't invest time in manually editing those files ! ;-)

Excel 2007 is fine to run my VBA code.
Let's proceed in steps with development:

Create program to parse existing txt files and generate separate "per game" xml files + global gamelist.xml.
Parsing requires source txt to have a defined structure.
Looking at those TXTs the first part is ok (assigning specific fields/tags value).
Then, after the"----------" there's the description. Last line always contains info source reference. Is that correct ? if so I will use these "rules" to parse the files.
Add customizable folder paths (preference settings) that will be used during xml generation. In this way people can have supporting files in the folder they prefer.

If I understood correctly for each game the rom name has to be added as it is not contained in the TXT file. All extras (boxart, video etc) are then named the same as the rom but, of course, with different extension and path.
This can be generated too by the program.

For existing XMLs (work you have already done) the program would read-in just the rom names and combine them with corresponding TXT data and generate XML.

I suppose some games will not have some of the extras, so generating ALL EXTRA tags would actually create some "broken links" to missing data. Not sure if that would make ES "unhappy" and generate an error or if ES just ignores this.

Concerning the case issue the program could simply generate all lower or all upper case.
In this way you just need to "case convert" all actual extra files accordingly. I'm pretty sure free windows utilities exist to do this.

I'll be away over the next days but can start working on it this weekend.

Used2BeRX

@UDb23 Hey man. Thanks again.

ES doesn't freak about links to files that aren't there. Many of the games don't have all of the artwork, especially the title shots on systems like the 2600 that didn't have many titles. I don't actually put any of the other media on the Pi Zero I'm working on either, so there have been no errors popping up for missing videos or manuals, etc. Now is there memory being burned up using tags with no file? That's above my pay grade. :)

Haha! Yeah, I know I shouldn't be manually making them. I thought that the first part was a lot quicker than it actually was. I couldn't believe I put 5 1/2 hours more into the 2600 until I checked the time when I was done.

As for your "steps with development"

A per game xml is great, even if we can't use it now. Hopefully somebody could change the code of ES somehow to make use of it. We had to break the synopsis down from one master file to individual games because on large systems the file was insanely huge and was causing memory issues. Hell... if you look at my 2600 xml it's almost 900kb and that's without any of the game data in them. My guess is just for 700 games that file is easily going to be 3MB when its done.

Not every game is going to have every tag. My original synopsis file years ago had every tag for every game, but they made an online editor that the community could use and the individual game files that it creates spit out only the fields that had info in them. Sadly, the people who developed it are gone now so I can't change that.

The last line "usually" contains an info source reference. I made that a big rule of mine when I was creating the system that if we took info from a website to have a link to the page it came from. I can't say for sure it was always followed, and sometimes there was no info found online and people wrote up their own stuff with no reference link.

customizable folder paths would be good. Not only for me, but for people using this for their own stuff in the future too. For me it's good because the media/roms folders are going to be different from system to system and it would not require you to build alternate versions for different systems or me to try to go in and change things and probably break it.

Yes, the rom name is not in the TXT file. The way our system worked was that it would find the matching "filename.txt" when you pulled up the game information.

The problem with the Case Issue is that some games are named, for example "Attack of the Mutant Camels", but the artwork is named "Attack Of The Mutant Camels". It's usually not an either/or situation. If we can't correct for that, I think I'd rather just have it use the exact case of the game and I'll check that everything works by hand afterward and change the file accordingly. If possible, I want to maintain the existing file structure as is, so people don't have to re-download the entire thing or have two copies of it. I might change my mind on this part. I'll have to think about it while you're gone.

Thanks for the help. Looking forward to seeing it in action. Let me know if I can clear anything else up.

herb_fargus

Camel casing articles is a very simple process

UDb23

@herb_fargus said in Making premade gamelist.xml with xtra media collections:

Camel casing

Didn't know that it's called that way. :-))

UDb23

@Used2BeRX Everything clear so far. Will keep you updated.

Used2BeRX

@UDb23 Sweet.

There was a little miscommunication with the other guy that was helping me work something out and he created a python script for Linux that will make individual "game".xml files with tags for all of the game info and put the 10 file tags above it. It actually looks for the media if it's in the proper folders so it puts the proper extension too. Pretty cool, but I'm a Linux dummy and the only way I could really use it is in the Linux console with the Pi. The problem is I've got the Pi Zero with a small SD card so I couldn't really make use of it that way.

I've got really high hopes for what you're talking about doing here. Maybe if we iron out all of the kinks it could be actually turned into that executable you had mentioned as well. I'm thinking that what you're doing could be a valuable tool for anybody with Extra media for their games, even if they're not using my setup.

I've taken your advice and stepped away from this for a while, until you come back with something. I probably could have gotten the first half of the NES gamelist.xml done yesterday if I worked on it all day, but I'd be kicking myself for it once I had your program to work with I'm sure.

Let me know! Thanks.

cafarellidigital

I just wanted to show my support for this project, for all of the work that you've done so far, and I look forward to see where it goes in the future!