Making premade gamelist.xml with xtra media collections

Used2BeRX

I just completed a gamelist.xml for the Atari 5200/800 that includes just over 230 games. It references the Boxart, Cartridges, Title screenshots, Action screenshots, video previews, manuals, gamefaqs and vgmaps for every game, as well as a detailed synopsis with many more tags than are currently used by any of the RetroPie skins. To see the full gamelist.xml, you can download it here.

https://www.sendspace.com/file/llqre7

Here is a sample game entry from the file:

	<game>
		<path>./(1) Atari 5200/(1) Licensed/Astro Chase.bin</path>
		<image>/home/pi/RetroPie/Media/ATARI800/Artwork/Box Front/Astro Chase.jpg</image>
		<cart>/home/pi/RetroPie/Media/ATARI800/Artwork/Cart/Astro Chase.jpg</cart>
                <title>/home/pi/RetroPie/Media/ATARI800/Artwork/Titles/Astro Chase.jpg</title>
                <action>/home/pi/RetroPie/Media/ATARI800/Artwork/Action/Astro Chase.jpg</action>
                <threedbox>/home/pi/RetroPie/Media/ATARI800/Artwork/3D Boxart/Astro Chase.png</threedbox>
                <video>/home/pi/RetroPie/Media/ATARI800/Movies/Astro Chase.wmv</video>
                <gamefaq>/home/pi/RetroPie/Media/ATARI800/GameFAQs/Astro Chase.zip</gamefaq>
                <manual>/home/pi/RetroPie/Media/ATARI800/Manuals/Astro Chase.zip</manual>
                <vgmap>/home/pi/RetroPie/Media/ATARI800/VGMaps/Astro Chase.zip</vgmap>
		<name>Astro Chase</name>
		<platform>Atari 5200</platform>
		<region>USA</region>
		<media>Cartridge</media>
		<controller>5200 Joystick</controller>
		<genre>Shoot 'em Up - Multi-Directional</genre>
		<gametype>Licensed</gametype>
		<releasedate>1983</releasedate>	
		<developer>First Star Software</developer>
		<publisher>Parker Brothers</publisher>
		<players>1</players>
		<programmer>Fernando Herrera</programmer>
		<desc>As pilot of a spaceship, your job is to save the planet Earth! Eight mega mines are slowly homing in, and you need to destroy them all before they reach Earth. Trying to stop you are a variety of flying saucers which will constantly be chasing you around the planet.

Your ship is equipped with lasers which can be used to destroy enemy saucers and the mega mines. Located throughout the space around Earth are eight shield depots; if you fly through one of these, you will temporarily become invincible. Your spaceship also has a limited energy supply; if you get too low on energy, it can be resupplied at one of the generators located in the four corners of the play area. When all of the mega mines have been destroyed, you will move on to the next, faster chase!

http://www.mobygames.com/game/atari-5200/astro-chase
		</desc>		
	</game>

It took me about 25 hours to write this file. Knowing what I know now it would have gone quicker, but it's still a ton of work that doesn't need to be done. I don't know how to write code to automate this though.

I'm looking for a way to automate the first 10 lines for the different items for every game.

Then I'm looking for a way to "scrape" the data from individual text files that are named the same as the rom file names to get all of the game synopsis data. Here is an example of the synopsis file called "Astro Chase.txt".

Astro Chase
Platform: Atari 5200
Region: USA
Media: Cartridge
Controller: 5200 Joystick
Genre: Shoot 'em Up - Multi-Directional
Gametype: Licensed
Release Year: 1983
Developer: First Star Software
Publisher: Parker Brothers
Players: 1
Programmer: Fernando Herrera
_________________________

As pilot of a spaceship, your job is to save the planet Earth! Eight mega mines are slowly homing in, and you need to destroy them all before they reach Earth. Trying to stop you are a variety of flying saucers which will constantly be chasing you around the planet.

Your ship is equipped with lasers which can be used to destroy enemy saucers and the mega mines. Located throughout the space around Earth are eight shield depots; if you fly through one of these, you will temporarily become invincible. Your spaceship also has a limited energy supply; if you get too low on energy, it can be resupplied at one of the generators located in the four corners of the play area. When all of the mega mines have been destroyed, you will move on to the next, faster chase!

http://www.mobygames.com/game/atari-5200/astro-chase

I've got a similar setup for nearly every console system up through 16bit. I'd rather not have to write this all by hand cause it would probably take me years to do. I was hoping some of the coding wizards here might be able to write a script to automate this for me. Preferably something that could be used in Windows.

All of this data has been scraped and collected years ago by a group I was a part of. With a small alteration of the file system we established and the XML files to get the data to load in RetroPie it is all easily adapted to the Pi from the system we originally made it for.

Any help would be greatly appreciated.

Thanks!

EDITED TO ADD: I made 4 alternate versions of the XML to test the other images in the collection. They changed the "cart", "titles", etc tags to <image> so they would show up in the skins and they all worked great.

Skins could probably be made to easily change from one type of artwork preview to another or to even show multiple previews at once, or a piece of art as well as a video at the same time.

I'm not exactly sure how the RetroPie could take advantage of the Manuals and Gamefaqs zips, but they would all be there and pointed to in the XML for each game. Our existing system is able to easily load the manuals and gamefaqs for every game that has them within the in-game front end menus.

herb_fargus

@Used2BeRX might be worth adding to thegamesdb so it's in a common place anyone can access with the scraper

Used2BeRX

@herb_fargus We've actually got everything uploaded, but it's not presented like GamesDB or other sites. There are packages that have been out there for about 8 years now too that in total are easily over 100GB, not including PSX or N64. Unfortunately, I can't actually give any links to anything because roms are included.

My plan here, once I've read the rules of course, is to put together a few small packages that anybody would be able to put in the proper place on their SD card that coupled with the XML files would run right out of the box. The only thing in these packs that would not be provided are the roms (I don't imagine anything else would be against site rules, but I will make sure before releasing them). If it's not against the site rules, I would also create a datfile that would have the names necessary to match everything in the XML.

My hope is that if people get a taste of these smaller systems and what we've got to offer on our side that I could get some help automating the XML process for the bigger and better systems. Also, I think that with these sets pre-made it would open up skin development to be more creative with multiple previews showing at the same time like the emulators we run.

Maybe we could even open up the ability to pull up Gamefaqs and manuals while in the game like we're able to do.

I'm no coder though. I'm just an OCD mule who had stepped away from the scene for a long time. Most of the guys I did work with have moved on years ago and we don't get the traffic and likely the potential for true talent that you guys are getting right now.

I'm looking forward to working with you guys, and I'll do everything in my power not to break any of the rules while sharing.

Used2BeRX

Here's a beta build of the Atari 7800 setup.

https://mega.nz/#!QKxmlCYL!z5ck_O7L7pNzL19Z4HdMZ64FikmYGC7J1mpfXOGvEsk

The testing was done on a Pi Zero, so the artwork is fairly small. I do have much more high quality artwork for all of the packs that I could include in future releases.

I'd like to know what you guys think about this and especially how the video previews work. Since I had the Pi Zero the videos would load, but they only played a second or two before freezing while the audio played. If you have a Pi Zero you're going to want to delete the videos, or at least use one of the many skins that don't support them. The Carbon skin prioritizes videos, so it will automatically show them if you have them in the "Media\A7800\Movies" folder in this setup.

As promised, the Roms and BIOS have been stripped from this release. I provided datafiles that work in RomCenter. There is one datafile per folder, and the roms for each datafile need to be in the exact folder otherwise the XML file won't work for the game.

Please, moderators, let me know if there is anything else that you find objectionable in this release and I will take it down immediately upon getting the notification (or feel free to remove it yourself, but please let me know why). There wasn't much in the rules about this sort of thing except for not uploading or linking to any copyrighted material. I believe this meets those requirements.

I hope this does a good job showing the potential of these setups, and I hope that some coder types in here see the benefits and can put together something to automate this process so I can quickly make more releases.

Used2BeRX

I decided that I would work on a Lynx collection next since the library is so small. While I've been testing games, I noticed that some of them are rotated incorrectly. I will spend the time to rotate them as well as change the controller configurations of the D-Pad accordingly so the games all work right with the right rotation.

I was wondering, which files do I need to copy and provide with the LYNX pack that will have all of these changes, and where are they located. Thanks in advance.

Used2BeRX

Is there a better place or a better way for me to provide this stuff here? I get the feeling that nobody knows that I put the one release out yet.

herb_fargus

@Used2BeRX Retroarch has a repository for thumbnails with similar type assets along with database files perhaps you could do something similar:

https://github.com/libretro/libretro-thumbnails

I made a simplified version for mame.

https://github.com/HerbFargus/Mame-Extras

I set up the paths in the xml to be relative so that the gamelists and snaps are stored in the same folder as the roms so its much more portable.

Other thing is not a whole lot of people use the atari 5200 or lynx so that may also be part of it.

Used2BeRX

I'll consider that, but probably not worth the effort especially since you think nobody is interested in the Atari systems.

I'm not willing to do the better systems because they're so huge, and I'm not going to do all of that work by hand. I've got somebody who says that they're going to try to code something for me on the other site, but they're not a code hero so I'm not expecting anything quick. I'm hoping to get somebody here ineterested that can get this done. It's foolish to be scraping things off of websites that often have tons of missing information and inferior artwork when we already have a great collection.

I'm willing to share it with the community here if I get some help putting this together and automating a lot of the work, but if somebody from the other site gets to it first than I'm just going to keep our stuff over there and release the full packs with roms there because there are no rules against it there.

Here's hoping somebody here can come up with something.

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'm looking for a way to automate the first 10 lines for the different items for every game.

What exactly should the "automation" code do ?

Then I'm looking for a way to "scrape" the data from individual text files that are named the same as the rom file names to get all of the game synopsis data. Here is an example of the synopsis file called "Astro Chase.txt".

Generating ES gamelist compliant xml file data based on these txt files could be done quite easily.

Still using these complete system gamelists would make sense if you have all roms for that systems (and with the same rom filenames).
Not sure how many people will have them all installed (even if occupied sd space will be limited I imagine).
Otherwise it would require a "scraper" to pick just the needed rom data out of your global XML based on actual roms installed; and that's not that easy to do.

herb_fargus

@Used2BeRX the idea of having preconfigured metadata packs that only work based on specifically named romsets doesn't really sit well with me in the sense that it tacitly implies recommending people go and find the romsets that match.

Rather I still think it would be of more benefit to the community to take the metadata and put it on one of the sources that are being scraped as many people including myself don't use full romsets.

It may be worth discussing with @screech as he maintains the screenscraper site and he may be able to help find a way at automating an import of the metadata you have, then everyone would be able to use it and generate gamelists that match the romsets they have.

Used2BeRX

@UDb23 Hey man. Nice to hear from you.

What exactly should the "automation" code do ?

The automation code, for the first 10 lines of every game, would create the lines you see in the example just with a different filename for each <game></game> entry. Not every single game has every single file associated with it, but it doesn't hurt anything to have the XML call for it when there's nothing in that folder. The location of these files would be exactly the same from system to system except the SYSTEM folder would be different of course.

Our biggest problem with making a code that automatically creates these 10 entries without looking for files matching the rom name first though is that the original setup did not take "case sensitivity" into account since the system it was designed for didn't care about that. Most files will match case, but some won't. If it's just a basic code to make the 10 lines to match the rom file name without actually looking for the data first, everything would need to be verified by hand to make sure that it is working right.

Example: "Astro Chase.bin" was the rom name. The txt file name was "astro chase.txt".

Generating ES gamelist compliant xml file data based on these txt files could be done quite easily.

I LOVE hearing this. :)

Still using these complete system gamelists would make sense if you have all roms for that systems (and with the same rom filenames).
Not sure how many people will have them all installed (even if occupied sd space will be limited I imagine).
Otherwise it would require a "scraper" to pick just the needed rom data out of your global XML based on actual roms installed; and that's not that easy to do.

I can provide data files for Romcenter that would rename your roms, assuming you have the right ones of course. I'm not sure if a scraper would be necessary like you mentioned, but I was wondering if there was a way to have a separate XML file for each game that is referenced in a master XML file instead of putting them all into one file. We had to do that with the TXT files ourselves. Smaller systems like Atari weren't a problem, but when you had systems like the GBA the master information file was getting so big that it was causing a lot of problems.

Perhaps if the games were in separate XML files it would be easier only to use the ones that have matching roms? I don't know.

I assume that not everybody who had these packages would keep everything inside of each one. There are a lot of prototype games, for example, that were included for historical purposes but really provide no entertainment value. They're easily avoided in the setup since everything is cataloged in sub-folders, but for space reasons people might manually remove these.

Please let me know what questions you have, and if I can further clarify anything. Thanks for the reply and I hope to hear back from you soon.

Used2BeRX

@herb_fargus I appreciate what you're saying, but I have no interest in spending any time re-designing this stuff to conform to other peoples stuff. I spent 5 years of my life putting it together with other people online and concluded my own work on it back around 2010. 7 years later people are still working on it. Everything we have is superior to any database that is scrapable online by a factor of 10.

That being said, what we have is very worthwhile to anybody willing to put effort into it. If people don't want to put that effort into it, they can continue to use what they've been using. I'm not getting a commission on this, so I have no stake in what people choose to do.

screech

I'm agree with @herb_fargus ^^

Making a Full pre-scraped folder is too "huge" for most of the user.
That's why "scraper" were invented ;) to catch only the info you need ;)
( It's a green way of thinking ^^ don't download 3Tb Zip if you need only 2 files ^^)

In the other way, offering your work to the comunity is cool ^^

Maybe you can visit : www.screenscraper.fr a lot's of metadata and media are already sorted/cleaned/referenced/... you can use it for your project if you want (it's Creative Comon Licence ;) )
But the best is the reverse way ;) don't hesitate to offer your work on the existing DB so every one can enjoy it with Scraper.

We have some existing process to make "massive upload" ;) so it's possible to use it for your work ;)

herb_fargus

@Used2BeRX I appreciate the effort spent in compiling the data base, and from the pieces I've looked at it looks like it's more or less standardised to some extent.

If there was a GitHub repository or database of sorts sans ROMs. I'm sure we could work some scripting magic out to adapt all the information into a central online database like screenscraper.

Obviously my primary interest is in making the metadata as easily accessible as possible. As I have free time I wouldn't mind taking a look to see what options there are.

Used2BeRX

@herb_fargus I'm okay with the stuff being put up here in a way where it could easily be scraped. I just don't have the time or desire to do that myself. I understand the no roms rule, but as you can imagine that puts a lot of unwanted burden on my end since I cannot just throw out a link and let you guys sort that out the way you want it done.

All I need personally is a way to automate the XML file creation which would save me probably 1,000 worth of manual work and would instantly "convert" the existing work for use on the Pi. I'm happy to share everything, but I put in more than my share of time years ago and pretty much every second of free time in my mid 20's doing it. I can't get that deep into it again all these years later.

As long as credit is given to the appropriate parties, I have no problems with the idea of people using our stuff here to create a superior scraper system.

Used2BeRX

I decided to make a 3rd upload for the 7800 system that is a lot smaller so maybe more people might try it out. I removed the game manuals and gamefaqs from this version since they aren't currently usable, and I also removed the WMV movies. They are still available in the link above if you want them.

This version is only about 25mb instead of over nearly 400mb.

https://mega.nz/#!pepgES5B!ptltqCN9QXaSTyeNbywN8qbOiIYoL5BDwIgtyb1tZaA

Keep in mind that these are the small images. Larger original images are also available for all systems, although some images for more obscure games were just pretty small to begin with, so not all images would be improved with the "HD" packs.

Maybe with such a small download more people will at least look at this example.

Used2BeRX

@UDb23 I've actually changed my mind about what I'm looking for here UDb23. I think I'm going to go ahead and just hand make the gamelist.xml files that have just the files and their locations as well as the title of the game. That way it will display the artwork and videos properly, and I will have to verify by hand that the case sensitivity is addressed on an individual basis anyways.

What would be a HUGE help to me is if there was a way we could make individual game.xml files based off of the txt files. Then, if there was a way that the "master" gamelist.xml could reference these instead of adding them to the master file.

Doing that first part by hand actually doesn't take that long. It's just a TON of copying and pasting and then finding and replacing. It sucks for sure, but at least when I do that I can verify personally that it's all done right.

What DOES take a ton of time on my end is filling in all of the information tags from the txt files. I'm looking at my Atari 2600 collection right now and if I did that I would have to copy information for over 700 games individually, which is more than double of what I had done for 3 other systems combined.

Worst case scenario, if I can't get any help here and the other guy falls through, I can at least have a collection that will work in "no-metadata" skins that will display whatever artwork I choose or a movie.

UDb23

@Used2BeRX if you have the single text files available I can create a program that reads the txt and generates corresponding XML.
Did something similar in the past; ideas is it would convert all txt files it finds in a specific folder.
Would make that in VBA meaning you need Excel to run it ( just to run, no need to do any data import in excel).
Could also do it in VB and compile it into an executable; this would require more time as I don't have VB installed and haven't been using it for years..
Let me know if excel VBA could be ok.

Concerning global gameslist XML referencing to single game (sub) xmls I don't know if this is supported by ES.

UDb23

@Used2BeRX Really interested in your work on Atari 2600 gameslist especially if you can provide high quality cover images.
Also interested in building Vic20 gameslist.
Wrote my own mobygames scraper for that and downloaded all metadata from the site. Have found separate hq cover images to be used in the gamelist XML. Will combine them sometime.
Currently priority is on building my first control panel. Correctly drilling holes in plywood board isn't as simple as I thought :-)

Used2BeRX

@UDb23 Here's a download for the synopsis files for the individual 2600 games. If you can do something with this, I'd be happy to zip up a package with all the other systems. These are all named exactly the same as the roms are, except of course the extention is .txt instead of .zip.

Synopsis 2600: https://www.sendspace.com/file/mgf38h

And here's an XML file for the 2600 I'm currently working on. It has the 10 files for each game already made, as well as the <name></name> field filled in (although I will have to go back and edit some of them since I'm not actually ripping that from the txt file and I'm just doing a copy/paste job.

2600 gamelist.xml wip: https://www.sendspace.com/file/km2sf2

As you can see from the parts of the XML that are completed, the file names for the Extras for the system are named exactly the same as the rom itself, so everything should work right... unfortunately, the system we did it on originally didn't care about case, so chances are some things in there are broke because the case doesn't match. That's the stuff that I would have to fix by hand after the files were created.

Excel isn't a problem for me. I don't have it installed right now, but I do have a copy of 2007 lying around here somewhere. Hopefully a version that old will still allow me to use your script.

No need to make a VB program. It's not that I don't have time to wait, but I'm really OCD-ing out here and I'm manually doing things right now I said that I wasn't going to do because I'm waiting for somebody to automate this and patience isn't one of my virtues ;)

Looking forward to see what you can come up with. Chances are the xml I'm working on won't be usable with what you make, but that still isn't enough to make me stop working on it.

I've got good Atari 2600 artwork. There is at least a Box Front and Action shot for every game. Not a lot of title shots since a lot of games didn't have title screens and probably 100-150 carts missing.

For the boxes that we couldn't find or were never made, we actually had a very talented graphic artist on "staff" that created alternate boxarts for prototypes, hacks, translations, unlicensed and other categories.

Looking forward to seeing what you can do man. You're the only hope for my sanity at this point. :)

EDITED TO ADD: Now that I think of it, it might be good if there were two different scripts. One script that would create all of the fields/tags like the XML example I gave you, and then a second one that would "rip" the information from the TXT files and fill in the empty tags made by the first file.

That would make sure that work I've already done on the first part would still be usable by the time you make this, but it would also allow me to make sure the files are all working (because of the case issue) before ripping all the data. Any chance we could do that? Maybe that's easier to make it a two step process anyway?

You may have noticed that the path is not just "./" There are many sub folders for a lot of the systems, so the script should be able differentiate that in the path folder based off of which folder the roms are in. I'm making them in alphabetical order, grouped together in the folders. (Maybe you should see this XML when it's completed to fully understand what I'm talking about). I'll upload it as soon as it's done. I'm hopped up on coffee right now so it shouldn't be long :)