Manually generate Mame2003 gamelist?

herb_fargus

With mamedb being down I've been trying to devise a clever way to generate a complete gamelist with all metadata and I think ive just about figured out the sources I'd need. Also as essentially everyone has the same mame2003 reference set, it should work for everyone and when complete the gamelist and images should just be able to be dumped into the same romfolder and it should work.

Here is my thought process:

ES needs the following tags:

<path>
<name>
<desc>
<image>
<rating>
<releasedate>
<developer>
<publisher>
<genre>
<players>

<path>:

Easy enough to do, I can generate a list of all the mame filenames for my mame 2003 romset

<name>:

This information is available in the mame dat:
http://www.progettosnaps.net/dats/MAME/MAME_Dats_078.rar

<desc>:

This is a bit more incomplete, though a good portion of the games have a description from the arcade history dat:

http://www.arcade-history.com/?page=download

<image>:

Box art can be sporadic so I personally prefer screenshots for mame (thinking I'll add in game snaps and titles depending on how I'm feeling)

http://www.progettosnaps.net/snapshots/

Can all more or less be generated from the above mentioned history dat though there may be some gaps with these.

<genre>:
The catver.ini file handles these nicely
http://www.progettosnaps.net/catver/

So once I've got all my sources I'll need to parse out the tag contents individually, then my filename will essentially be my primary key, and I'll mash it altogether into a functioning gamelist XML.

I found a Perl script that can likely be adapted to utilise all the resources I've gathered and parsed:

https://www.raspberrypi.org/forums/viewtopic.php?f=78&t=70957

So anyone have any input or am I daft for attempting such a thing in such a manual way?

UDb23

@herb_fargus That's a great idea. I was thinking of doing something similar but under windows; this would run on the Pi: much better !

The Perl script you mentioned overwrites any existing gamelist.xml; if possible, I'd suggest to modify it to check if items already exist in current gamelist.

The xml file may have been previously edited manually or generated from different sources / tools so it would be cool to be able to just integrate (update) specific data/images in an existing gamelist.xml.

paradadf

Any reason why don't you just use the Universal XML Scraper instead?
https://retropie.org.uk/forum/topic/5291/soft-universal-xml-scraper-v2-easy-scrape-with-high-quality-picture

Rion

@paradadf As of now many of the games and the clones displays with the same name. There are talks about getting the information from http://adb.arcadeitalia.net and incorporating this into screenscraper.fr but as of now this is not the case just yet. More info here

The author of http://adb.arcadeitalia.net @motoschifo also wanted to help sselph to incorporate his database into sselph's scraper but @sselph has been MIA or not seen missed this. Post from motoschifo here about just that.

I also posted this on @sselph github here just to make sure he did not miss it.

paradadf

@Rion thanks for the explanation! I know about the naming problem of SS, but maybe is it easiar to modofied the names wirh some script after using UXS (until SS modifies the way of handling arcade roms) that doong all the steps of the first post. Just an idea.

windale

@herb_fargus I would use the specific CatVer for Mame 0.78 as newer CatVer.ini won't match a lot of the names as they have changed a lot over time.

http://catlist.mameworld.info/files/arc78.zip

Also, because of this, the latest Arcade History dat may get a few games wrong or miss them out completely because of the different zip names. For instance, Rainbow Islands is rainbow.zip in older Mame but has changed to rbisland.zip in the latest Mame.

Rion

@windale http://adb.arcadeitalia.net has a built in functions to sort mame list using many different setting and generate lists according to your specifications. Generation of mame lists from version 0.001 to the latest.

But i agree that using the right carver for Mame 0.78 is the right way to go when scraping for Mame2003 but for others like fba it would be better to use the other method.

UDb23

@paradadf I like UXS's concept of mix templates but, as far as I understood it can only use images scraped from screenscraper.fr, not local images (on your PC or Pi).
Also, as mentioned before, I'm looking for a tool that can also update/overwrite selectable specific data or images related to roms already existing in the xml.

herb_fargus

@windale how I was figuring is I'd use my filename list as a basis and query against the latest set where they don't join, then I could generate an update table that corresponds the game names from .78 to the most recent 180 romset.

But yes if at all possible I'd like to use dats specific to .78 to remove any extra steps

hansolo77

You might send a message over to Tom at GameEx and see what how he works his metadata. If not Tom, then surely Circo over at EmuMovies can provide some insight. EmuMovies has a LOT of content, not just videos. It's the site GameEx scrapes data from (including details, box/cart/title/screen etc). It's worth a shot in any case.

herb_fargus

Alrighty. Quick update on my battle with mame.

Originally I was thinking great I'll just match the filename with the snap pack. But... Mame changes filenames and versions quite frequently it would seem. So. I found a dat that shows what files have been renamed and in what version as well as a tool that is kinda like clrmamepro where it renames the files to an older version that you define based on the information from the rename dat.

In case you're curious- rename dat:

http://www.progettosnaps.net/renameset/

Rename tool:

http://www.progettosnaps.net/tools/

In theory you can use clrmamepro to fix up your snaps too but I think you need a dat specific to .78 which is too old to have a posted reference on the site, so here I am manually generating it.

I pulled a .170 snaps set and created a txt list and used that as my reference to copy out the snaps based on filenames but without renaming them there were ~800 unaccounted for. After renaming there were ~93 unaccounted for, so I am going to have to figure out what those 93 correspond to in the full .170 set. Then once that is complete I can generate a .78 dat that can be used to match update snaps in the future.

hansolo77

The missing ROMs might just be junk that's newly available with the more recent romset. Things that weren't running before that have been fixed. Or ROMs that were broken in thew newer set that the MAME dev team took off their compatible list. You might also have ROMs for things that are mechanical, mahjong, etc that nobody ever plays anyway.

herb_fargus

@hansolo77 I think there are also some are some bugs in the tool because it doesn't know how to handle some of the same filenames eg blockr2 and blockr swapped names in one and I don't think it knew how to reconcile the order. I'll probably post the images/gamelist upon completion or at the very least see if there is a way to use the dat to rebuild.

herb_fargus

As an update: got most of the snaps and titles for mame 2000 and mame 2003.

Now just need to generate gamelist. May include flyers and marquees at some point

Repo here:

https://github.com/HerbFargus/Mame-Extras

caver01

@herb_fargus The timing for your project happened to match my own research. I have been using my own manually generated gamelist for a while, but I need to add some other meta data like descriptions and genre. Thanks for the links to sources. I was about to ask for advice on those!

As for actually generating the XML, I built mine in a google sheet with a series of formulas. It starts with a copy of the compatibility list for MAME2003 linked in the wiki here, then I added sheets that list the other metadata columns like path, image, description, and so on. Finally, because I don't have EVERY ROM installed, I do a lookup and generate a list of ROMS I marked, then I build out the XML using a series of vlookup functions, basically constructing the XML lines, game by game, only it's essentially automatic. If I mark a new ROM for installation, it appears in the XML sheet. I then copy all rows from the XML sheet and paste it into the text file.

It's not super elegant, but it works.

caver01

@herb_fargus Any ideas how you are going to parse the history.dat? I opened it with a text editor to see if I could systematically grab descriptions, but there is a LOT of detail in that file. It was too much to hope for my description on a line with ROM,DESCRIPTION, in front of it!

herb_fargus

@caver01 I've been trying to think of a solution for that as well. From what I gather it's more or less in the same format though some may not even have a description equivalent as it seems to defer to the parent for the descriptions for some clones and bootlegs. My preliminary though was to do something like we do with the ini configs in the setup script where it parses a text block based on keywords before and after but could get messy.

Overall as far as metadata goes I like your idea in that we can generate an expanded compatibility list if you will into one sheet and then we can parse that out to a gamelist XML once complete. I have some dat tools do split what's available (that's how I created the compatibility lists in the first place)'

But yeah if you'd like to help out perhaps we can come to a solution

caver01

@herb_fargus If we have a sheet of metadata (or simply more columns in the compatibility list to accommodate our XML needs), I can apply my formula to generate the XML, which can then be copy/pasted into a text file.

Come to think of it, it wouldn't have to be dynamic. Mine today is lacking a few columns, but it dynamically builds the XML in a worksheet based on my selected games--but we could just as easily build a master gamelist XML file that contains the details for every possible ROM in the set regardless of whether or not the ROM is installed. The idea would be that if you just drop an image into the right folder using the right filename it will display. It's a bit like "scraping the possibility" of a ROM, rather than an actual scrape of what is really there. Then, just add images.

It strikes me that some of this work might be done already. I seem to recall an earlier build when I installed an entire ROMset and started the scraper. I quickly ran out of patience with image downloads, but I did end up with a gamelist.xml that had a lot of the details already there. That might be a better starting place for columns like genre, description, publisher, etc.

Or. . .am I remembering it wrong? Can we leverage a complete romset and scrape that to get a good starting set of XML metadata?

herb_fargus

@caver01 some has been done but I want to pull from as many official sources as possible rather than user created data as it's often incomplete and fraught with errors

caver01

@herb_fargus That makes sense. I assumed the scrapers that build XML were using official data sources, but from some of my own investigations I recall sites requesting help populating their databases.