Manually generate Mame2003 gamelist?

[UDb23]

@paradadf I like UXS's concept of mix templates but, as far as I understood it can only use images scraped from screenscraper.fr, not local images (on your PC or Pi).
Also, as mentioned before, I'm looking for a tool that can also update/overwrite selectable specific data or images related to roms already existing in the xml.

[herb_fargus]

@windale how I was figuring is I'd use my filename list as a basis and query against the latest set where they don't join, then I could generate an update table that corresponds the game names from .78 to the most recent 180 romset.

But yes if at all possible I'd like to use dats specific to .78 to remove any extra steps

[hansolo77]

You might send a message over to Tom at GameEx and see what how he works his metadata. If not Tom, then surely Circo over at EmuMovies can provide some insight. EmuMovies has a LOT of content, not just videos. It's the site GameEx scrapes data from (including details, box/cart/title/screen etc). It's worth a shot in any case.

[herb_fargus]

Alrighty. Quick update on my battle with mame.

Originally I was thinking great I'll just match the filename with the snap pack. But... Mame changes filenames and versions quite frequently it would seem. So. I found a dat that shows what files have been renamed and in what version as well as a tool that is kinda like clrmamepro where it renames the files to an older version that you define based on the information from the rename dat.

In case you're curious- rename dat:

http://www.progettosnaps.net/renameset/

Rename tool:

http://www.progettosnaps.net/tools/

In theory you can use clrmamepro to fix up your snaps too but I think you need a dat specific to .78 which is too old to have a posted reference on the site, so here I am manually generating it.

I pulled a .170 snaps set and created a txt list and used that as my reference to copy out the snaps based on filenames but without renaming them there were ~800 unaccounted for. After renaming there were ~93 unaccounted for, so I am going to have to figure out what those 93 correspond to in the full .170 set. Then once that is complete I can generate a .78 dat that can be used to match update snaps in the future.

[hansolo77]

The missing ROMs might just be junk that's newly available with the more recent romset. Things that weren't running before that have been fixed. Or ROMs that were broken in thew newer set that the MAME dev team took off their compatible list. You might also have ROMs for things that are mechanical, mahjong, etc that nobody ever plays anyway.

[herb_fargus]

@hansolo77 I think there are also some are some bugs in the tool because it doesn't know how to handle some of the same filenames eg blockr2 and blockr swapped names in one and I don't think it knew how to reconcile the order. I'll probably post the images/gamelist upon completion or at the very least see if there is a way to use the dat to rebuild.

[herb_fargus]

As an update: got most of the snaps and titles for mame 2000 and mame 2003.

Now just need to generate gamelist. May include flyers and marquees at some point

Repo here:

https://github.com/HerbFargus/Mame-Extras

[caver01]

@herb_fargus The timing for your project happened to match my own research. I have been using my own manually generated gamelist for a while, but I need to add some other meta data like descriptions and genre. Thanks for the links to sources. I was about to ask for advice on those!

As for actually generating the XML, I built mine in a google sheet with a series of formulas. It starts with a copy of the compatibility list for MAME2003 linked in the wiki here, then I added sheets that list the other metadata columns like path, image, description, and so on. Finally, because I don't have EVERY ROM installed, I do a lookup and generate a list of ROMS I marked, then I build out the XML using a series of vlookup functions, basically constructing the XML lines, game by game, only it's essentially automatic. If I mark a new ROM for installation, it appears in the XML sheet. I then copy all rows from the XML sheet and paste it into the text file.

It's not super elegant, but it works.

[caver01]

@herb_fargus Any ideas how you are going to parse the history.dat? I opened it with a text editor to see if I could systematically grab descriptions, but there is a LOT of detail in that file. It was too much to hope for my description on a line with ROM,DESCRIPTION, in front of it!

[herb_fargus]

@caver01 I've been trying to think of a solution for that as well. From what I gather it's more or less in the same format though some may not even have a description equivalent as it seems to defer to the parent for the descriptions for some clones and bootlegs. My preliminary though was to do something like we do with the ini configs in the setup script where it parses a text block based on keywords before and after but could get messy.

Overall as far as metadata goes I like your idea in that we can generate an expanded compatibility list if you will into one sheet and then we can parse that out to a gamelist XML once complete. I have some dat tools do split what's available (that's how I created the compatibility lists in the first place)'

But yeah if you'd like to help out perhaps we can come to a solution

[caver01]

@herb_fargus If we have a sheet of metadata (or simply more columns in the compatibility list to accommodate our XML needs), I can apply my formula to generate the XML, which can then be copy/pasted into a text file.

Come to think of it, it wouldn't have to be dynamic. Mine today is lacking a few columns, but it dynamically builds the XML in a worksheet based on my selected games--but we could just as easily build a master gamelist XML file that contains the details for every possible ROM in the set regardless of whether or not the ROM is installed. The idea would be that if you just drop an image into the right folder using the right filename it will display. It's a bit like "scraping the possibility" of a ROM, rather than an actual scrape of what is really there. Then, just add images.

It strikes me that some of this work might be done already. I seem to recall an earlier build when I installed an entire ROMset and started the scraper. I quickly ran out of patience with image downloads, but I did end up with a gamelist.xml that had a lot of the details already there. That might be a better starting place for columns like genre, description, publisher, etc.

Or. . .am I remembering it wrong? Can we leverage a complete romset and scrape that to get a good starting set of XML metadata?

[herb_fargus]

@caver01 some has been done but I want to pull from as many official sources as possible rather than user created data as it's often incomplete and fraught with errors

[caver01]

@herb_fargus That makes sense. I assumed the scrapers that build XML were using official data sources, but from some of my own investigations I recall sites requesting help populating their databases.

[herb_fargus]

@caver01 it pulls from some official sources for hash matching and such but as far as content there aren't official sources for some of the metadata like ratings as they are subjective.

Really the only technically proper way to handle mame is to set up a versioned database that takes into account all the many changes through each revision. Clrmamepro handles some but there isn't any real API or anything for it at least not any complete ones. Most are static based on the latest mame version

[caver01]

@herb_fargus Ok, so maybe this is just restating the obvious, but the XML generation seems like the easier part (for me) and I am happy to contribute what I have done via spreadsheets if that's helpful. The real work is populating the columns of metadata. All I have done is construct a lookup mechanism. I imagine there are script wizards that can one-up me on that just using command line.

[herb_fargus]

@caver01 yes populating the tags and XML generation for the static mame2003 list is where is like to start. If you want to do a google sheets collaboration with what you have I'd like to do that so we have a central place to work on it

[UDb23]

@herb_fargus, @caver01 Some time ago I wrote an excel VBA scaper for Mobygames for Vic20 games.
I could modify it to get their Arcade games metadata and provide these in xls or csv format adding related mame .78 rom name column.
Mobygames lists about 1800 roms (so unfortunately it is not complete set).

[herb_fargus]

@UDb23 it's a good way to get the bulk, could be done. If you haven't noticed by now I'm very particular about the quality/completeness of my data.

First I'd like to identify any complete sources we have, which means starting with MAME itself extracting information/dats/XML directly from the mame binary . Those are simple enough to parse with basic regex/sed.

Next level of completion of sorts is progettosnaps (which also has a source of the dats from the mame executables) primarily he has full sets of snaps.

Last night the issue I ran into is he has snaps for .170 not .78 so I had to use a rename dat to fix the filenames not accounted for but there are some ROMs that are hacks/bootlegs that have been dropped from mame as they aren't really "real" mame games and don't fit their archival purposes. So I hashed out the rest of those.

Then of course there are ES tags that are arbitrary or that there is no official mame source for them, that's where random scraping/user content is necessary.

Clrmamepro really is supposed to be able to handle rebuilds like this but it really only works if you have proper dats and there aren't a whole lot of dats for metadata outside of just rom info so... Makes it a bit more of a manual process.

[caver01]

@UDb23 Wow, that would be a big help for me, but part of this project is to ensure the details come from a known or respected source. Do we know where the original source data comes from for the Arcade section of MobyGames?

[UDb23]

@caver01 I could get the descriptions also from progettoemma.net. It seems to be reliable. See example here; metadata in italian or english.
Contains all mame roms and I already did .78 vs actual mame rom set renaming check for the Mame resolution DB.