Manually generate Mame2003 gamelist?
-
@herb_fargus That's a great idea. I was thinking of doing something similar but under windows; this would run on the Pi: much better !
The Perl script you mentioned overwrites any existing gamelist.xml; if possible, I'd suggest to modify it to check if items already exist in current gamelist.
The xml file may have been previously edited manually or generated from different sources / tools so it would be cool to be able to just integrate (update) specific data/images in an existing gamelist.xml.
-
Any reason why don't you just use the Universal XML Scraper instead?
https://retropie.org.uk/forum/topic/5291/soft-universal-xml-scraper-v2-easy-scrape-with-high-quality-picture -
@paradadf As of now many of the games and the clones displays with the same name. There are talks about getting the information from http://adb.arcadeitalia.net and incorporating this into screenscraper.fr but as of now this is not the case just yet. More info here
The author of http://adb.arcadeitalia.net @motoschifo also wanted to help sselph to incorporate his database into sselph's scraper but @sselph has been MIA or not seen missed this. Post from motoschifo here about just that.
I also posted this on @sselph github here just to make sure he did not miss it.
-
@Rion thanks for the explanation! I know about the naming problem of SS, but maybe is it easiar to modofied the names wirh some script after using UXS (until SS modifies the way of handling arcade roms) that doong all the steps of the first post. Just an idea.
-
@herb_fargus I would use the specific CatVer for Mame 0.78 as newer CatVer.ini won't match a lot of the names as they have changed a lot over time.
http://catlist.mameworld.info/files/arc78.zip
Also, because of this, the latest Arcade History dat may get a few games wrong or miss them out completely because of the different zip names. For instance, Rainbow Islands is rainbow.zip in older Mame but has changed to rbisland.zip in the latest Mame.
-
@windale http://adb.arcadeitalia.net has a built in functions to sort mame list using many different setting and generate lists according to your specifications. Generation of mame lists from version 0.001 to the latest.
But i agree that using the right carver for Mame 0.78 is the right way to go when scraping for Mame2003 but for others like fba it would be better to use the other method.
-
@paradadf I like UXS's concept of mix templates but, as far as I understood it can only use images scraped from screenscraper.fr, not local images (on your PC or Pi).
Also, as mentioned before, I'm looking for a tool that can also update/overwrite selectable specific data or images related to roms already existing in the xml. -
@windale how I was figuring is I'd use my filename list as a basis and query against the latest set where they don't join, then I could generate an update table that corresponds the game names from .78 to the most recent 180 romset.
But yes if at all possible I'd like to use dats specific to .78 to remove any extra steps
-
You might send a message over to Tom at GameEx and see what how he works his metadata. If not Tom, then surely Circo over at EmuMovies can provide some insight. EmuMovies has a LOT of content, not just videos. It's the site GameEx scrapes data from (including details, box/cart/title/screen etc). It's worth a shot in any case.
-
Alrighty. Quick update on my battle with mame.
Originally I was thinking great I'll just match the filename with the snap pack. But... Mame changes filenames and versions quite frequently it would seem. So. I found a dat that shows what files have been renamed and in what version as well as a tool that is kinda like clrmamepro where it renames the files to an older version that you define based on the information from the rename dat.
In case you're curious- rename dat:
http://www.progettosnaps.net/renameset/
Rename tool:
http://www.progettosnaps.net/tools/
In theory you can use clrmamepro to fix up your snaps too but I think you need a dat specific to .78 which is too old to have a posted reference on the site, so here I am manually generating it.
I pulled a .170 snaps set and created a txt list and used that as my reference to copy out the snaps based on filenames but without renaming them there were ~800 unaccounted for. After renaming there were ~93 unaccounted for, so I am going to have to figure out what those 93 correspond to in the full .170 set. Then once that is complete I can generate a .78 dat that can be used to match update snaps in the future.
-
The missing ROMs might just be junk that's newly available with the more recent romset. Things that weren't running before that have been fixed. Or ROMs that were broken in thew newer set that the MAME dev team took off their compatible list. You might also have ROMs for things that are mechanical, mahjong, etc that nobody ever plays anyway.
-
@hansolo77 I think there are also some are some bugs in the tool because it doesn't know how to handle some of the same filenames eg blockr2 and blockr swapped names in one and I don't think it knew how to reconcile the order. I'll probably post the images/gamelist upon completion or at the very least see if there is a way to use the dat to rebuild.
-
As an update: got most of the snaps and titles for mame 2000 and mame 2003.
Now just need to generate gamelist. May include flyers and marquees at some point
Repo here:
-
@herb_fargus The timing for your project happened to match my own research. I have been using my own manually generated gamelist for a while, but I need to add some other meta data like descriptions and genre. Thanks for the links to sources. I was about to ask for advice on those!
As for actually generating the XML, I built mine in a google sheet with a series of formulas. It starts with a copy of the compatibility list for MAME2003 linked in the wiki here, then I added sheets that list the other metadata columns like path, image, description, and so on. Finally, because I don't have EVERY ROM installed, I do a lookup and generate a list of ROMS I marked, then I build out the XML using a series of vlookup functions, basically constructing the XML lines, game by game, only it's essentially automatic. If I mark a new ROM for installation, it appears in the XML sheet. I then copy all rows from the XML sheet and paste it into the text file.
It's not super elegant, but it works.
-
@herb_fargus Any ideas how you are going to parse the history.dat? I opened it with a text editor to see if I could systematically grab descriptions, but there is a LOT of detail in that file. It was too much to hope for my description on a line with ROM,DESCRIPTION, in front of it!
-
@caver01 I've been trying to think of a solution for that as well. From what I gather it's more or less in the same format though some may not even have a description equivalent as it seems to defer to the parent for the descriptions for some clones and bootlegs. My preliminary though was to do something like we do with the ini configs in the setup script where it parses a text block based on keywords before and after but could get messy.
Overall as far as metadata goes I like your idea in that we can generate an expanded compatibility list if you will into one sheet and then we can parse that out to a gamelist XML once complete. I have some dat tools do split what's available (that's how I created the compatibility lists in the first place)'
But yeah if you'd like to help out perhaps we can come to a solution
If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/
Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first
-
@herb_fargus If we have a sheet of metadata (or simply more columns in the compatibility list to accommodate our XML needs), I can apply my formula to generate the XML, which can then be copy/pasted into a text file.
Come to think of it, it wouldn't have to be dynamic. Mine today is lacking a few columns, but it dynamically builds the XML in a worksheet based on my selected games--but we could just as easily build a master gamelist XML file that contains the details for every possible ROM in the set regardless of whether or not the ROM is installed. The idea would be that if you just drop an image into the right folder using the right filename it will display. It's a bit like "scraping the possibility" of a ROM, rather than an actual scrape of what is really there. Then, just add images.
It strikes me that some of this work might be done already. I seem to recall an earlier build when I installed an entire ROMset and started the scraper. I quickly ran out of patience with image downloads, but I did end up with a gamelist.xml that had a lot of the details already there. That might be a better starting place for columns like genre, description, publisher, etc.
Or. . .am I remembering it wrong? Can we leverage a complete romset and scrape that to get a good starting set of XML metadata?
-
@caver01 some has been done but I want to pull from as many official sources as possible rather than user created data as it's often incomplete and fraught with errors
If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/
Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first
-
@herb_fargus That makes sense. I assumed the scrapers that build XML were using official data sources, but from some of my own investigations I recall sites requesting help populating their databases.
-
@caver01 it pulls from some official sources for hash matching and such but as far as content there aren't official sources for some of the metadata like ratings as they are subjective.
Really the only technically proper way to handle mame is to set up a versioned database that takes into account all the many changes through each revision. Clrmamepro handles some but there isn't any real API or anything for it at least not any complete ones. Most are static based on the latest mame version
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.