lr-mame2003 driver improvement and backport

grant2258

fixed altered beast this bug will probably cause problems with a few 68k games
details are here

https://retropie.org.uk/forum/topic/16833/mame-2003-gamelist-xml-with-data-from-the-binary/12

markwkidd

@riverstorm said in lr-mame2003 driver improvement and backport:

Thanks Mark I did see it and will take a look in a few weeks. We are heading out to Trinidad for Carnival in the morning and will be down there for a few weeks.

If this new feature inspired by @grant2258 works as it seems to, you will not need to do any more manual DAT creation: https://github.com/libretro/mame2003-libretro/pull/348/commits

markwkidd

@grant2258 are you set up with mame2003-plus? If so would you be willing to compile it from source and check this?

arcadez just merged in the "Generate XML DAT" feature but I'm having trouble getting it to work the same as it does in regular mame2003 in my environment. I haven't done much with mame2003-plus before now so I am hoping its something trivial on my end.

I'm really dying to get my hands XML DAT for mame2003-plus to begin testing all the new backports. It feels so close.

grant2258

sure ill have a look not a problem mark shoudl be off work monday and tuesday. Ill try compile the code when i wake up and have a peek. To rule out file permissions run as root

Riverstorm

@markwkidd said in lr-mame2003 driver improvement and backport:

@grant2258 are you set up with mame2003-plus? If so would you be willing to compile it from source and check this?

arcadez just merged in the "Generate XML DAT" feature but I'm having trouble getting it to work the same as it does in regular mame2003 in my environment. I haven't done much with mame2003-plus before now so I am hoping its something trivial on my end.

I'm really dying to get my hands XML DAT for mame2003-plus to begin testing all the new backports. It feels so close.

Will you guys be posting here when you do get a DAT generated? There's some nice updates I wouldn't mind playing too. ;)

grant2258

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Riverstorm

Thanks Grant, will do. I've been following all the changes you and Mark have been doing and it's impressive. You guys are in the zone! :)

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Grant, sorry, this is the DAT from lr-mame2003 and not mame2003-plus? It has the same errors as lr-mame2003. I was wondering when you guys pull a DAT from mame2003-plus if you could post a link. Reading on Github you guys are close?

<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

markwkidd

That is definitely the DAT file for mame2003-plus. It's identical to what I finally just got mame2003-plus-libretro to generate for me in Windows with the latest commit to the repo this afternoon.

For example, you're not going to see entries like that for mslug4 and mslug5 in the standard mame2003-libretro DAT.

On the other hand, problems that have been fixed in standard mame2003-libretro still exist in mame2003-plus-libretro because it was split off months ago to give time for arcadez to finish his epic backport project.

Once things are tested out with mame2003-plus-libretro the real fun will be in trying to merge them back together. No reason why that can't eventually happen.

Edit: I added the mame2003.xml dat to the repository.

Folks can download the mame2003-plus DAT directly from the mame2003-plus repository or generate your own right at home from the Tab menu!

For anyone else seeing this for the first time -- as grant2258 said: this core and DAT is all still considered experimental.

Testers are needed for the new romsets, but new ROMs will have to be hunted down and rebuilt into mame2003-plus romsets with a tool like clrmamepro.

Rebuilding with clrmamepro is a destructive process, so don't rebuild with your only set of roms! Make backups.

dankcushions

@markwkidd

Once things are tested out with mame2003-plus-libretro the real fun will be in trying to merge them back together. No reason why that can't eventually happen.

i was quite excited about the backport project, but whilst initially i thought the only stumbling block was the absense of a corrected .dat (which you seem to have solved, pretty much!), i think there's a more fundamental issue... romset.

whilst the scene will no doubt eventually package up and circulate a mame2003plus (0.78 + extras) romset, and updated supplementary files (samples, for example), it will likely never be as prolific and easily-obtainable like the 0.78 one is. that's a reason why 2003 works so well as a default - the romset and supplementary files are trivial to obtain. indeed, people have been running this romset in their retropie builds for years now, with no reason to update or anything like that. we can't just 'break' a bunch of games overnight, with a new version of retropie/mame2003.

i think these changes are very cool but will be very difficult to support if we just merge them in.

for me i see two options going forward:

mame2003plus becomes a branch of mame2003. this means retropie and i assume retroarch's build bot can still compile the two projects, but it makes it neater to keep in sync the common parts of the code-base.
mame2003plus should be renamed to liteMAME (or something else without 2003 in it) to satisfy the license, and because it eliminates anyone getting the wrong romset (this probably should happen either way), and we go nuts with whatever drivers work best on our SOC/console targets. romsets be damned!

i like 2) the best, but that's more of a project direction choice for gamezfan and yourself!

markwkidd

@dankcushions Those are good points.

I think an important question we will be able to answer soon is: will integrating mame2003-plus cause any regressions for anyone with an existing mame 0.78 collection?

If merging in mame2003-plus means only that games which didn't work can work, or that new romsets which aren't in mame 0.78 will work if they are present, then that's a different scenario than if there are regressions for people with 'classic' mame2003 collections.

This should be high on the list of considerations as we figure out how this works.

dankcushions

@markwkidd said in lr-mame2003 driver improvement and backport:

@dankcushions Those are good points.

I think an important question we will be able to answer soon is: will integrating mame2003-plus cause any regressions for anyone with an existing mame 0.78 collection?

If merging in mame2003-plus means only that games which didn't work can work, or that new romsets which aren't in mame 0.78 will work if they are present, then that's a different scenario than if there are regressions for people with 'classic' mame2003 collections.

This should be high on the list of considerations as we figure out how this works.

totally agreed. i believe the flashpoint that caused all the gamezfan changes to be backed out of mame2003 master was various games with updated samples no longer working with their old (mame 0.78 specific) samples, so just having no sound. there was some skuttlebutt about games stopping working entirely, but i'm not sure if that was ever true. i guess you can compare the projects and find out exactly what checksum changes are there for existing + working games.

BuZz

I'm against merging them (or basically you mean replacing the current mame2003 as the modified version can easily include fixes from mame2003). If that happens I will fork the current mame2003.

markwkidd

@dankcushions and anyone else, I'm going to post some findings.

I'm starting with a complete, Full Non-Merged MAME 0.78 collection, except for CHDs. It is fully not merged, ie it was checked with Scanner set to "Non-Merged" and "Split BIOS Sets" disabled in the Advanced menu.

I will be working with those same settings as the target for the mame2003-plus DAT. I have turned on all of the CRC/hash check features. I let ClrMamePro make any fixes it proposed.

Here is the full list of missing and incomplete sets

This should probably be the priority list for testing. Of course, once we know what sets can be used as sources to rebuild them.

This is the missing and incomplete list in ClrMamePro fixdat format

I wish I had a complete mame2010 collection nearby to throw into ClrMamePro. I'm hoping that such a thing would take care of many of these.

markwkidd

Based on some spot checking, I think that using a mame2010 collection to help with the rebuild will provide 2/3 of the missing ROMs for mame2003-plus.

dankcushions

@markwkidd if you're attempting to build the set for testing, i think you'll get the best possible hitrate with a current mame set, and the corresponding 'rollback' set. i'm not sure if any of that list are hacks? (mame never has hacks i believe). if so you'll need to get them from a fba set of some kind.

Riverstorm

@dankcushions said in lr-mame2003 driver improvement and backport:

@markwkidd if you're attempting to build the set for testing, i think you'll get the best possible hitrate with a current mame set, and the corresponding 'rollback' set. i'm not sure if any of that list are hacks? (mame never has hacks i believe). if so you'll need to get them from a fba set of some kind.

The list of missing games/roms are mostly added/fixed ROMs. I recognize a majority from creating the original DAT. It's not bad at all except it's balking on some existing sets due to updated samples and not necessarily a ROM change Donkey Kong for example. You might be able to modify the DAT or have samples separate. I am building fresh with a current MAME set and rollback. I'll let you know what's missing using an "official" set. Whatever is missing I would assume are hacked ROMs or at a minimum not official MAME ROMs.

Mark if you have a link to the exact mame2010 DAT you want tested I will build it then use it as my source for building a mame2003-plus set and pull the differences or a fixdat.

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Grant, sorry, this is the DAT from lr-mame2003 and not mame2003-plus? It has the same errors as lr-mame2003. I was wondering when you guys pull a DAT from mame2003-plus if you could post a link. Reading on Github you guys are close?
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

yes it will be the same because the original dat was edited by hand. I will fix these in the code its basically duplicate rom names with a different crc you wont notice it unless you compare the filenames

this is what you dont see that causes it
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>
<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

grant2258

@dankcushions

There is nothing stopping us using 2 sets of samples just change the sample filenames and add them both. This can easily be fixed the new samples can be renamed to not cause any issues with mame 2003. We cant do this by hand editing dats its from the source. I will fix the bad merges up all that is required is a filename change nothing major. Regardless if it merges or not i think we made some nice updates in the core and killed a few bugs along the way that can only be a good thing

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm said in lr-mame2003 driver improvement and backport:
@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Grant, sorry, this is the DAT from lr-mame2003 and not mame2003-plus? It has the same errors as lr-mame2003. I was wondering when you guys pull a DAT from mame2003-plus if you could post a link. Reading on Github you guys are close?
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
yes it will be the same because it was edited out by had i will fix these in the code its basically duplicate rom names with a different crc you wont notice it unless you compare the filenames

this is what you dont see that causes it
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>
<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

Thanks Grant, you're showing me the code vs. the DAT? This will cause an error in ClrMamePro such as "Can't merge set due to equal names for different hashes" so when building any type of merge set (split or merged) will rename it in the parent. Adding or removing the SHA1 from either so they match will get rid of the error.

What I was pointing out was the "baddump" status will also throw an error. When your prompted in ClrMamePro on how to handle these errors it's confusing for first time users. Basically they both need the baddump flag or removed on both. Some of the errors are unavoidable and you need to tell people to Answer "Yes to All" or "No to All", etc.

I definitely can't do what you do but mostly comfortable with ClrMamePro. I was just pointing out it had some of the same errors as the original DAT but Mark said it's a work in progress. If I can do anything to help just let me know.