lr-mame2003 driver improvement and backport

markwkidd

That is definitely the DAT file for mame2003-plus. It's identical to what I finally just got mame2003-plus-libretro to generate for me in Windows with the latest commit to the repo this afternoon.

For example, you're not going to see entries like that for mslug4 and mslug5 in the standard mame2003-libretro DAT.

On the other hand, problems that have been fixed in standard mame2003-libretro still exist in mame2003-plus-libretro because it was split off months ago to give time for arcadez to finish his epic backport project.

Once things are tested out with mame2003-plus-libretro the real fun will be in trying to merge them back together. No reason why that can't eventually happen.

Edit: I added the mame2003.xml dat to the repository.

Folks can download the mame2003-plus DAT directly from the mame2003-plus repository or generate your own right at home from the Tab menu!

For anyone else seeing this for the first time -- as grant2258 said: this core and DAT is all still considered experimental.

Testers are needed for the new romsets, but new ROMs will have to be hunted down and rebuilt into mame2003-plus romsets with a tool like clrmamepro.

Rebuilding with clrmamepro is a destructive process, so don't rebuild with your only set of roms! Make backups.

dankcushions

@markwkidd

Once things are tested out with mame2003-plus-libretro the real fun will be in trying to merge them back together. No reason why that can't eventually happen.

i was quite excited about the backport project, but whilst initially i thought the only stumbling block was the absense of a corrected .dat (which you seem to have solved, pretty much!), i think there's a more fundamental issue... romset.

whilst the scene will no doubt eventually package up and circulate a mame2003plus (0.78 + extras) romset, and updated supplementary files (samples, for example), it will likely never be as prolific and easily-obtainable like the 0.78 one is. that's a reason why 2003 works so well as a default - the romset and supplementary files are trivial to obtain. indeed, people have been running this romset in their retropie builds for years now, with no reason to update or anything like that. we can't just 'break' a bunch of games overnight, with a new version of retropie/mame2003.

i think these changes are very cool but will be very difficult to support if we just merge them in.

for me i see two options going forward:

mame2003plus becomes a branch of mame2003. this means retropie and i assume retroarch's build bot can still compile the two projects, but it makes it neater to keep in sync the common parts of the code-base.
mame2003plus should be renamed to liteMAME (or something else without 2003 in it) to satisfy the license, and because it eliminates anyone getting the wrong romset (this probably should happen either way), and we go nuts with whatever drivers work best on our SOC/console targets. romsets be damned!

i like 2) the best, but that's more of a project direction choice for gamezfan and yourself!

markwkidd

@dankcushions Those are good points.

I think an important question we will be able to answer soon is: will integrating mame2003-plus cause any regressions for anyone with an existing mame 0.78 collection?

If merging in mame2003-plus means only that games which didn't work can work, or that new romsets which aren't in mame 0.78 will work if they are present, then that's a different scenario than if there are regressions for people with 'classic' mame2003 collections.

This should be high on the list of considerations as we figure out how this works.

dankcushions

@markwkidd said in lr-mame2003 driver improvement and backport:

@dankcushions Those are good points.

I think an important question we will be able to answer soon is: will integrating mame2003-plus cause any regressions for anyone with an existing mame 0.78 collection?

If merging in mame2003-plus means only that games which didn't work can work, or that new romsets which aren't in mame 0.78 will work if they are present, then that's a different scenario than if there are regressions for people with 'classic' mame2003 collections.

This should be high on the list of considerations as we figure out how this works.

totally agreed. i believe the flashpoint that caused all the gamezfan changes to be backed out of mame2003 master was various games with updated samples no longer working with their old (mame 0.78 specific) samples, so just having no sound. there was some skuttlebutt about games stopping working entirely, but i'm not sure if that was ever true. i guess you can compare the projects and find out exactly what checksum changes are there for existing + working games.

BuZz

I'm against merging them (or basically you mean replacing the current mame2003 as the modified version can easily include fixes from mame2003). If that happens I will fork the current mame2003.

markwkidd

@dankcushions and anyone else, I'm going to post some findings.

I'm starting with a complete, Full Non-Merged MAME 0.78 collection, except for CHDs. It is fully not merged, ie it was checked with Scanner set to "Non-Merged" and "Split BIOS Sets" disabled in the Advanced menu.

I will be working with those same settings as the target for the mame2003-plus DAT. I have turned on all of the CRC/hash check features. I let ClrMamePro make any fixes it proposed.

Here is the full list of missing and incomplete sets

This should probably be the priority list for testing. Of course, once we know what sets can be used as sources to rebuild them.

This is the missing and incomplete list in ClrMamePro fixdat format

I wish I had a complete mame2010 collection nearby to throw into ClrMamePro. I'm hoping that such a thing would take care of many of these.

markwkidd

Based on some spot checking, I think that using a mame2010 collection to help with the rebuild will provide 2/3 of the missing ROMs for mame2003-plus.

dankcushions

@markwkidd if you're attempting to build the set for testing, i think you'll get the best possible hitrate with a current mame set, and the corresponding 'rollback' set. i'm not sure if any of that list are hacks? (mame never has hacks i believe). if so you'll need to get them from a fba set of some kind.

Riverstorm

@dankcushions said in lr-mame2003 driver improvement and backport:

@markwkidd if you're attempting to build the set for testing, i think you'll get the best possible hitrate with a current mame set, and the corresponding 'rollback' set. i'm not sure if any of that list are hacks? (mame never has hacks i believe). if so you'll need to get them from a fba set of some kind.

The list of missing games/roms are mostly added/fixed ROMs. I recognize a majority from creating the original DAT. It's not bad at all except it's balking on some existing sets due to updated samples and not necessarily a ROM change Donkey Kong for example. You might be able to modify the DAT or have samples separate. I am building fresh with a current MAME set and rollback. I'll let you know what's missing using an "official" set. Whatever is missing I would assume are hacked ROMs or at a minimum not official MAME ROMs.

Mark if you have a link to the exact mame2010 DAT you want tested I will build it then use it as my source for building a mame2003-plus set and pull the differences or a fixdat.

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Grant, sorry, this is the DAT from lr-mame2003 and not mame2003-plus? It has the same errors as lr-mame2003. I was wondering when you guys pull a DAT from mame2003-plus if you could post a link. Reading on Github you guys are close?
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

yes it will be the same because the original dat was edited by hand. I will fix these in the code its basically duplicate rom names with a different crc you wont notice it unless you compare the filenames

this is what you dont see that causes it
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>
<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

grant2258

@dankcushions

There is nothing stopping us using 2 sets of samples just change the sample filenames and add them both. This can easily be fixed the new samples can be renamed to not cause any issues with mame 2003. We cant do this by hand editing dats its from the source. I will fix the bad merges up all that is required is a filename change nothing major. Regardless if it merges or not i think we made some nice updates in the core and killed a few bugs along the way that can only be a good thing

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm said in lr-mame2003 driver improvement and backport:
@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m not tested yet so rebuild roms somewhere else dont mess with your original set

Grant, sorry, this is the DAT from lr-mame2003 and not mame2003-plus? It has the same errors as lr-mame2003. I was wondering when you guys pull a DAT from mame2003-plus if you could post a link. Reading on Github you guys are close?
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
yes it will be the same because it was edited out by had i will fix these in the code its basically duplicate rom names with a different crc you wont notice it unless you compare the filenames

this is what you dont see that causes it
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>
<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

Thanks Grant, you're showing me the code vs. the DAT? This will cause an error in ClrMamePro such as "Can't merge set due to equal names for different hashes" so when building any type of merge set (split or merged) will rename it in the parent. Adding or removing the SHA1 from either so they match will get rid of the error.

What I was pointing out was the "baddump" status will also throw an error. When your prompted in ClrMamePro on how to handle these errors it's confusing for first time users. Basically they both need the baddump flag or removed on both. Some of the errors are unavoidable and you need to tell people to Answer "Yes to All" or "No to All", etc.

I definitely can't do what you do but mostly comfortable with ClrMamePro. I was just pointing out it had some of the same errors as the original DAT but Mark said it's a work in progress. If I can do anything to help just let me know.

Riverstorm

I downloaded the DAT from the repository Mark linked above and Arcadez (gamez-fan) was true to his word. He removed virtually every hacked ROM. This is the entire list left of what's needed after running it through the current MAME and rollback. Maybe they can be swapped for official ROMs?

Hyper Street Fighter II: The Anniversary Edition (Asia 040202) [folder: hsf2a - size: 45mb]
missing rom: hs2ax.03 [size: 524288] [CRC32: 5f3d7397] [SHA1: 96f327dd998105ad5dc46bc9d3b741805a840d68]
missing rom: hs2ax.04 [size: 524288] [CRC32: 59acf108] [SHA1: e68fe233681175b29a35badab249c2b892b23af3]

The King of Fighters 2003 (Decrypted C) [system: Neo-Geo - folder: kof2003d - size: 124mb]
missing rom: 271-v1d.bin [size: 16777216] [CRC32: 2964f36e]

Snk Vs Capcom : Svc Chaos [system: Neo-Geo - folder: svcchaos - size: 114mb]
missing rom: 269-m1d.bin [size: 131072] [CRC32: fd4f0ff9] [SHA1: e97e864aaa4789d854363512986e80c0606da996]

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

will ren

the answer to your question is yes to all. No you dont remove the sha they are different roms the problem is they have the same name so when you do a full merge clrmame renames them. This isint a big deal because if mame doesnt find the right name it will match the crc/sha. I am showing you the xml but thats generated from the code. The bad dump isint throwing the error an error because its named the same in another set with a different crc. This baddump means there is no dump for this particular board that is different from the parent.

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

the answer to your question is yes to all. No you dont remove the sha they are different roms the problem is they have the same name so when you do a full merge clrmame renames them. This isint a big deal because if mame doesnt find the right name it will match the crc/sha. I am showing you the xml but thats generated from the code. The bad dump isint throwing the error an error because its named the same in another set with a different crc. This baddump means there is no dump for this particular board that is different from the parent.

Ah, sorry, there's 3 here, I missed there's a second clone. I was only comparing the parent to the first clone with the bad dump flag but matching hashes. Just to clarify you're saying "hydrap2" is a different ROM file from what's located in "hydra" and "hydrap" but it's named the same? Being a bad dump you have neither the CRC or SHA1. How is the ROM name determined on creating a merge set then? I always thought the merge tag was needed for equal names of different hashes.

<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

There is nothing stopping us using 2 sets of samples just change the sample filenames and add them both.

Another option even easier is just add all the new sample ROMs to the same archive. lr-mame2003 wouldn't care if there's extra samples and it wouldn't require a name change but only adding a few extra lines to the existing DAT game sample section and both would work flawlessly. Donkey Kong for example has all unique names in both the original and the arcadez fix. Hash isn't used in samples so a merge would be fairly simple and affect nothing.

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

DAT gam

all that is happening is mame is processing nodumps in the xml i have fixed that there is another two roms i need to look at in detail when i have time. the new dat is in the same link. You will only have 2 bad merges now. Ill let arcadez and mark look at anything i change before it gets commited. Forgot to answer your question yes its a clone of hydra thats why the bad merge issue came about because the nodump had the same romname

https://drive.google.com/open?id=1fXvqlTYJAXKX-aP_3d_rYZxjULW1n4_m

grant2258

@riverstorm

ok the dat is 100% fixed for the merge errors the other problem was just the bios wasnt created so you will gave a new file called acpsx.zip. The same link for the updated file if you want to test it mr riverstorm

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

ok the dat is 100% fixed for the merge errors the other problem was just the bios wasnt created so you will gave a new file called acpsx.zip. The same link for the updated file if you want to test it mr riverstorm

It looks real good. The same few missing ROMs from using current MAME and rollback:

Hyper Street Fighter II: The Anniversary Edition (Asia 040202) [folder: hsf2a - size: 45mb]
missing rom: hs2ax.03 [size: 524288] [CRC32: 5f3d7397] [SHA1: 96f327dd998105ad5dc46bc9d3b741805a840d68]
missing rom: hs2ax.04 [size: 524288] [CRC32: 59acf108] [SHA1: e68fe233681175b29a35badab249c2b892b23af3]

The King of Fighters 2003 (Decrypted C) [system: Neo-Geo - folder: kof2003d - size: 124mb]
missing rom: 271-v1d.bin [size: 16777216] [CRC32: 2964f36e]

Snk Vs Capcom : Svc Chaos [system: Neo-Geo - folder: svcchaos - size: 114mb]
missing rom: 269-m1d.bin [size: 131072] [CRC32: fd4f0ff9] [SHA1: e97e864aaa4789d854363512986e80c0606da996]

I see you removed "hydr1037.bin" from "hydrap2" (the one in question we been talking about) and of course the error is gone now. Is the game flagged as not working in the code? I guess my question is just removing it from the DAT to get rid of the error didn't really fix the issue? Basically the DAT created a non-working ROM set due to "hydrap2" now has missing ROMs? There was several ROMs in "hydrap2" with the same issue that were removed (1037, 1038 & 1039). I was just focusing on the first one in the set.

I always thought the merge tag was used to handle "equal names but different hash conflicts" and then of course the code would look for "game file 1" or "game file 2" when hashing the ROMs at startup before running the game (clone "hydrap2" in this case) depending on if the set is built as Non-Merged or Merged/Split.

If Non-Merged look for "game file 1" and if Merged look for "game file 2". It's the same file but only named different in the parent in merged mode (to avoid conflicts) vs being separate in Non-Merged mode.

		<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>
		<rom name="hydr1038.bin" size="65536" region="sound1" status="nodump" offset="10000"/>
		<rom name="hydr1039.bin" size="65536" region="sound1" status="nodump" offset="20000"/>

This the other thing with Hydra. The parent ROM "hydra" and clone "hydrap" have a ROM with the same name, size and hash values but the clone ROM in "hydrap" is flagged as bad dump but the parent ROM "hydra" is not. Exact same file when looking at the sample pulled from the DAT below.

<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

Hydra and it's clones are an odd duck.

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

@grant2258 said in lr-mame2003 driver improvement and backport:

@riverstorm

ok the dat is 100% fixed for the merge errors the other problem was just the bios wasnt created so you will gave a new file called acpsx.zip. The same link for the updated file if you want to test it mr riverstorm

It looks real good. The same few missing ROMs from using current MAME and rollback:
Hyper Street Fighter II: The Anniversary Edition (Asia 040202) [folder: hsf2a - size: 45mb]
missing rom: hs2ax.03 [size: 524288] [CRC32: 5f3d7397] [SHA1: 96f327dd998105ad5dc46bc9d3b741805a840d68]
missing rom: hs2ax.04 [size: 524288] [CRC32: 59acf108] [SHA1: e68fe233681175b29a35badab249c2b892b23af3]

The King of Fighters 2003 (Decrypted C) [system: Neo-Geo - folder: kof2003d - size: 124mb]
missing rom: 271-v1d.bin [size: 16777216] [CRC32: 2964f36e]

Snk Vs Capcom : Svc Chaos [system: Neo-Geo - folder: svcchaos - size: 114mb]
missing rom: 269-m1d.bin [size: 131072] [CRC32: fd4f0ff9] [SHA1: e97e864aaa4789d854363512986e80c0606da996]
I see you removed "hydr1037.bin" from "hydrap2" (the one in question we been talking about) and of course the error is gone now. Is the game flagged as not working in the code? I guess my question is just removing it from the DAT to get rid of the error didn't really fix the issue? Basically the DAT created a non-working ROM set due to "hydrap2" now has missing ROMs? There was several ROMs in "hydrap2" with the same issue that were removed (1037, 1038 & 1039). I was just focusing on the first one in the set.

I always thought the merge tag was used to handle "equal names but different hash conflicts" and then of course the code would look for "game file 1" or "game file 2" when hashing the ROMs at startup before running the game (clone "hydrap2" in this case) depending on if the set is built as Non-Merged or Merged/Split.

If Non-Merged look for "game file 1" and if Merged look for "game file 2". It's the same file but only named different in the parent in merged mode (to avoid conflicts) vs being separate in Non-Merged mode.
		<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>
		<rom name="hydr1038.bin" size="65536" region="sound1" status="nodump" offset="10000"/>
		<rom name="hydr1039.bin" size="65536" region="sound1" status="nodump" offset="20000"/>
This the other thing with Hydra. The parent ROM "hydra" and clone "hydrap" have a ROM with the same name, size and hash values but the clone ROM in "hydrap" is flagged as bad dump but the parent ROM "hydra" is not. Exact same file when looking at the sample pulled from the DAT below.
<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>
Hydra and it's clones are an odd duck.

If you know whats going on in the backgound it makes sense the changes are here. Just for you information a bad dump does have a crc and sha. A nodump has never been dumped . This translates to we have a bad dump use hydr1037.bin from the parent set in dat file

https://github.com/arcadez/mame2003-plus-libretro/pull/17/files/863ed97ef07b9feea0c0b9ddac232a6109aa5c2e

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

If you know whats going on in the backgound it makes sense the changes are here.

There's plenty happening in the background that I don't understand. I am just trying to understand what doesn't make sense to me. Yes I know a bad dump vs no dump or the hash value in DAT would be truly random. :)

Explain this to me. The two files between parent/clone are 100% identical why flag one as "bad dump" and other not? Yes you showed me the code that you patched MAME so that it doesn't process baddump files.

On the other hand ClrMamePro isn't aware of the MAME code when processing a DAT. It's just building sets according to the how the DAT is built. ClrMamePro can use the "baddump" flag when processing (depending on options selected) but once complete there's no indication of bad dump in the ROM set so why the different flags on identical files shown below?

<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

The other issue I see is the original DAT has hash collisions. Look at the 3 lines below. It's the same file from 3 games (1 is the parent and 2 are the clones). If you run this through ClrMamePro it will not process correctly due to insufficient information and create no entries in the parent. That's the error ClrMamePro is warning about. The lack of hash information in the "hydrap2" ROM treats it as a different hash when actually it's no hash.

Anothe way is when doing a merge set (not split) you would need to at least use the hash collision detection option which creates a whole other challenge of subdirectories of the hash collisions. Well actually all merge files and not just hash collisions. You can test this with the original DAT if you want to see it for yourself.

Your fix was to remove the offending files from the DAT so now you have an imcomplete set that doesn't work but the warnings in ClrMamePro are fixed. The better fix seems to either fix the game and get the correct information or just entirely remove the clone because you already ripped 3 files to avoid the errors.

<rom name="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

<rom name="hydr1037.bin" size="65536" region="sound1" status="nodump" offset="0"/>

My questions are more about "quick" fixes don't seem correct to me vs. doing it right. It's your project and I'll support it for sure as I would love to see these changes get merged to the main. To stay static and never add another game does seem silly to me when there's some nice changes that could enhance it and you and Mark have the know how to do it.

Another game over the weekend I noticed that didn't work was Alien Storm that's part of these enhancements. Donkey Kong full sound samples. Actually there's several that look good to me and would be great additions.