lr-mame2003 driver improvement and backport

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

This is the way the dat is staying the old way it was, consider this beta1 only changes will be games added. Unless something else pops up i think we are good to go

Just to let you know I don't mind if hydra and clones work. In fact I don't even know the game. :) I was just focused on DAT correctness/completeness portion of the discussion.

I never put much thought into whether it was a manual process or automatic but seeing the commit and it was released shortly prior I figured it was automatic. Not sure that changes much.

I spent quite a bit of time working on a DAT of the original changes arcadez did and I spent a lot of time modifying the DAT and loading it over and over so you kind of get an idea of what causes ClrMamePro to balk.

I saw the hydra entries were removed and I figured that doesn't seem like a fix really. The errors it was causing only really pertain to a merge set. In split or non-merged set it will just remove the parent/clone relationship.

I did come from a programming background and I've yet to meet a programmer not dialed to detail so I figured you were pulling my leg a little on some of the questions I was asking.

I wish I had more time to focus/learn programming as I do enjoy it but at the end of the day I usually spend time with family until our youngest goes to bed. Then I work out, which I ramped up to 7 days these past few years. I round robin weight, cardiovascular, even some yoga (I used to think it was for sissies and pansies but it's great for core) as it's tougher to maintain muscle mass as we age.

My oldest is married now and could have kids anytime but my youngest well I hope to stay fairly active at least through her informative years. She's at an age where we race through department stores..."Come on Dad, let's race!" and I don't much care if I look silly watching her run and cat call while laughing as long as I am active with her. Nothing like life through the eyes of a child. So pure, innocent & enthusiastic. Then eat, maybe a little on the computer, read or whatever and sleep. We have pretty "boring" week but I actually like my home life.

RetroPie "testing" allows for the whole family to play. I prefer MAME and ScummVM, the wife likes NES and SNES and Lexi likes it all. Programming well not really a group activity but I did some pretty fun overclocking this past weekend as we spent the whole weekend at home.

Anyhow, off track, I will pull down the new DAT later today for testing. I was wondering if their would be any benefit to creating a difference a difference DAT between mame2003-plus and lr-mame2003. Not at a ROM file level but at a set level.

Basically any ROM set that has at least one ROM file in the archive that is different create a DAT entry for the whole set. I have an idea how I would go about it and I don't think it would be to difficult. It wouldn't be as eloquent as a program. The thought it would allow a folder of just changed ROMs vs a whole set which would save several GB. Thoughts on that or a waste of time?

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

of ju

Well sets will change a little in the driver i updated in some cases chingat for example but its no new roms it will fix on a rebuild i posted on the github page what i done. I wasnt pulling your leg at all sorry if i gave that impression. I was just trying to tell you the problem was at the driver level it shows itself before we even touched the dat with bad merges. I hope you understand what i meant now if you re read it. I myself dont have as much free time as i used too but i used to program in mame years ago so am familiar with some of its workings. The problem simplified is no dump doesnt display the crc and sha and when the filename clashed it cause the merge error instead of merging

Riverstorm

Yeah I understand. I do have a habit or rereading threads hoping some crystal ball clarity and tone will come through writing that just never happens...what are they trying to explain/say! ;) I surprisingly don't text except for emergencies even though work requires 24/7 phone availability, I don't care about social media and prefer a phone call or even better yet come darken my doorstep for a beer/coffee whatever even unannounced is fine. I love technology but like "time away".

I think we are loosing some interpersonal and face-to-face communication skills in the digital age. Simply how to sit around and shoot the breeze face to face without feeling anxious. Hopefully we don't all end up agoraphobic working out of clones like Surrogates. :)

Did you have any thoughts on a difference DAT between lr-mame2003 and mame2003-plus? I have a habit of building complete Non-Merged sets, zipping them up and storing but with all the sets RetroPie supports it's a bit space consuming. It might be useful though for emulator swapping/testing but then again most of the sets are additions it seems.

markwkidd

@riverstorm said in lr-mame2003 driver improvement and backport:

Anyhow, off track, I will pull down the new DAT later today for testing. I was wondering if their would be any benefit to creating a difference a difference DAT between mame2003-plus and lr-mame2003. Not at a ROM file level but at a set level.

That is what I did and posted earlier, but is out of date now. I don't have much time to read the forums or work on emulation for another week or two but this is my process.

Please excuse me if you already know this, since it sounds like you've spent plenty of time with ClrMamePro too.

Start with a clean, Full Non-Merged MAME 0.78 collection: Non-Merged in the Scanner and Rebuilder settings, plus uncheck Separate BIOS Sets in the Scanner and Rebuilder advance settings
Open the mame2003-plus XML DAT in ClrMamePro (for now say yes to any merge errors :) )
Again set the Scanner and Rebuild to Non-Merged with Separate BIOS Sets disabled
Run Rebuild with the MAME 0.78 Full Non-Merged collection as the source
Run the Scanner on the new set that is rebuilt during the previous step
Right click on the scanner results to save a fixdat for all selected issues

The fixdat only has records for the individual ROMs and samples that one needs to track down in order to upgrade a MAME 0.78 set to a MAME 2003+ set.

Riverstorm

@markwkidd said in lr-mame2003 driver improvement and backport:

The fixdat only has records for the individual ROMs and samples that one needs to track down in order to upgrade a MAME 0.78 set to a MAME 2003+ set.

Ok, well, shoot, your method is much easier and quicker. I created two separate sets, and then torrentzip to create a consistent hash (which I always do). Taking it further you could WinMerge the differences moved to a new folder and run it against DatUtil.

Riverstorm

I did run the new DAT it works fine except the few same missing ROMs that have always been there.

You guys may know this already but I thought I would share it just in case it was helpful pertaining to DATs. When I said you can remove the SHA1 yesterday it was not explained well. I should have said "field".

Now that hydrap2 has the same hash values as hydra and hydrap you can still use the "no dump" flag on hydrap2 if it is really no dump and not bad bump. It will be fine. Basically if hydrap2 is more accurate as "no dump" it can be flagged that way and leave hydrap as "bad dump". ClrMamePro has no issue with doing it that way.

Just to use a few examples that are illustrative only. The two lines below would be consider the same to ClrMamePro. Notice the only difference is the status? hyrdap can be "baddump" and hydrap2 can be "no dump".

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

>rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="no dump" offset="0"/>

Sticking with the same two line these will also be considered a match and the same. Notice the SHA1 is removed but to ClrMamePro this is still 100% match and will not produce warnings on profile load.

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0"  region="sound1" status="baddump" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0"  region="sound1" status="no dump" offset="0"/>

One last example using the same two lines. Everything is identical except the size field is missing in one of the lines. The two lines below are NOT a match and will throw an warning in ClrMamePro. I used this flexibility to my advantage for locating obscure ROM files that were hard to find.

<rom name="hydr1037.bin" merge="hydr1037.bin" size="65536" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="baddump" offset="0"/>

<rom name="hydr1037.bin" merge="hydr1037.bin" crc="b974d3d0" sha1="67ecb17386f4be00c03661de14deff77b8ca85d0" region="sound1" status="no dump" offset="0"/>

The only requirement is the name, size, crc & sha1 match but you can remove any or all fields (except name I think it has to remain) as long as it's consistent among the parent and all clones.

Merge tags are excluded from the parent for obvious reasons.

grant2258

@riverstorm said in lr-mame2003 driver improvement and backport:

"no dump" it can be flagged that way and leave hydrap as "bad dump". ClrMamePro has no issue with doing it that way.
Just to use a few examples that are illustrative only. The two line

mame ouputs nodumps with no crc or sha its has to be changed to badump or goodump in the driver to output the sha mr river thats why i removed the no dumps in the first place then added them back on because it was thought as a quick fix . I still think they arent needed but wont do any harm so will leave then in.

grant2258

check the changes to the datfile on github there is a history of what has changed there is a history button at the bottom. It will save you doing a lot of diffs https://github.com/arcadez/mame2003-plus-libretro/blob/master/metadata/mame2003.xml. post any issues there its easier for us to find the information all input is good :)

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

I still think they arent needed but wont do any harm so will leave then in.

I guess what I don't understand is why not leave them as "no dump" and just answer the warnings in ClrMamePro? That's what they are (no dump) and it's 100% accurate. The official set does have these errors in abundance. I can't even remember that last time or ever I saw a perfectly clean set. These errors are normal and ok even if they can be confusing when you first see them.

A couple of sets I see that were in lr-mame2003 but have been removed from mame2003-plus.

Cabal (US set 2) (cabal2)
Urashima Mahjong (urashima)

check the changes to the datfile on github there is a history of what has changed there is a history button at the bottom. It will save you doing a lot of diffs

Sounds good.

Riverstorm

@markwkidd said in lr-mame2003 driver improvement and backport:

The fixdat only has records for the individual ROMs and samples that one needs to track down in order to upgrade a MAME 0.78 set to a MAME 2003+ set.

@markwkidd - I was thinking about this last night and it's incredibly accurate down to the individual ROM's. What I was proposing is slightly different.

If you take an existing lr-mame2003 set and run it through you get exactly what's missing down to the ROM. So take for example "game a" needs "rom 1", "rom 2" & "rom 3" to run. After I run the lr-mame2003 set through with the mame2003-plus DAT it builds "game a" adding "rom 1" & "rom 2" but it can't find "rom 3" because it was updated for mame2003-plus.

If you create a set with that diff DAT now I have ROMs in 2 partial sets that still need to be merged to make a working set.

What I want to do is say "game a" is missing any file then I want to create the DAT entry that includes the whole game doing this for the whole set. For example "game a" is missing "rom 3" then I would create a DAT entry "game a" with roms 1, 2 & 3.

So let's say I am running mame2003-plus and I want to run "game a" all I need to do is copy it from my new folder and it's playable but with your DAT "game a" is not playable until the two partial sets have been merged.

I am not sure I am explaining that well as it's a slight difference but makes the ROMs more usable and lr-mame2003 it still 100% accurate for the ROMs that are transferable are in the set with the newly added or changed ROM sets if copied over an existing set. Completely new sets will obviously be ok as they aren't overwriting anything but just being added to the game roms folder.

grant2258

@riverstorm

hiya mate is the way mame outputs the nodump there is no crc and sha1 and thats the way it should be the hydra prototypes are both missing dumps as well as chinagat (parent rom). These games borrow other sets roms to work so they are considered badumps for that set . See the latest mame does it the same way to keep the parent clone relationship. It makes extra files if your using spilt sets that are not needed. It also causes merge errors . If it makes you feel better ill make you a data with no tags in for your personal use. The only other alternative is remove the prototypes because they both missing roms or make the prototype unplayable

latest mame

https://github.com/mamedev/mame/blob/master/src/mame/drivers/atarig1.cpp

line 670 hydrap2

someone posted a issue about two set being missing on the github im sure arcadez will add them back

Riverstorm

@grant2258 said in lr-mame2003 driver improvement and backport:

These games borrow other sets roms to work so they are considered badumps for that set .

It's been real and it's been interesting but not real interesting. :) If you can borrow set roms and make it (hydrap2) work then it might become an instant classic for me! ;) I'll be the first in line to test it.

In the first revision removing them was your answer. (I think I would have left them as "no dump" as long as the parent is working.)

The second revision you marked the 19 "no dump" ROMs files as "bad dump" and pulled them down from the parent. Again I would have just left them as "no dump" so they align with the original lr-mame2003 DAT and it reflects proper.

As it sits now over half your ROM files (19 to be exact) are not original. You can call it a hack or hybrid or hacked prototype or hacked hybrid prototype or whatever feels comfortable for you but it doesn't seem that important to get a, as far as know, unpopular prototype working.

No need to create a special DAT. You know that's only a flag and has no bearing on the set build unless certain options are enabled? If you can get it working why change the flags to "no dump". You understand how ClrMamePro works?

Also just because it's flagged as "no dump" (in the DAT) doesn't mean you can't keep the parent/clone relationship. Of course "no dump" doesn't have a crc or sha1 because one doesn't exist.

I guess you can give a go and we'll see what you can do.

Cheers, mate! ;)

grant2258

@riverstorm

I know how clrmame works im not coming at it from clrmame pro im coming at it from mame. If you mark nodump in mame the dat wont output the crcs and you loose the parent clone relationship when building the roms. Thats why your loosing the parent/child relationship which is using the roms from hydra from the prototypes(the errors on the original dat). Im my first revision yes i removed the nodumps and said the drivers needed fixed and i fixed the drivers and put the no dumps back in at your request of being a quick fix.

ps the nodump borrows there roms from hydra as bad merges its what the drivers is doing as bad way originally. Im not sure what your issue is

what really confuses me is your happy to say nothing about hydrap doing this the way it originally was

Riverstorm

If you wish to see a good example of parent/clone relationship with "no dump" ROMs as part of the set check out hydra and hydrap2 in the lr-mame2003 DAT. It's a good example and implemented proper the way it should be. Known files have the correct field information and "no dumps" are marked accordingly.

Thats why your loosing the parent/child relationship which is using the roms from hydra from the prototypes(the errors on the original dat).

This is only partially true. When working in "full merge" mode. Reread the warning you still not understanding it. You don't loose the parent/clone relationship in split or non-merged with hydra and hydrap2 the warning is ONLY for full merged mode but again this is still irrelevant due to the fact they are "no dump" ROMs and no ROM file exists to add to the parent in full merge mode. The error warning is a bit cyclic but it is true and how it presents itself.

If you mark nodump in mame the dat wont output the crcs

Of course it can't output the crc because it doesn't exist. In addition you wrote the commit to not process "no dump" flagged ROMs.

I think whoever created the 0.78 DAT did it correctly. You should be able to do the same or they can help with the code to get a proper DAT output? They added in the known ROM file information (name, size, crc, hash) and added "no dump" to the unknown ROMs because the information was not known at the time of DAT creation. Hence see hydra & hydrap2 in the lr-mame2003 DAT on how that works.

I respectfully am going to bow out here as this is doing no good and if you edit that post one time it might explode!

grant2258

@riverstorm

https://github.com/grant2258/mame2003-libretro/commits/master/metadata/MAME 0.78 XML.dat

mark edited this file to fix it originally. There is no new files with the fix hydra fix and the parent clone is correct form split sets it the way this roms should be with what i changed.

you should take this subject up with mark if you believe the bad merges are right again it was hand edited to be fixed before as well as chinagat to be fixed

grant2258

https://retropie.org.uk/forum/topic/16833/mame-2003-gamelist-xml-with-data-from-the-binary/54

maxbeanz

Could you also put a Samples DAT for mame2003plus on the Github so we can get an exact set ?

Riverstorm

@maxbeanz said in lr-mame2003 driver improvement and backport:

Could you also put a Samples DAT for mame2003plus on the Github so we can get an exact set ?

Samples are already built into the DAT. If you only want Samples just uncheck Sets, ROMs & CHDs in the Scanner and it should work fine. You have to set a Samples directory so they will be built in there own directory separate from the ROMs.

maxbeanz

@riverstorm It does, thanks. It doesn't seem to delete uneeded ones though. I could put a copy of dkong.zip and it doesn't delete it. I'll have to check the filenames of all the games it detects and make sure there are no surplass.

EDIT: All done now.

Riverstorm

@riverstorm It doesn't seem to delete uneeded ones though.

It doesn't delete them? I am not quite sure what you mean.

Samples are an odd when processed. They have no hash value so if you have the correct name with an incorrect sample it will accept it as valid.

It should remove extra samples that are not part of a set. Say dkongjr.zip contains dkstomp.wav (unneeded sample for that set) it will remove it from dkongjr.zip.

Also it removes incorrect sets altogether (a sample set that is not part of the samples i.e. - rocket.zip or something for example that doesn't exist in mame2003-plus).

The thing it does not do is say dkong.zip is missing dkstomp.wav but dkongjr.zip contains dkstomp.wav. It will remove it from dkongjr.zip but it will not add it to dkong.zip. So you can run ClrMamePro to clean up unneeded samples but then you have to manually add the missing ones back if that makes sense. Even a Rebuild with the correct path/backup path will not add them back to the correct set.

This is only with Samples and not ROMs though.