Development of module-script generator for lr-mess, lr-mame and mame standalone

Folly

EDIT :
The scripts can be found here.

first issue has been fixed in this commit
change test versions to 253.07
- test versions can now update itself using 00-workdir-00
- moved TEST scripts to 00-workdir-00/supplementary/
- fix github download function for ubuntu 22.04 using a different string variable which fixes downloading retroarch joypad autoconfigs
- Mark them as TEST versions
change test versions to 253.08
- add sha1sum of updated runcommand.sh which improves adding %SOFTLIST% variable in runcommand
for more improvements see here

I have been busy for 3 days now and I am about ready to share a major update.
I made quite some changes in the runcommand loaders, adding new replace tokens to improve many things.
The standard %ROM$ and %BASENAME% are such replace tokens.

I added :

%ROMDIR%
The "%ROMDIR% token" that can be used to add the rompath in a runcommand.
When using a runcommand using %BASENAME$, the rom can always be found, even in a sub-directory. This also makes it possible to categorise roms in different folders within a roms directory.
%DQUOTE% :
Though the "%ROMDIR% token" does not have double quoting.
Why? Because lr-mame and lr-mess can't cope with it.
For this reason the "%DQUOTE% token" is added.
The "%DQUOTE% token" can be placed in other places to get the runcommand working especially when using subdirectories that have spaces in them.
For MAME standalone this will work :
-rompath /home/pi/RetroPie/BIOS/mame;%DQUOTE%%ROMDIR%%DQUOTE%
For lr-mame and lr-mess this will work :
-rompath %DQUOTE%/home/pi/RetroPie/BIOS/mame;%ROMDIR%/%DQUOTE%
All other attempts to add double quotes to runcommands in a different way failed so using the "%DQUOTE% token" seems to only way to go.
%CLEANBASENAME% :
The "%CLEANBASENAME% token" is needed for creating a proper per game config directory so mame system configs can be created per game.
For MAME standalone this would not be necessary but lr-mame and lr-mess can't cope with spaces and double quotes in the wrong places.
Therefor the "%CLEANBASENAME% token" can be used which removes all spaces and special characters and is without any double quotes.
The "%CLEANBASENAME% token" could also be used to make a clean savestate file.
%SOFTLIST% :
The "%SOFTLIST% token" is needed to force proper softlist loading.
For example pacmania is in msx1_cass and in msx2_cart.
If one loads a game with a basename loader, for example pacmania, which comes from msx2_cart, then normally the the soflist msx1_cass has priority over msx2_cart.
So mame thinks it's pacmania from msx1_cass but it is actually pacmania from msx2_cart.
To fix this issue we need to force the correct softlist before the "%BASENAME% token".
The %SOFTLIST% token is the last folder name where the rom is in plus an extra => :.
For pacmania we will have to place it in the folder "msx2_cart".
In the loader you will see it as %SOFTLIST%%BASENAME%, eventually for pacmania it will look like this "msx2_cart:pacmania".
If the rom is in a rompath without => _ then the "%SOFTLIST% token" will be empty and mame will guess the correct softlist.
If "%SOFTLIST% token" gets a value for some reason, because a _ char is in the path, then mame will refuse to load the game.
A check in the runcommand.log will reveal the issue.
( although it's an improvement, for example, msx1_bee_card softlist will not benefit from it, as it needs the beepack inserted as a slot. So softlists that need slot options will still not work with the regular basename loaders )

To be able to use these tokens they have to be in the runcommand.sh script.
My new scripts are able to add that runcommand.sh.

My both scripts start now with a clean mamedev_csv array now.
Now both scripts work fine when switching from one to the other.

I also made a minor change to the run_mess.sh script to accommodate spaces in the rompath.

Perhaps I forgot something will look at that later.

I would like to discuss the new stuff before I commit because it's quite a change and testing needs to be done.

What do you think about the changes and the testing ?

Folly

@Ashpool

You are welcome to join in and test the new scripts.

Ashpool

@Folly Well, I am just creating a fresh ubuntu vm to get a "fresh" base... on the raspi install I tried it on (manual retropie install over raspberry legacy os), 1st thing i noticed/came into trouble was the Game and Watch Setup. The Script (still) lists mame/lr-mess/lr-gw, and this post of yours sounded that mame/lr-mess are still usable, but the es_systems.cfg just lists .mgw|.MGW as extensions and only lr.gw is available as an emulator from within the runcommand menu.
But that OS "install" has become a mess in recent time and i am just waiting for an excuse to dump it in favour of bullseye 64bit (as that install wasn't really meant to incorporate retropie at 1st, retropie not supporting that ain't really a drawback here), so it may be a "just on my system"-case and not a fault from your script.

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

@Folly Well, I am just creating a fresh ubuntu vm to get a "fresh" base...

Good idea to make a fresh vm.
Do you use VWware or something else ?
I would suggest pure debian (I am using mate desktop) instead of ubuntu.
Although ubuntu looks very nice I had many issues with ubuntu in the past.
Only thing with debian is that you need to add your user name to the /etc/sudoers file to be able to use the command sudo.
If you are ready then I would also suggest to put my scripts in the ext directory :
RetroPie-Setup/ext/RetroPie-Share/scriptmodules/supplementary
That way the 3rd-party module-scripts are nicely separated.
I prefer using the mamedev.sh script now over the add-system-mamedev.sh script because :

it's faster
it will not generate module-scripts but will install systems or categories directly

Only disadvantage is removing a system or category but that can always be done manually if really needed.
Though, testing needs to be done on both versions as I still update both.
Remember : the test versions (253.06) can't be updated right now because they are still in the 00-workdir-00. If update is selected it will downgrade to 253.05.

on the raspi install I tried it on (manual retropie install over raspberry legacy os), 1st thing i noticed/came into trouble was the Game and Watch Setup. The Script (still) lists mame/lr-mess/lr-gw, and this post of yours sounded that mame/lr-mess are still usable, but the es_systems.cfg just lists .mgw|.MGW as extensions and only lr.gw is available as an emulator from within the runcommand menu.

Looks like you missed the update.
The thing you describe was the issue.
It should be solved or you have installed lr-gw the regular way again ( not from my script ), that could cause this issue also.
I checked my code and it should be ok, though the help message can be updated a bit.
I like to know if you still experience it when you do it on the fresh vm.
@DTEAM and I also implemented predefined way of loading the madrigals roms so switching between emulators is not needed between mame/lr-mess or lr-gw when putting all mame and madrigals in one folder.
(this is already in the help btw.)

But that OS "install" has become a mess in recent time and i am just waiting for an excuse to dump it in flavour of bullseye 64bit (as that install wasn't really meant to incorporate retropie at 1st, retropie not supporting that ain't really a drawback here), so it may be a "just on my system"-case and not a fault from your script.

I have been using it on Debian bullseye 64 bit for quite a while now. I have VM's on both x86_64 and on aarch64(arm).
Although earlier on I had to patch some stuff seems not really needed anymore using it on 64bit.
It should work fine when the basics are good.

Let me know how it goes with the fresh vm.

DTEAM

@Ashpool

@Folly said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

@DTEAM and I also implemented predefined way of loading the madrigals roms so switching between emulators is not needed between mame/lr-mess or lr-gw when putting all mame and madrigals in one folder.
(this is already in the help btw.)

See this post : https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-lr-mame-and-mame-standalone/1338?_=1684160654975

as @Folly said, the script takes care of it now.

Ashpool

@Folly

Remember : the test versions (253.06) can't be updated right now because they are still in the 00-workdir-00. If update is selected it will downgrade to 253.05.

That was the one I've used, update option not choosen and no previous (AFAIR)/later lr-gw install.

Do you use VWware or something else ?

I've tried virtualbox in pre-Win7 times, but since vmware(-player) offered native support for xpmode vms, i've switched to that player and well, stayed with it so far (though for legacy systems as win9x, i would prefer QEMU/PCem/DOSBox-x)) .

Still, I guess that I have to raise the memory used by the VM from 4 GB to 8+, the mame compile (initiated via mamedev.sh) failed/retropie crashed back to the desktop... in the meantime i also tried add-mamedev*.sh on a fresh &updated retropie image (the basic retropie install one i've created for my backdrop experiments) and at least there g&w installed fine.
Will play with that setup in the next few days and report back (I've never really tried software lists/mess systems and maybe elite on the archimedes could be a self-set milestone I want to accomplish ;] )

Edit: At least the ArchElite (ever wanted to play that since I read the revue in a Computer magazine back in its time, and though Ian Bell himself is hosting various elite versions [1] to be used on emulators/real machines since aeons - i never came to try it) goal may be troublesome, i thought of giving it a try 'cause of nonmame mentioning the acorn/archimedes systems as mature in mame, but querying the systems from the xml/arcadeitalia-database ain't that promising. We may see ( honestly, previously i only ever used your scripts for g&w out of childhood memories ;] ).

1: And also "Elite - the musical", I really love "a contract is a contract" from that ;D

Edit: -> P.S.: OK, maybe those memories/anectodes would be better placed/suited within the Old Computer Appreciation Thread. ...

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

Still, I guess that I have to raise the memory used by the VM from 4 GB to 8+, the mame compile (initiated via mamedev.sh) failed/retropie crashed back to the desktop...

Indeed, compiling mame takes a lot of memory.
I use 8 GB, that should work.
You have to know that the install-lines for mame/lr-mame and lr-mess are just links/redirections to the original module-scripts.
They are there so everything is in reach and that someone doesn't forget to install it.
Basically it doesn't matter if you install them from the experimental packages or from my module-script.
Is this information I should add in the help for these lines ?

in the meantime i also tried add-mamedev*.sh on a fresh &updated retropie image (the basic retropie install one i've created for my backdrop experiments) and at least there g&w installed fine.

Aha, good to hear !

Will play with that setup in the next few days and report back (I've never really tried software lists/mess systems and maybe elite on the archimedes could be a self-set milestone I want to accomplish ;] )

Sorry some bad news ;-)
I just tested elite and it will show the files when opening 0: but it will not boot when selected.
In the hash table archimendes.xml you can find elite as not supported.
So you probably can't get it working on archimedes with mame or lr-mess.

Ashpool

@Folly said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

I use 8 GB, that should work.
You have to know that the install-lines for mame/lr-mame and lr-mess are just links/redirections to the original module-scripts.

Just to report back on the VM... (raspi/binary installs and tests still going fine): to be sure I've used 12GB and mame compiled without a flaw, now it is lr-mess wich breaks (no crash, just fail):

Compressing src/mame/layout/zoo.lay...
make: *** [makefile:1333: build/projects/retro/mamemess/gmake-linux/Makefile] Segmentation fault (core dumped)

No Fun, and not a fault of the script (as you mentioned it just calls the native retropie script)... bloody thing is, i installed all that in a previous VM (including games&watch via add-mamedev-systems.sh) earlier this year and it all went without a problem (also on Ubu 22.04) - sigh! But I hadn't encountered this "error" back then, now I have :[

Well, to cite a somewhat famous movie series "I am getting to old for this s*..", otherwise said "It may be that we are indeed living in interesting times" ;]

P.S.: Damned and curses to hear about "<software name="elite" supported="no">", wasn't there a way to disable softlist s and load images via CMD (Just to see what will happen)?

Edit: Wasn't there an option to convert(/copy?) mame art to be used by lr-mess? I can't find that (at the moment) in the (current) tool. If mame is selected as emulator, the downloaded artwork works fine, but it ain't present in lr-mess)

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

Just to report back on the VM... (raspi/binary installs and tests still going fine): to be sure I've used 12GB and mame compiled without a flaw, now it is lr-mess which breaks (no crash, just fail):

Not that much time but if I can I will try to compile lr-mess and see what happens.

Well, to cite a somewhat famous movie series "I am getting to old for this s*..", otherwise said "It may be that we are indeed living in interesting times" ;]

Indeed, nice quote ;-)

P.S.: Damned and curses to hear about "<software name="elite" supported="no">", wasn't there a way to disable softlist s and load images via CMD (Just to see what will happen)?

It will load with softlist enabled, you will get into the OS of Archimedes and the files can be seen when opening the drive in the explorer. But the game will not run it will freeze. You can test with softlist disabled, that means just using the runcommand with the media option -flop (this will disable the softlist loading in the specific custom-core-options.cfg and is loaded via the retroarch.cfg.add).
Not tested yet, but this will likely have the same result.
Sure, you can use a .cmd file, if that is what you mean, but you will have to create one, and it will have the same result as mentioned earlier.
(if having a good working .cmd file, that is)

Edit: Wasn't there an option to convert(/copy?) mame art to be used by lr-mess? I can't find that (at the moment) in the (current) tool. If mame is selected as emulator, the downloaded artwork works fine, but it ain't present in lr-mess)

You have to download the artwork first and then you will be able to convert the artwork into Retroarch backgrounds/overlays.
First select :

Download/update mame artwork (+/-30 min.)
Then select :
Create RetroArch background-overlays from artwork
Create RetroArch 16:9 bezel-overlays from artwork

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

now it is lr-mess wich breaks (no crash, just fail):

Indeed it does not compile.
Tested Debian 10 (x86), Debian 11 and Ubuntu 22.04 on x86_64 VM.
Seemed an issue with the Makefile.libretro that has been changed 5 days ago.
Though I tried the older Makefile.libretro and sadly still fails.

What a nightmare again ;-(
Will have a look later.

I will open a separate thread for this :
https://retropie.org.uk/forum/topic/34238/lr-mess-will-not-compile-17-may-2023

Edit :
lr-mame has the same issue !

Folly

@Ashpool

One or more commits, in lr-mess from May, have broken something that will break compiling lr-mess.

There is no solution yet, but you can use an earlier commit to use a bit older source from april to compile.

The only thing you have to do is add the commit to the original lr-mess module-script, like this (cd5.....1b):

rp_module_repo="git https://github.com/libretro/mame.git master cd5f7d077cdab2eac09faf642ddd121d80f3ad1b"

The only disadvantage is that about 1,5Gb is dowloaded as source so downloading will take a bit longer.

Once the source is ok again you can revert the line again to :

rp_module_repo="git https://github.com/libretro/mame.git master"

ps :
lr-mame has the same issue because it comes from the same source !
Above temporary solution will also work for lr-mame.

Ashpool

@Folly Well, after some meddling with the script and mess systems, I can only say: so far (as said, had just used it before for the handhelds) I was pleasantly surprised that it covered a lot of the hazards I feared so far whence using mess systems. But somewhat disappointing is that from all things I tried (appleII|appleIIc/bbcb/pc8801/lynx), only lynx was somewhat satisfying and the other ones had various issues, but as those are not related to the scriptmodules, i also have to say that neither i had found anything that I could report in this regards :]

Folly

@Ashpool

Thanks for testing ;-)
Did you already discover that you can update the test versions from within the script ?
I am planning to do some more work when I have the time.

Will try those systems you mentioned and see what results I get.
Indeed much is related to the source code of mame itself.
For bbcb I know it works but other emulators do a better job.
What was your issue with bbcb ?

Btw,
I am working and testing now on Ubuntu 22.04 ~~and all seems to work pretty good now.~~

I ~~only~~ had 2 issues so far :

auto removing the top-bar so emulationstation will use the whole screen. (installing extension manager and enable hide top-bar)
changing some shortcut key-combinations in OSX so they don't interfere with ctrl in combination with the cursor keys in the Ubuntu VM (^->,^<-, etc)

Edit :
Sadly, Ubuntu 22.04 has a lot more issues described over here ;-(

Ashpool

@Folly said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

Did you already discover that you can update the test versions from within the script ?

Nope, good to hear ;) last think I remember is:

 Remember : the test versions (253.06) can't be updated right now because they are still in the 00-workdir-00. If update is selected it will downgrade to 253.05.

What was your issue with bbcb ?

gfx glitch in "elitea", the cobras "console"/interior ploted fine, but the "outside" view/grafic wasn't fitting the screen (only ~1/4 visible, as if zoomed into the topleft position of a virtual display).

I am working and testing now on Ubuntu 22.04 and all seems to work pretty good now.
I only had 2 issues so far :
auto removing the top-bar so emulationstation will use the whole screen. (installing extension manager and enable hide top-bar)

Mh, didn't know that hiding the top-bar was possible, I avoided it by "falling back" from wayland to xorg.

Folly

@Ashpool

I am adding update info in this post now :
https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-lr-mame-and-mame-standalone/1364

You first have to update manually, after that updating is possible from within the script.

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

gfx glitch in "elitea", the cobras "console"/interior ploted fine, but the "outside" view/grafic wasn't fitting the screen (only ~1/4 visible, as if zoomed into the topleft position of a virtual display).

Indeed too bad, I have the same issue.
Also tested bbcm but that makes no difference.

DTEAM

@Folly

Hi Folly,

I tried to use th PCECD add-on with:

mame-system-pcecd-cdrm-autoframeskip = "/opt/retropie/emulators/mame/mame -v -c -autoframeskip -ui_active pce pcecd  -cdrm %ROM%"

I tried to use the "no intro romset" and it doesn't work.

If I don't use the "basename" command, normally I should be able to use a ROM no matter the name? Right?
I don't remember if it's different for .chd (cd ) games.

Here's my log. If I use the good MAME Rom, It should work, but I'm trying to use it without the hashfile.

Parameters: 
Executing: /opt/retropie/emulators/mame/mame -v -c -autoframeskip -ui_active pce pcecd  -cdrm "/home/pi/RetroPie/roms/tg-cd/Buster Bros. (US).chd"

"pcecd" approximately matches the following
supported software items (best match first):

* Software list "pcecd" (NEC PC Engine / Turbografx-16 CD-ROMs) matches: 
fanspc1           PCEngine Fan Special CD-ROM Volume I (Japan)
hyprcata          PCEngine Hyper Catalog Duo-RX - Disc A (Japan)
hyprcatb          PCEngine Hyper Catalog Duo-RX - Disc B (Japan)
hypercap          PCEngine Hyper Catalog (CD-ROM Capsule) (Japan)
policeco          Police Connection (Japan)
prinmak2          Princess Maker 2 (ACD, SCD)(Japan)
spcfzone          Space Fantasy Zone (CD, SCD)(Japan, prototype)
psycdet3          Psychic Detective Series Vol. III - Aya (Japan)
psycdet4          Psychic Detective Series Vol. IV - Orgel (Japan)
psycdet3d         Psychic Detective Series Vol. III - Aya - Auto Demo (Japan)
hyperdyn          Hyper Dyne SideArms Special (Japan)
princesm          Princess Minerva (Japan)
priveyed          Private eye dol (ACD, SCD)(Japan)
psychics          Psychic Storm (Japan)
spaceinv          Space Invaders - The Original Game (Japan)
supersch          Super Schwartzschild (CD, SCD)(Japan)

* Software list "pce" (NEC PC Engine HuCards) matches: 
pcdenj            PC Denjin - Punkic Cyborgs
pcdenja           PC Denjin - Punkic Cyborgs (alt)
pcgenj            PC Genjin - Pithecanthropus Computerurus
pacland           Pac-land
pcgenja           PC Genjin - Pithecanthropus Computerurus (alt)
pcgenj2           PC Genjin 2 - Pithecanthropus Computerurus
pcgenj3           PC Genjin 3 - Pithecanthropus Computerurus
pcpachi           PC Pachi-slot
gecd              Games Express CD Card
circusld          Circus Lido
cyberdod          Cyber Dodge
pcgenj3t          PC Genjin 3 - Pithecanthropus Computerurus (Taikenban)
spaceinv          Space Invaders - Fukkatsu no Hi
makaipri          Makai Prince Dorabocchan
cybrcore          Cyber Core
jchan             Jackie Chan

Folly

@DTEAM

Just found something about pcecd I read that it's an addon.

Will have a look if I can find the files and how to load it.

This command gives information that -cdrm (.chd) file can be loaded with just using pce.

pi@pi-2204:~$ /opt/retropie/emulators/mame/mame -listmedia pce
SYSTEM           MEDIA NAME       (brief)    IMAGE FILE EXTENSIONS SUPPORTED
---------------- --------------------------- -------------------------------
pce              cdrom            (cdrm)     .chd  .cue  .toc  .nrg  .gdi  .iso  .cdr  
pce              cartridge        (cart)     .pce  .bin

So, again, I think you have typo in the runcommand :
pce pcecd

Just use :
pce

Edit :
Above did not work.
It will ask for a cartridge.
Tried gecd boots but Loom.chd did not work.
(Haven't got your game yet.)
Will try some more.

DTEAM

@Folly
Same problem on my side. It ask for a cartridge. I'll take a look on the notes in the pce_cd.cpp file

http://adb.arcadeitalia.net/download_file.php?tipo=sourcefile&codice=pcecd&emulator=964&oper=view

Folly

@DTEAM

Everything is down now so I can't get any files.
Will try to get bbros tomorrow so I can test with softlist loading and eventually without.