Development of module-script generator for lr-mess, lr-mame and mame standalone

checking and installing is done when the retroscraper line is selected (so no check, in the beginning anymore, when starting the mamedev.sh modulescript)
except for pip, all others python modules are checked and installed with fixed versions (which makes it faster and easier)
speed is improved by sending "pip list" to a variable which is used in the for loop
using the variable increases the speed about 10 seconds on a pi4 (from about 18 seconds to about 8 seconds)
update all retroscraper python module dependencies to the latest versions

253.04 :

add proper detection for system astrocde (Bally Professional Arcade)

Folly

@DTEAM

Hi,
I decided to wait with updating the script to 0254.
This because it takes quite a lot of time and I also noticed that the binaries of lr-mame and lr-mess are still on 0.251.
I think updating is better when also lr-mame and lr-mess more in sync or at least more up to date.
For now I want to use the time to improve some stuff if I can.

Nevertheless if someone already runs standalone mame 0.254 the Galaxian (Bandai) and Bandai Gunfighter can be added in the proper folder and can be played.

Discussions of moving lr-mame and lr-mess to 0254 :

Folly

EDIT :
The scripts can be found here.

first issue has been fixed in this commit
change test versions to 253.07
- test versions can now update itself using 00-workdir-00
- moved TEST scripts to 00-workdir-00/supplementary/
- fix github download function for ubuntu 22.04 using a different string variable which fixes downloading retroarch joypad autoconfigs
- Mark them as TEST versions
change test versions to 253.08
- add sha1sum of updated runcommand.sh which improves adding %SOFTLIST% variable in runcommand
for more improvements see here

@DTEAM

I have been busy for 3 days now and I am about ready to share a major update.
I made quite some changes in the runcommand loaders, adding new replace tokens to improve many things.
The standard %ROM$ and %BASENAME% are such replace tokens.

I added :

%ROMDIR%
The "%ROMDIR% token" that can be used to add the rompath in a runcommand.
When using a runcommand using %BASENAME$, the rom can always be found, even in a sub-directory. This also makes it possible to categorise roms in different folders within a roms directory.
%DQUOTE% :
Though the "%ROMDIR% token" does not have double quoting.
Why? Because lr-mame and lr-mess can't cope with it.
For this reason the "%DQUOTE% token" is added.
The "%DQUOTE% token" can be placed in other places to get the runcommand working especially when using subdirectories that have spaces in them.
For MAME standalone this will work :
-rompath /home/pi/RetroPie/BIOS/mame;%DQUOTE%%ROMDIR%%DQUOTE%
For lr-mame and lr-mess this will work :
-rompath %DQUOTE%/home/pi/RetroPie/BIOS/mame;%ROMDIR%/%DQUOTE%
All other attempts to add double quotes to runcommands in a different way failed so using the "%DQUOTE% token" seems to only way to go.
%CLEANBASENAME% :
The "%CLEANBASENAME% token" is needed for creating a proper per game config directory so mame system configs can be created per game.
For MAME standalone this would not be necessary but lr-mame and lr-mess can't cope with spaces and double quotes in the wrong places.
Therefor the "%CLEANBASENAME% token" can be used which removes all spaces and special characters and is without any double quotes.
The "%CLEANBASENAME% token" could also be used to make a clean savestate file.
%SOFTLIST% :
The "%SOFTLIST% token" is needed to force proper softlist loading.
For example pacmania is in msx1_cass and in msx2_cart.
If one loads a game with a basename loader, for example pacmania, which comes from msx2_cart, then normally the the soflist msx1_cass has priority over msx2_cart.
So mame thinks it's pacmania from msx1_cass but it is actually pacmania from msx2_cart.
To fix this issue we need to force the correct softlist before the "%BASENAME% token".
The %SOFTLIST% token is the last folder name where the rom is in plus an extra => :.
For pacmania we will have to place it in the folder "msx2_cart".
In the loader you will see it as %SOFTLIST%%BASENAME%, eventually for pacmania it will look like this "msx2_cart:pacmania".
If the rom is in a rompath without => _ then the "%SOFTLIST% token" will be empty and mame will guess the correct softlist.
If "%SOFTLIST% token" gets a value for some reason, because a _ char is in the path, then mame will refuse to load the game.
A check in the runcommand.log will reveal the issue.
( although it's an improvement, for example, msx1_bee_card softlist will not benefit from it, as it needs the beepack inserted as a slot. So softlists that need slot options will still not work with the regular basename loaders )

To be able to use these tokens they have to be in the runcommand.sh script.
My new scripts are able to add that runcommand.sh.

My both scripts start now with a clean mamedev_csv array now.
Now both scripts work fine when switching from one to the other.

I also made a minor change to the run_mess.sh script to accommodate spaces in the rompath.

Perhaps I forgot something will look at that later.

I would like to discuss the new stuff before I commit because it's quite a change and testing needs to be done.

What do you think about the changes and the testing ?

Folly

@Ashpool

You are welcome to join in and test the new scripts.

Ashpool

@Folly Well, I am just creating a fresh ubuntu vm to get a "fresh" base... on the raspi install I tried it on (manual retropie install over raspberry legacy os), 1st thing i noticed/came into trouble was the Game and Watch Setup. The Script (still) lists mame/lr-mess/lr-gw, and this post of yours sounded that mame/lr-mess are still usable, but the es_systems.cfg just lists .mgw|.MGW as extensions and only lr.gw is available as an emulator from within the runcommand menu.
But that OS "install" has become a mess in recent time and i am just waiting for an excuse to dump it in favour of bullseye 64bit (as that install wasn't really meant to incorporate retropie at 1st, retropie not supporting that ain't really a drawback here), so it may be a "just on my system"-case and not a fault from your script.

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

@Folly Well, I am just creating a fresh ubuntu vm to get a "fresh" base...

Good idea to make a fresh vm.
Do you use VWware or something else ?
I would suggest pure debian (I am using mate desktop) instead of ubuntu.
Although ubuntu looks very nice I had many issues with ubuntu in the past.
Only thing with debian is that you need to add your user name to the /etc/sudoers file to be able to use the command sudo.
If you are ready then I would also suggest to put my scripts in the ext directory :
RetroPie-Setup/ext/RetroPie-Share/scriptmodules/supplementary
That way the 3rd-party module-scripts are nicely separated.
I prefer using the mamedev.sh script now over the add-system-mamedev.sh script because :

it's faster
it will not generate module-scripts but will install systems or categories directly

Only disadvantage is removing a system or category but that can always be done manually if really needed.
Though, testing needs to be done on both versions as I still update both.
Remember : the test versions (253.06) can't be updated right now because they are still in the 00-workdir-00. If update is selected it will downgrade to 253.05.

on the raspi install I tried it on (manual retropie install over raspberry legacy os), 1st thing i noticed/came into trouble was the Game and Watch Setup. The Script (still) lists mame/lr-mess/lr-gw, and this post of yours sounded that mame/lr-mess are still usable, but the es_systems.cfg just lists .mgw|.MGW as extensions and only lr.gw is available as an emulator from within the runcommand menu.

Looks like you missed the update.
The thing you describe was the issue.
It should be solved or you have installed lr-gw the regular way again ( not from my script ), that could cause this issue also.
I checked my code and it should be ok, though the help message can be updated a bit.
I like to know if you still experience it when you do it on the fresh vm.
@DTEAM and I also implemented predefined way of loading the madrigals roms so switching between emulators is not needed between mame/lr-mess or lr-gw when putting all mame and madrigals in one folder.
(this is already in the help btw.)

But that OS "install" has become a mess in recent time and i am just waiting for an excuse to dump it in flavour of bullseye 64bit (as that install wasn't really meant to incorporate retropie at 1st, retropie not supporting that ain't really a drawback here), so it may be a "just on my system"-case and not a fault from your script.

I have been using it on Debian bullseye 64 bit for quite a while now. I have VM's on both x86_64 and on aarch64(arm).
Although earlier on I had to patch some stuff seems not really needed anymore using it on 64bit.
It should work fine when the basics are good.

Let me know how it goes with the fresh vm.

DTEAM

@Ashpool

@Folly said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

@DTEAM and I also implemented predefined way of loading the madrigals roms so switching between emulators is not needed between mame/lr-mess or lr-gw when putting all mame and madrigals in one folder.
(this is already in the help btw.)

See this post : https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-lr-mame-and-mame-standalone/1338?_=1684160654975

as @Folly said, the script takes care of it now.

Ashpool

@Folly

Remember : the test versions (253.06) can't be updated right now because they are still in the 00-workdir-00. If update is selected it will downgrade to 253.05.

That was the one I've used, update option not choosen and no previous (AFAIR)/later lr-gw install.

Do you use VWware or something else ?

I've tried virtualbox in pre-Win7 times, but since vmware(-player) offered native support for xpmode vms, i've switched to that player and well, stayed with it so far (though for legacy systems as win9x, i would prefer QEMU/PCem/DOSBox-x)) .

Still, I guess that I have to raise the memory used by the VM from 4 GB to 8+, the mame compile (initiated via mamedev.sh) failed/retropie crashed back to the desktop... in the meantime i also tried add-mamedev*.sh on a fresh &updated retropie image (the basic retropie install one i've created for my backdrop experiments) and at least there g&w installed fine.
Will play with that setup in the next few days and report back (I've never really tried software lists/mess systems and maybe elite on the archimedes could be a self-set milestone I want to accomplish ;] )

Edit: At least the ArchElite (ever wanted to play that since I read the revue in a Computer magazine back in its time, and though Ian Bell himself is hosting various elite versions [1] to be used on emulators/real machines since aeons - i never came to try it) goal may be troublesome, i thought of giving it a try 'cause of nonmame mentioning the acorn/archimedes systems as mature in mame, but querying the systems from the xml/arcadeitalia-database ain't that promising. We may see ( honestly, previously i only ever used your scripts for g&w out of childhood memories ;] ).

1: And also "Elite - the musical", I really love "a contract is a contract" from that ;D

Edit: -> P.S.: OK, maybe those memories/anectodes would be better placed/suited within the Old Computer Appreciation Thread. ...

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

Still, I guess that I have to raise the memory used by the VM from 4 GB to 8+, the mame compile (initiated via mamedev.sh) failed/retropie crashed back to the desktop...

Indeed, compiling mame takes a lot of memory.
I use 8 GB, that should work.
You have to know that the install-lines for mame/lr-mame and lr-mess are just links/redirections to the original module-scripts.
They are there so everything is in reach and that someone doesn't forget to install it.
Basically it doesn't matter if you install them from the experimental packages or from my module-script.
Is this information I should add in the help for these lines ?

in the meantime i also tried add-mamedev*.sh on a fresh &updated retropie image (the basic retropie install one i've created for my backdrop experiments) and at least there g&w installed fine.

Aha, good to hear !

Will play with that setup in the next few days and report back (I've never really tried software lists/mess systems and maybe elite on the archimedes could be a self-set milestone I want to accomplish ;] )

Sorry some bad news ;-)
I just tested elite and it will show the files when opening 0: but it will not boot when selected.
In the hash table archimendes.xml you can find elite as not supported.
So you probably can't get it working on archimedes with mame or lr-mess.

Ashpool

@Folly said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

I use 8 GB, that should work.
You have to know that the install-lines for mame/lr-mame and lr-mess are just links/redirections to the original module-scripts.

Just to report back on the VM... (raspi/binary installs and tests still going fine): to be sure I've used 12GB and mame compiled without a flaw, now it is lr-mess wich breaks (no crash, just fail):

Compressing src/mame/layout/zoo.lay...
make: *** [makefile:1333: build/projects/retro/mamemess/gmake-linux/Makefile] Segmentation fault (core dumped)

No Fun, and not a fault of the script (as you mentioned it just calls the native retropie script)... bloody thing is, i installed all that in a previous VM (including games&watch via add-mamedev-systems.sh) earlier this year and it all went without a problem (also on Ubu 22.04) - sigh! But I hadn't encountered this "error" back then, now I have :[

Well, to cite a somewhat famous movie series "I am getting to old for this s*..", otherwise said "It may be that we are indeed living in interesting times" ;]

P.S.: Damned and curses to hear about "<software name="elite" supported="no">", wasn't there a way to disable softlist s and load images via CMD (Just to see what will happen)?

Edit: Wasn't there an option to convert(/copy?) mame art to be used by lr-mess? I can't find that (at the moment) in the (current) tool. If mame is selected as emulator, the downloaded artwork works fine, but it ain't present in lr-mess)

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

Just to report back on the VM... (raspi/binary installs and tests still going fine): to be sure I've used 12GB and mame compiled without a flaw, now it is lr-mess which breaks (no crash, just fail):

Not that much time but if I can I will try to compile lr-mess and see what happens.

Well, to cite a somewhat famous movie series "I am getting to old for this s*..", otherwise said "It may be that we are indeed living in interesting times" ;]

Indeed, nice quote ;-)

P.S.: Damned and curses to hear about "<software name="elite" supported="no">", wasn't there a way to disable softlist s and load images via CMD (Just to see what will happen)?

It will load with softlist enabled, you will get into the OS of Archimedes and the files can be seen when opening the drive in the explorer. But the game will not run it will freeze. You can test with softlist disabled, that means just using the runcommand with the media option -flop (this will disable the softlist loading in the specific custom-core-options.cfg and is loaded via the retroarch.cfg.add).
Not tested yet, but this will likely have the same result.
Sure, you can use a .cmd file, if that is what you mean, but you will have to create one, and it will have the same result as mentioned earlier.
(if having a good working .cmd file, that is)

Edit: Wasn't there an option to convert(/copy?) mame art to be used by lr-mess? I can't find that (at the moment) in the (current) tool. If mame is selected as emulator, the downloaded artwork works fine, but it ain't present in lr-mess)

You have to download the artwork first and then you will be able to convert the artwork into Retroarch backgrounds/overlays.
First select :

Download/update mame artwork (+/-30 min.)
Then select :
Create RetroArch background-overlays from artwork
Create RetroArch 16:9 bezel-overlays from artwork

Folly

@Ashpool said in Development of module-script generator for lr-mess, lr-mame and mame standalone:

now it is lr-mess wich breaks (no crash, just fail):

Indeed it does not compile.
Tested Debian 10 (x86), Debian 11 and Ubuntu 22.04 on x86_64 VM.
Seemed an issue with the Makefile.libretro that has been changed 5 days ago.
Though I tried the older Makefile.libretro and sadly still fails.

What a nightmare again ;-(
Will have a look later.

I will open a separate thread for this :
https://retropie.org.uk/forum/topic/34238/lr-mess-will-not-compile-17-may-2023

Edit :
lr-mame has the same issue !