Development of module-script generator for lr-mess, lr-mame and mame standalone

Folly

All changes up until now :

(18-10-2021)

all slotdevice parameters converted to "extra options"
added lr-mess overlay config creation for .7z files
(because roms are often in .7z format now)
improvements in menu : Download > Download/update

(14-10-2021)

fix : first line of the CSV was not displayed in some menus
now the front-end uses the mame0236 database
added the possibility to install the available gamelists and media per system
- should be future proof when adding new system on the google-drive
added the older and newer missing mame databases (now we have 0228 - 0236 )
add systems in the generator part script for : classich and gameandwatch
updated the handheld databases

https://github.com/FollyMaddy/RetroPie-Share/blob/0b3a8f2fa4588098411eda5b860cde8a9f00ac6c/00-workdir-00/add-mamedev-systems-2v4.sh
(right-click on raw to download)

Folly

@dteam said in Development of module-script generator for lr-mess and mame standalone:

NEW EDIT: here: https://www.progettosnaps.net/renameset/

you have a file category.ini (0.236) (always up to date) . Inside, you have a file named arcade.ini and one mess.ini. Everything is there to do the split.

I have found a way to match these files with our data.
And I think I can also match the files from http://adb.arcadeitalia.net/lista_mame.php .
We could even add our handhelds/plug&play to it this way.
After that I hope that I can implement it in the scripts so we can sort upon the "catagories".
The file will look something like this :

Driver 1292apvs (1292 Advanced Programmable Video System): *non-arcade
Driver 1392apvs (1392 Advanced Programmable Video System): *non-arcade
Driver 15lions (15 Lions (10166211, Queensland)): *arcade
Driver 18w (18 Wheeler (set 1)): *arcade
Driver 18w2 (18 Wheeler (set 2)): *arcade
Driver 18wheelr (18 Wheeler (deluxe, Rev A)): *arcade
Driver 18wheelro (18 Wheeler (deluxe)): *arcade
Driver 18wheelrt (18 Wheeler (deluxe, Rev T)): *arcade
Driver 18wheels (18 Wheeler (standard)): *arcade
Driver 18wheelu (18 Wheeler (upright)): *arcade
Driver 1942abl (1942 (Revision A, bootleg)): *arcade*bootlegs*cabinets

The disadvantage is that "matching" takes some time to do.
And I have to improve the category names.
For instance, I want to convert "mess" into "non-arcade".
So, for now, it's a slow development.

Using grep we can sort on arcade , for example :

DTEAM

@folly said in Development of module-script generator for lr-mess and mame standalone:

I have found a way to match these files with our data.

This is really cool!!

Folly

@dteam

I am finished with the matching.
After converting the files, so it would contain only the systems, I matched the files against our database.
It takes about 3 hours to build.

I uploaded the file to github :
https://github.com/FollyMaddy/RetroPie-Share/blob/main/00-databases-00/mame/mame0236_systems_sorted_info

For now something to check and experiment with.

The categories that I added are :
all_in1
arcade
ball-and-paddle
board-game
bootlegs
cabinets
calculator
casino
classich
climbing
clones-arcade
computer-graphic-workstation
computer
digital-simulator
driving
fighter
freeplay
game-console
gameandwatch
handheld
jakks
konamih
maze
mechanical-arcade
medal-game
medical-equipment
misc
monochrome
multi-game
non-arcade
screenless
tigerh
tigerrz

DTEAM

@folly

WOW !!! A lot of work ! Some of those are not playable. Will you keep them (like mechanical-arcade, medical-equipment, ball-and-paddle, board-game, etc.) ?

Folly

@dteam

I like the idea that we now can see what which system is and that the most data is included.

I think it's not a problem that non-working systems are in there because we should be able to sort on it, if I add it from the ini files.
If I am wrong, I can always remove some entry's.

I did not yet add the info of these ini files :
arcade_BIOS
arcade_NOBIOS
CHD (no BIOS)
CHD Working
Non Bootlegs
Non Mechanical Arcade
Not Working Arcade
Originals Arcade
Players
resolution
Working Arcade Clean
Working Arcade

This is the only bash code I used to add the information to the existing data file,
using the simple .ini txt files containing the systems per category that are with or without windows line-endings :

for file in $(ls -r *.ini);do echo busy with $file;cat $file |sed 's/\r//g'|while read line;do grep -i "Driver $line " mame0236_systems_sorted_info|sed -i "s/Driver $line .*\:/&*$(basename $file .ini)/" mame0236_systems_sorted_info ;done;done
sed -i 's/)\:/)\: /g' mame0236_systems_sorted_info

Folly

@dteam

I have made the first working code to generate the sorted lists.
Perhaps a bit premature as is not yet how I want it to be.

It's just a peek in the future on how it will work.
I must say I am very satisfied with the first tests ;-)

(edit : third update)
https://github.com/FollyMaddy/RetroPie-Share/blob/94f47159441265d13869eeb72fb8640b6aadb9ee/00-workdir-00/add-mamedev-systems-2v4.sh

menu improved
overall improved sorting on --dictionary-order
(fast)create sorted lists (and equal to) on patterns up to 5 options ($2 -$6)
(slower)create sorting (not equal to) on patterns for one option ($2) adding an ! after the pattern
(in combination with 4 (and equal to) patterns up to 4 options ($3 -$6) is possible)
possible to make sorted lists with "system" names
( with "system description" names it was already possible )

DTEAM

@folly
l’ll try it monday👍🏻

DTEAM

@Folly

Your last add-mamedev-systems-2v4.sh works very well. Great job! I like your subfolders for gamelist.xml. Some options can be confusing for newbies. Maybe you could add a logigram or a tree diagram in your first post.

I like this version, it's easy and full access to MESS and MAME with Artworks, cheat codes and "games scraping" for unusual systems... it's huge.

Folly

@dteam

Thanks for the feedback.

Indeed I have to work on it, structuring it better.
This is basically caused by the fact that I am working on it.
First I want to make sure things are working correctly.
When I am done adding/testing stuff I will try to reorganize.
Perhaps we can discuss this later on, to make it simpler.
Or indeed add a diagram in the first post.

Perhaps you have noticed, the forum lists are now in two sections. (old and new)
The list can now be build up from the new database we created.
Basically this means I can remove it from the old section, where it used it's own data.

Well, I have noticed now, working on it, that there is still much to be added and things that have to be changed.

Btw.
You can experiment with making some sorting lines, if you want.
That way we can build the new lists that are important to be in there.
You can add it between lines 86 and 107, and share them if you like.

Folly

@dteam

I did a huge update.
You can test some new stuff ;-)

https://github.com/FollyMaddy/RetroPie-Share/blob/432d2c9db4b1497ba2da938cef1968621c5c2570/00-workdir-00/add-mamedev-systems-2v4.sh

Front-end 2v4 improved

cleaned up a bit
(removed tho old forum selection list, now there is a new one)
(removed some functions and added some functions)
re-ordered the menu
(hopefully, in time, it will evolve into a better and simpler structure)
add experimental help lines
(perhaps not a definitive solution !)
add saving and removing database file locally (for offline usage)
(file is save in or deleted from : /opt/retropie/emulators/mame)
add more sorted lists (remove test lines)
add SEARCH form, to create a sorted list on the fly
(still an issue in the dialog form : "cancel" button doesn't work)
add download option with form
(only available with the correct input !)
(still an issue in the dialog form : "cancel" button doen't work)

DTEAM

@folly
When I updated Retropie, something happened with my Retropie Setup. I can't run Retropie Setup anymore. I suspect the Valerino script. Any idea. Maybe it's helpers.sh.

Folly

@dteam

Don't know.

You can try and rename the directory and git clone the repository again.
That way you have the whole directory fresh and a backup of the old directory.

Then try if it works ! (edit)

Later you can copy the module-scripts, you had, back in.

Is it a solution ?

DTEAM

@folly
I already copy fresh files from here

I'll try to do some cherry picking from here

Folly

@dteam

I just did my trick here, using :

Renaming the old RetroPie-Setup to RetroPie-Setup.old

cloning :

cd 
git clone --depth=1 https://github.com/RetroPie/RetroPie-Setup.git

running (4.7.19) :

cd RetroPie-Setup
sudo ./retropie_setup.sh

Looks fine here, so I suspect it's something else.

If you have it running I recommend you add our -ext script from now on in :
/home/pi/RetroPie-Setup.ok/ext/RetroPie-Share/scriptmodules/supplementary/add-mamedev-systems-2v4-ext.sh

This places all generated module-scripts in :
/home/pi/RetroPie-Setup.ok/ext/RetroPie-Share/scriptmodules/libretrocores

And in the RetroPie-Setup they will be arranged on external repository name .

DTEAM

@folly said in Development of module-script generator for lr-mess and mame standalone:

cd RetroPie-Setup

It's do this:

pi@retropie:~ $ cd RetroPie-Setup
pi@retropie:~/RetroPie-Setup $ sudo ./retropie_setup.sh
sudo: ./retropie_setup.sh: command not found

or this

pi@retropie:~/RetroPie-Setup $ sudo /home/pi/RetroPie-Setup/retropie_packages.sh
/home/pi/RetroPie-Setup/scriptmodules/packages.sh: line 903: source: /home/pi/RetroPie-Setup/scriptmodules/supplementary/add-mamedev-systems-2v4.sh: cannot execute binary file
Module  is missing valid rp_module_id
Module  is missing valid rp_module_desc

Folly

@dteam

remove our script
add-mamedev-systems-2v4.sh

and try to run the setup again without it.
You should use this :

sudo ./retropie_setup.sh

Could you check the permission of the files again for me ? :
I have :

pi@raspberrypi:~/RetroPie-Setup $ ls -l
total 76
-rw-r--r-- 1 pi pi   146 Oct 28 21:16 AUTHORS
-rw-r--r-- 1 pi pi   417 Oct 28 21:16 CONTRIBUTING.md
-rw-r--r-- 1 pi pi   635 Oct 28 21:16 COPYRIGHT.md
-rw-r--r-- 1 pi pi 32265 Oct 28 21:16 LICENSE.md
-rw-r--r-- 1 pi pi  2033 Oct 28 21:16 README.md
drwxr-xr-x 2 pi pi  4096 Oct 28 21:16 ext
drwxr-xr-x 2 pi pi  4096 Oct 28 21:53 logs
-rw-r--r-- 1 pi pi  6374 Oct 28 21:16 platforms.cfg
-rwxr-xr-x 1 pi pi  2314 Oct 28 21:16 retropie_packages.sh
-rwxr-xr-x 1 pi pi   520 Oct 28 21:16 retropie_setup.sh
drwxr-xr-x 7 pi pi  4096 Oct 28 21:16 scriptmodules

Folly

@dteam

Do you have these permissions for the file ? :

ls -l
-rw-r--r-- 1 pi pi   520 Oct 28 21:16 retropie_setup.sh

Then it's not executable, and you will get :

pi@raspberrypi:~/RetroPie-Setup $ sudo ./retropie_setup.sh 
sudo: ./retropie_setup.sh: command not found

DTEAM

@folly
The problems seems to be your script. I removed it and now it's fine.

Folly

@dteam

That sounds bad.
I don't have any problems over here.
I will pull it from github to see if I get any problems.

I assume it was the latest.

edit:
I installed everything fresh on my x86 laptop too.
I found some compatibility problems for the sorted lists, so I will look at that.
But no problems in crashing or recognising whatsoever.

Did you try to edit the script and perhaps made a mistake in doing so ?
Can you open the script into a text editor to check if it's a good file ?