Development of module-script generator for lr-mess, lr-mame and mame standalone

DTEAM

@folly said in Development of module-script generator for lr-mess and mame standalone:

And It would be nice if all files can be downloaded from the same source.

Cheat codes for MAME/MESS : Pugsy's Cheats , you must get cheat.7z and put It in /home/pi/RetroPie/BIOS/mame/cheat

MAME Artworks: (Handheld systems without arcade artwork), could be moved somewhere else on another server.

some observations (with your 3 days ago version):
For MAME systems (TigerH, KonamiH, Jakks, etc., ) - MAME Romset

all options working well lr-mess-cmd, etc. (except -- > lr-mess-basename - It freezes)

For systems like coleco, nes or others,

Lr-Mess-system and mame-system always working with No-intro, Tosec, MESS, etc. romsets
lr-mess-cmd works with mess romset
lr-mess-basename, mame-basename and mame-cmd never work

For systems using a nomenclature in RetroPie different from that of MAME (e.g .: jaguar = atarijaguar)

only lr-mess-system and mame-system working with No-intro, Tosec, MESS, etc. romsets

Have you the same results?

Folly

@dteam

Thanks, I will read the information and see what we can do with it.

I think I have similar results, but still I have to do some tests.

Summarized :

It seem I can remove the old :

lr-mess-basename,
mame-basename

For the handhelds this should be kept :

mame-cmd

Folly

@dteam

I have made some progress on the downloading of cheats and artwork.
But google drive is a pain !
For google drive I found a nice script, but it doesn't get all the files.
So I made an issue on Github.
Hopefully the maker of the google drive script will do something with it.
To be continued...

DTEAM

@folly said in Development of module-script generator for lr-mess and mame standalone:

But google drive is a pain !

I put It there because It was easy to do for me for storage, but If you prefer, you can download those files and transfer It somewhere else where It will be easier to manage It.

Folly

@dteam

I will first wait on the response.

If you have some extra capacity an other solution could be to add some separate handheld folders for this purpose.

We can discuss this later.

Folly

@dteam

So you see, if this works, it would be great.
Just using a few lines ;-)

If you are interested in the code I already made.
This is it :
(OK)

#gets the cheat.7z and places it in the correct path
#works perfect
wget -N -P /tmp http://cheat.retrogames.com/download/cheat0221.zip
unzip -o /tmp/cheat0221.zip cheat.7z -d $HOME/RetroPie/BIOS/mame/cheat
rm /tmp/cheat0221.zip

EDIT ~~(look a few posts ahead for better code for artwork !)~~ :
gdrivedl got only a part of the html code(old version).
If you would browse the website, you will see that when you scroll-down, more info is added to the page.
I think that is the reason why it isn't working.
So I created an issue on github which is solved now.

#get all artwork files and put it in the correct path
#first test got about 42 of 162 artwork files :-( 
#second test with improved version of gdrivedl.py gets all files :-)
wget -nv -O /tmp/gdrivedl.py https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py
python /tmp/gdrivedl.py https://drive.google.com/drive/folders/1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3 -P $HOME/RetroPie/roms/mame/artwork
rm /tmp/gdrivedl.py

( issue is posted here and gdrivedl.py is improved, accordingly : https://github.com/matthuisman/gdrivedl/issues/12 )

DTEAM

@Folly
Great, I’ll try those and give you news.
For the artworks on google drive, if you prefer, I can split It per systems in folders

Folly

@dteam

I have a solution already ;-) (too bad, not the solution after all ! :-( )

This is the code to get all 162 files the artwork files which are then placed in the correct directory ( still testing but seems to be ok / damn stuck at 114 files ! ) :

wget -nv -O /tmp/gdrivedl.py https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py
curl https://drive.google.com/embeddedfolderview?id=1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3#list | egrep -o '(\w|-){26,}' | grep entry | sed 's/entry-//g' | while read fileid ; do python /tmp/gdrivedl.py https://drive.google.com/file/d/$fileid -P $HOME/RetroPie/roms/mame/artwork ; done
rm /tmp/gdrivedl.py

This code should be working but google drive seems to block after a while.
(see https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-and-mame-standalone/41 )

This is how it works :

with https://drive.google.com/embeddedfolderview?id=1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3#list you can list all the files in one page.
with egrep -o '(\w|-){26,}' | grep entry | sed 's/entry-//g' you filter out only the file-id's.
then it uses these file-id's every time in the loop, creating a command one by one for each file.

DTEAM

@folly
Some files are for arcade folder, is It for that you stuck at 114?

Folly

@dteam

There is another reason why this is not working.

Haha, I think I tested too much, keep getting => 403 : Forbidden.

Read the output of one file that I was trying.

Sorry...Sorry... We're sorry... ... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

Can you get all the files with the code ?

DTEAM

@folly

#gets the cheat.7z and places it in the correct path
#works perfect
wget -N -P /tmp http://cheat.retrogames.com/download/cheat0221.zip
unzip -o /tmp/cheat0221.zip cheat.7z -d $HOME/RetroPie/BIOS/mame/cheat
rm /tmp/cheat0221.zip

Like you said, It works perfect. It's great

wget -nv -O /tmp/gdrivedl.py https://raw.githubusercontent.com/matthuisman/gdrivedl/master/gdrivedl.py

curl https://drive.google.com/embeddedfolderview?id=1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3#list | egrep -o '(\w|-){26,}' | grep entry | sed 's/entry-//g' | while read fileid ; do python /tmp/gdrivedl.py https://drive.google.com/file/d/$fileid -P $HOME/RetroPie/roms/mame/artwork ; done

rm /tmp/gdrivedl.pyrm /tmp/gdrivedl.py

I got only 38 files ( 271 Mb). I'm gonna try another run (same thing)
Example of Errors I had:

Folly

@dteam

Thanks for trying.

Basically it also says : HTTP Error 403: Forbidden.

It looks like google detects it and then will blocks you for a time.
;-(

I will search again for a solution on the internet.

DTEAM

@folly
If I zip all those files in one file to download, and after that, you manage It on the Pi. Could It be a solution?

Folly

@dteam said in Development of module-script generator for lr-mess and mame standalone:

@folly
If I zip all those files in one file to download, and after that, you manage It on the Pi. Could It be a solution?

If everything else fails, then we can try that ^^, but large files have other problems too solve..

Whoow, I just had a full test run and all files are downloaded :-)
I Basically used the sample script on this page just using 2 curl lines :
https://gist.github.com/tanaikech/f0f2d122e05bf5f971611258c22c110f
I injected the filenames and the file-id's from a handmade comma seperate file (.csv) into this sample script.

I have to rewrite this a bit.
Now I am hopefull !!!

Folly

@dteam

Done rewriting.
Can you test this for me ? :

fileids=(); filenames=()
weblist=$(curl https://drive.google.com/embeddedfolderview?id=1sm6gdOcaaQaNUtQ9tZ5Q5WQ6m1OD2QY3#list)
IFS=$'\n' GLOBIGNORE='*' command eval  'fileids=($(echo $weblist | egrep -o "(\w|-){26,}" | grep entry | sed "s/entry-//g"))'
IFS=$'\n' GLOBIGNORE='*' command eval  'filenames=($(echo $weblist | tr ">" "\n" | tr "<" "\n" | grep zip | grep -v application | grep -v : ))'
mkdir -p $HOME/RetroPie/roms/mame/artwork
cd $HOME/RetroPie/roms/mame/artwork
for index in "${!filenames[@]}"; do 
echo downloading : ${filenames[$index]}
curl -c ./.cookie -s -L "https://drive.google.com/uc?export=download&id=${fileids[$index]}" > /dev/null
curl -Lb ./.cookie "https://drive.google.com/uc?export=download&confirm=`awk '/download/ {print $NF}' ./.cookie`&id=${fileids[$index]}" -o ${filenames[$index]}
done
cd $HOME

How does it work :

create empty array's (see it as an excel column with more cell's with data)
get all data of the website and put it in the string : weblist
( using embedded folder view : http://stackoverflow.com/questions/20681974/ddg#20681975 )
IFS is used to get all data it reads, when encountering a newline (\n), and puts it in the array
sed, tr and grep commands filter the website so IFS will read the data
all data, gathered in the array's, is injected in the curl commands by the for loop

DTEAM

@folly
It running right now. It seems to be faster than the other script to get the files.
new edit
162 zip files + 1 cookie file ... It works !!

Folly

@dteam

Yeahhh.. That is great !

I am now using a normal curl example and no scripts from others here,
perhaps faster that way.

Maybe I have to improve it a bit :

remove cookie afterwards or place in /tmp directory.
try to improve writing directly in the artwork directory, without going to the directory

But for now, it will do.

Are all files coming in the correct place or do some files have to be moved ?

DTEAM

@folly said in Development of module-script generator for lr-mess and mame standalone:

Are all files coming in the correct place or do some files have to be moved ?

Mame artwork go in /home/pi/RetroPie/roms/mame/artwork

.cfg files for overlays are generated in propers folders

like: /home/pi/RetroPie/overlays/konamih/kgradius.cfg

kgradius.cfg

overlays = 1
overlay0_overlay = kgradius.png
overlay0_full_screen = false
overlay0_descs = 0

kgradius.png is not automatically generated

In the MAME artworks kgradius.zip we found
4 .png files and only one is the backgroung.

Do you want to add the entire handheld overlay or just get the background? I,m saying that because I saw this picture for MAME in your previous post .

Personally, with the background only It's fine for me. I made some of the default.lay files that we find on google drive (svmu, pockstat, gp32, mc_tv200, sy889, etc.) and I don't know how to turn a default.lay file into a .cfg for retroarch overlay with position, size, etc.

gradius

Folly

@dteam said in Development of module-script generator for lr-mess and mame standalone:

kgradius.png is not automatically generated

The code you used here is only for downloading the artwork files :
https://retropie.org.uk/forum/topic/29682/development-of-module-script-generator-for-lr-mess-and-mame-standalone/46

I still have to program the extraction of the .png files , but it seems it is doable / quite easy. ;-)
EDIT : I just added extraction of Background.png's (classich, gameandwatch, konamih, tigerh) in : generate-overlay-configs.sh / get-cheats-artwork-overlays.sh
I cloned it also to get-cheats-artwork-overlays.sh so we can add the download stuff later into that script.

For some artwork files there are no "Background.png" files. So we will have to find a solution for that.

Do you want to add the entire handheld overlay or just get the background? I,m saying that because I saw this picture for MAME in your previous post .
Personally, with the background only It's fine for me.

I only want to do the background.

DTEAM

@folly

For cheat codes with MAME, I think you will have to add

cheatpath /home/pi/RetroPie/roms/mame/cheat
cheat 1

in mame.ini.