Creating ES gamelists with media for MAME/lr-mame/lr-mess

Folly

Summary by Folly
Summary by DTEAM
Summary by Jamrom2

Folly

Goal for this thread :

What we want here is to make some standardised gamelists, if possible, so :

the path structure is somewhat the same
support for both .zip and .7z
all/most, mame files, of that particular system, is added
and, if possible, videos and more info is added
to be shared with others via goodle-drive link and also to download in the add-systems-mamedev.sh frond-end
share/read into the retroscraper of @kiro so we are also able to scrape these games with retroscraper

Tips, tricks and downloads :

post 78 and post 174: Fixing speed recording issues, recording video with mame
post 09 : How to re-encode and/or cut parts from video using ffmpeg
post 15 : Example how to create videos from all roms or to re-encode multiple videos ( in one folder )
my gamelists mirror

DTEAM

@folly

Link to MAME ES gamelist.xml's who are not fully supported by sites like Screenscraper or others.

Copy and paste in the roms folder

MAME roms naming is required
See Development of module-script generator for lr-mess and mame standalone first post to install those systems

jamrom2

forth post reserved.

Will cover MAME/lr-mess based Coco1/2/3 and Ti99 emulation.

Folly

@dteam said in Development of module-script generator for lr-mess and mame standalone:

Yes, it's a good idea but we should do that for MAME roms (.zip and .7z) only. Otherwise there are too many different kinds of roms names for a single game.

@jamrom2 said in Development of module-script generator for lr-mess and mame standalone:

I think we can offer a basic list of gamelist.xmls based on what we tested with? Most of the TOSEC and other archived games are where people pull ROMs from. It's the other websites I used that might be a problem...and we can't point to those saying "use this site or our files".

I pasted both your replies here.
If we want to work together we should make a plan on how we can achieve both.

I have an idea on how we could achieve this.

We could try to make a database-structure that incorporates the sha1sums of the files that we find worthwhile to add in a gamelist.

Basically it comes down to the fact that different persons :

can have the same files in the same directory
can have the same files in different directories
can have the same files but with different filenames

To tackle this problem we can choose to read the files of these different persons by :

reading the sha1sums of the media/rom files
reading the path of the media/rom files

With this information we can match the database information with the files that a person has and generate a gamelist.xml on the fly.

What do you both think of this idea ?

Edit:
On second thought perhaps my above idea is a bit too much.

Perhaps I can extract the data of the mame hash tables for making gamelists for the regular solftlist files.

I will experiment with some things.

mitu

@folly said in Creating ES gamelists with media for MAME/lr-mess:

With this information we can match the database information with the files that a person has and generate a gamelist.xml on the fly.

Have you looked into ARRM ? It may already support this scenario (local art data, based on ROM checksum ?).

DTEAM

@mitu
Good point !

Folly

@mitu

Thanks, I will have a look.

Folly

@dteam

I am trying to make a gamelist with media for videopac.

Your post, creating a good mp4, was very useful :
https://retropie.org.uk/forum/topic/29987/retropie-convert-non-functional-videos-to-functional-videos-easy-free-batch-method-on-windows-also-size-reducer

I used this manual line in linux (for 1 file):

ffmpeg -i 2aswbat.avi -pix_fmt yuv420p -crf 30 -vcodec libx264 -acodec aac 2aswbat.mp4

Or skip the first 7 seconds :

ffmpeg -ss 00:00:07 -i 2aswbat.avi -pix_fmt yuv420p -crf 30 -vcodec libx264 -acodec aac 2aswbat.mp4

Or cut out from 7 seconds to 17 seconds :

ffmpeg -ss 00:00:07 -to 00:00:17 -i 2aswbat.avi -pix_fmt yuv420p -crf 30 -vcodec libx264 -acodec aac 2aswbat.mp4

Or cut out from 2.5 seconds to 22.5 seconds :

ffmpeg -ss 00:00:02.500 -to 00:00:22.500 -i 2aswbat.avi -pix_fmt yuv420p -crf 30 -vcodec libx264 -acodec aac 2aswbat.mp4

Result of the first game :

Folly

@dteam

Videopac is quite complete now.
I added the files here :
https://github.com/FollyMaddy/media

Can you place them on the google-drive ?

Folly

@jamrom2

Will cover MAME/lr-mess based Coco1/2/3 and Ti99 emulation.

Are you able to push these files to my repository, when you are ready ?

DTEAM

@folly
Nice job !! It's on Google Drive now !

Folly

@dteam

Thanks,
I just tried downloading it with the script.
It seems we have a problem.
When the google-drive script should download the gamelist.xml it gives an error (found in log) :
google drive ERROR: content-disposition not found

This also affects downloading all gamelists with media because it will stop when it encounters the gamelist for allin1.

I will try it again later, hopefully it will work again.
If it doesn't I will probably report the issue to Matt, so he can fix it.

I reported the issue :
https://github.com/matthuisman/gdrivedl/issues/20

EDIT : MATT has updated his script, It's working again ;-)

jamrom2

@folly Unfortunately, no. I'm sorry... when I put all of my gamelists together manually, I had no idea that we'd be offering them up for download. So the locations in mine are not anything close to what you have. Mine use all the original locations of .\emulationstation\gamelists\system xx and .\emulationstation\downloaded_images\systemxx.

I'd have to remake all of my files for each system I did, and of course I did it to the games I picked individually. I also do not use videos, and I custom named many of my games as I pulled them from different sources as well.

Not sure that would help anyone.

Folly

@jamrom2

I see the problem here for you.
Converting it all manually is a ton of work.
Though gamelists, in general, can be patched on the fly to get the locations we use.
So I understand if you don't want to do all this hard work.
Though, I ask you to reconsider.
In this forum we are trying to help each other and share information so we can do things in an easier way.

Earlier we choose our directory structure so :

we can backup the gamelists together with our roms
gamelists + media could be shared
original gamelists are not overwritten

The problem is that scrapers don't work perfectly.
What we are trying to do here is make gamelists that will match the mame driver data and/or the mame solftlist hash tables and add more if we want to.

So the work is done 1 time and then can be shared/downloaded within our script.
So when everyone is making a gamelist with media the workload is shared and it will be easier in the long run focusing later on improving stuff rather then building it up again and again.

To give you an idea of how I did it for videopac.
I have made a script that will extract data from a desired mame hash table and converts it into a csv.
Then I convert it back to a gamelist.xml.

Only thing I have to do is add info that wasn't in the hash table.
I used the internet and the scraper to get some cover images.
The movies can be made simply by running the mame command for every rom in the directory, one by one, and add the -aviwrite ./<filename> option, saving it it in the same folder as the rom, like this :

for game in *.zip;do [[ ! -f /home/pi/RetroPie/roms/videopac/media/emulationstation/$(basename $game .zip).mp4 ]] && /opt/retropie/emulators/mame/mame -cfg_directory /opt/retropie/configs/odyssey2/mame -rompath /home/pi/RetroPie/BIOS/mame\;/home/pi/RetroPie/roms/odyssey2 -aviwrite ./$(basename $game .zip).avi -v -c -ui_active videopac $(basename $game .zip);done

Then we can convert all the movies to mp4 and cut out the first 3 seconds like this :

for game in *.avi;do ffmpeg -ss 00:00:03 -i $game -pix_fmt yuv420p -crf 30 -vcodec libx264 -acodec aac $(basename $game .png).mp4;done

I hope you see the bigger picture here.

Folly

This post is deleted!

jamrom2

@folly I'm happy to post them here, maybe to be converted into whatever they can be used for. I took the time to research each game from several sites as to assure I had Meta data for each. I'd hate to see all of that time go to waste.

I'm definitely not saying no to offering what I have, it's just I gave up on the scrapers finding all of these titles, so when I did that, I used my own method that worked for my rig... unfortunately, it didn't help this situation.

Here is an example of the Ti99_4a game Bigfoot using the bigfoot.rpk rom.

<game>
	<path>./bigfoot.rpk</path>
	<name>Bigfoot</name>
	<desc>In Bigfoot, your ultimate goal is to scale a mountain and capture the eponymous creature, who stands at the summit. Along the way, you collect food and gold. However, in order to collect one bundle of gold (yellow piles), you must first collect one bundle of food (green boxes). Using the fire button, you throw ropes which are used for climbing up the platforms. Once thrown, you can scale up and down the ropes. Bigfoot hurls boulders at you as you go. If your rope is near the edge of a platform instead of near the middle, you run the risk of being smashed by a boulder as you climb up the rope, so throwing the rope near the center of a platform is safest. During later levels, birds will come along and snag the ropes - fatal if you happen to be on a snagged rope at the time.</desc>
	<image>~/.emulationstation/downloaded_images/ti99_4a/bigfoot-image.jpg</image>
	<releasedate>19830101T000000</releasedate>
	<developer>Milton Bradley</developer>
	<publisher>Texas Instruments</publisher>
	<genre>Action</genre>
</game>

I don't use videos as many of these older systems don't have any, so there is no listing for them...and I didn't want them in my build anyway. That would need to be added along with any other extra display pieces aside from what I have listed for "image".

Folly

@jamrom2

Do you have a github account ?

Then you can fork my "media repository" and add your files to that.
Just keep your gamelist.xml the same and add the ti99_4a directory to your fork just like my videopac directory and place the pictures in ti99_4a/media/emulationstation.

I can then look at the files and convert the gamelist.xml to the correct locations.

If you have more that 100 files then can't add them all at once, then you have to add them in 2 or 3 steps.

jamrom2

@folly I don't. I'm not as tech savy as that... lol.

I'm better off emailing you each set one at a time if you want to do that instead.

Folly

@jamrom2

That would be ok, but email has probably a limit in how big the attachment can be.
Probably about 50Mb.
So if you stay below that limit we are probably ok.

On the other hand I searched for a reasonably good website that we can use to share these files.
It looks like this website is a good alternative :
https://ufile.io/
Just zip your files to 1 file and share the link.
I did a test with videopac gamelis+media files :
https://ufile.io/tutqgnwc