Scrape MAME Metadata?

hansolo77

So I have my system now running MAME, and it's great. The included dat's provide a perfect game listing, without all the crap showing up. I tried to scrape through the Selph Scraper, but no matter what options I choose (scraper source / List type (no-intro, filename, etc)) all I get is a screen full of errors about "hash not found". I spent (according to the tool) 7.8 hours last night scraping through the Windows UXS Scraper. It came back with a lot of good artwork and details. But it changed the game list. Rather than 1 listing, for a game, I now have 5-6 for each one. I assume this is because the scraper extracted the .zip files and identified the files inside as separate ROMs. So now my 4k or so games are now up to around 20k, and it's a mess. Luckily, erasing the correct gamelist.xml simply reset it back to the original (non-scraped) list. I still have the artwork though. All I need is the metadata details that tells you about the game. Is there a better way to scrape for all this specifically for MAME?

hansolo77

Wow, either nobody has ANY clue, or this post just got lost in all the others. So (shameful to say)..

Bump!

herb_fargus

@hansolo77 I am actually trying to find the same solution for myself right now, what scraper did you use to get the images? Also what type of Romsets are you running, split, merged, or non merged?

Shakz

@hansolo77 Just curious did you have emulation station running when you ran it? THe newest version kills it for you.

hansolo77

When I run it manually from a terminal, Selph's scraper (using the -mame flag) will scrape but it looks like it skips over anything that isn't Neo-Geo. The UXS scraper grabs all the images, details, etc but creates a non-merged gamelist.xml. Maybe there is a setting I'm overlooking?

I assume I'm using a merged set. I have 1 zip per game, with multiple roms inside. I don't know if they're parent/clone though. I'd post a link to the torrent location via PM if you REALLY gotta know where I got it to check for yourself. The site is kinda -secret- lol.

@Shakz - Both Selph and UXS will only work with Emulation Station closed. Selph's I ran via the terminal (not SSH, actually in the terminal screen after closing ES). UXS prompts if you want to close ES before trying to scrape, which I said yes (even though I already had it closed).

I haven't tried the built in ES scraper. Maybe I'll try that tonight while I'm sleeping. Turn off the "user input" setting first of course lol. It's easy enough to let it run and delete the gamelist.xml and images folder to start over if it doesn't work.

Shakz

Well friend with mine I didnt see any issues. I chat with @Screech frequently and if its doing something crazy with your romset he will want to know.
Can you create a fresh folder for the old version and give it a go?
https://github.com/Universal-Rom-Tools/Universal-XML-Scraper/releases/download/2.0.0.5/Universal_XML_Scraper64.exe
also if you would not mind providing a http://hastebin.com/ of ls -latr in your in your mame rom directory?

hansolo77

Are you sure you want me to use the 2.0.0.5 version and not the 2.0.0.7 from 3 days ago? I should mention that I was using 2.0.0.6 the last time I scraped...

I don't know what hastebin is or how to use it, but here is a pastebin:

http://pastebin.com/b7aikc7Z

hansolo77

I think maybe I need to rebuild my rom list in clrmamepro. Get rid of all the dupes to make it 1 game 1 rom. I did that with my consoles.

http://pastebin.com/6ccVRfYK

This is my 5-minute UXS test. There are multiple entries for my 19XX game. But when I compare it with the rom folder, I have:

19xx.zip
19xxa.zip
19xxh.zip
19xxj.zip
19xxjr1.zip

So I dunno, maybe it's working correctly? I just don't like seeing the same game listed 5 times. I guess I can change the UXS naming format. It's default to Standard. Which just does "The Game Name". Maybe if I change it to Gamename+Country it will look better? I'll try that.

Well, not much progress. This is what my game list inside ES looks like now for 19XX:

19XX: The War Against Destiny (USA)
19XX: The War Against Destiny
19XX: The War Against Destiny
19XX: The War Against Destiny
19XX: The War Against Destiny (Japon)

So it's somewhat there. The 3 that didn't display a country are Non-Working, HISPANIC, and a Japanese. Japon is also Japanese. So maybe the reason they're being displayed like that is because the scraper's database is incomplete? Like, it doesn't have the country information so it doesn't include it? What's even worse is down at "720 Degrees" where it's listed 6 times without any country information. So this isn't helpful. I'm going to try the Gamename+Bracket mode next. It says it does it as "The Game Name + The Rom FilName Bracket". So maybe that will help distinguish. I really feel like I should probably just go in and rebuild a clean set without all the dupes/clones.

Ok, re-scraped up the the A's with the Gamename+Bracket and it didn't do anything. It scrapes and makes the gamelist.xml just have the game name. It completely skipped adding the (filename) to the game (this is definitely a bug, NONE of the games had their filenames attached to the game name). So I went back and tried to scrape using the internal ES scraper. It scrapes just the flyer artwork, and also fails like the UXS to identify differences in clones on the game list. So I think what I'm going to do is go back to the Gamename+Country mode which at least showed the countries where the data exists. I'll just have to live with that I guess. I'd hate to have to rebuild a romset because of this. There are differences that are advantageous in clones. Things like Japanese only games being "illegally" translated into English, games hacked to use joysticks instead of steering wheels, etc. So having clones is a good thing. I just wish there was a scraper out there that could identify those games, rather than simply saying the obvious name of it multiple times without no distinction.

Shakz

Ah ok this makes sense now.
You don't need to use the old version. UXS is doing exactly what it is supposed to do. You need to merge your romset into one zip file. Right now everything is separated out with the parent and the clones from all the different regions. Merging them will combine all the parents and clones in one zip.
For UXS to work right you will need to merge them.
Don't forget to clear out all the old art as well before scraping again....it can start to consume quite a bit of space on a tiny SD card.
You might want to try with just a handful of games to start...until you're sure we have this figured out ;-)

dankcushions

@hansolo77 sselph scraper uses mamedb, which has been down for ages. many posts about this.

Rion

@hansolo77 This is a limitation of the current database on screenscraper.fr.

More info about it here

hansolo77

@Shakz Will merging the ROMs cause me to lose the ability to play various clones? How does it display them in the ES if they're merged?

Shakz

Yeah it will....squishes them all up under the parent rom and thats what you play. :-(
Unless someone knows a trick in lr-mame
If they are merged they just display as one game....the parent.

hansolo77

Bummer. I think I'll just use what I got then, and wait for an updated database that has more details about the country/etc.

screech

hi @hansolo77

So just for information :

Name + Country will try to match your rom country to the database (and if weel register in DB and found) will show you the country.
Name + Bracket will catch the bracket from your Rom Filname (ex : mario (jap)(v2).zip will give you : Mario Bros. (jap)(v2) )

So for now the DB on Arcade game is not complete... And we have a lot's of work to do on it...

The main thing is, today, we can't specify so much things about clone (except some country).

the Screenscraper V2 website and DB is in progress, when it's finish, we can add lot's more information about clone (country all of them, clone, and some other stuff). So when it's done I can put these information in UXS to scrape.

The best way (today) is to merge your rom to avoid duplicate. Sorry.

hansolo77

Name+Bracket only generates a gamelist.xml with the Game Name. mario (jap)(v2).zip creates Mario Bros.

In my previous example... Name+Bracket creates:

19xx.zip
19xxa.zip
19xxh.zip
19xxj.zip
19xxjr1.zip

to

19XX: The War Against Destiny
19XX: The War Against Destiny
19XX: The War Against Destiny
19XX: The War Against Destiny
19XX: The War Against Destiny

Where as GameName+Country will at least provide the (country) in the name if the data exists in the db. Games like Pacman, which is a clone of Puckman, just doesn't show up. I think it should be listed as Pacman (Clone of Puckman).

screech

that's normal...

19xx.zip
19xxa.zip
19xxh.zip
19xxj.zip
19xxjr1.zip

haven't anything between bracket ;)

if you have something like :
19xxjr1 (1 player).zip then name will be 19XX: The War Against Destiny (1 player)

Like I say, the next DB version will provide more information on clone (at least "clone") so I can add these info in UXS ;)

hansolo77

Alright, cool. :)

hansolo77

I'm back again trying to scrape MAME. Using the options available in the latest version of USX I'm still getting a simple gamelist without all the extra bits. Has the database been updated any with the information on clones/etc yet? I know you said this was in the works before, but this thread is 8 months old and I thought maybe SOME progress has been made...