RetroPi Media - gamelist.xml generator

MotoGp

@SuperPitendo said in RetroPi Media - gamelist.xml generator:

@MotoGp I know, some dudes like marquees and others dont.
But the effort you put in this tool will not be appreciated as it should if you target (unintenionally) a splitgroup (metadata user) of a splitgroup (without interest in marquees). You see, the target group become pretty small.
My advise: more flexibility!!!

Anything that can be scraped, should be an option (like marquees)

I like my metadata in one folder per system (like sselphscraper does it), not in three different folders. here should be an option too.

I see there is a kind of check if there are some missing roms. But missing in comparison to what? Wich list from where is taken to check the missing roms? I think being able to add the newest dat.files by yourself would be an amazing feature in your tool.

The one thing I really miss in any scraper and tool is a prepared (maybe drop down) list of publisher, developer and genre. After many scraping session with different scrapers, I was always unamused about the inkonsistencies in spelling of these tags and sorting by these tags was useless...

Cheers!

This app isn't a scraper, it does not download anything. I've added the marquess to the setup.
You can put all of your data in one folder like roms\systemname\media\
the check is this. If you use either a gamelist from someone or a Hyperlist, it check to see if you have the roms that match the list. I'm not sure what the dat.files are.
I'm note usre how to handle this one.

The overall intent of my app was to see what roms you had and the cooresponding images. It just basically shows you what you have and what is missing.

It allows you to edit the gamelist file. But again, its not a scraper app. The whole scraper thing is foreign to me. I understnad what it does but I think the results of the ones I tried and pretty bad.

If to others this app serves no purpose, I'll just use it myself. I just thought I'd share it and maybe would be useful to someone else. I made a similar app for Hyperspin years ago and was useful for that system but Hyperspin didn't "scrape" to create your gamelist, the lists were already made and you downloaded them to your systema and sometimes you had a game in the menu but you actually did not have the rom, so I was avoiding that issue.

rand0m

@MotoGp App looks great, would it be possible to get Attract-Mode lists from it? They are in txt format so will require output in simple text rather then xml.

MotoGp

@rand0m I haven't setup attract mode on mine. can you show me the file that is used for it? I'm sure I can create it.

bhakta

Can I give it a go?
I wanted to convert hyperspin and nfo files to pegasus but it's painfully slow to manually do it. even using scripts and trying to get everything right in regex.

TIA

MotoGp

Here is the first release. I added the Marquee files

As always try one system first, back up your data first.

There is nothing to install, just unzip the file and run RetroPieMedia.exe. Do not try to run from within the zip file. Keep all the files from the zip in the same folder. The folder name doesn’t matter.

RetroPie Media Basic Instructions:

Setup folders and extensions
Browse to rom Folder for the desired system.
Browse and select Hyperlist.xml file for system.
Click Import button. This process can take a little time depending on how many games are listed in the xml file. Wait for the “Success” message to appear.
Now the Hyperlist.xml will be displayed in the data grid. Anything in green means the file was located. Red means the file was not found. If all of your roms are red double check you entered the correct rom extension. Also all roms must have the same extension.
At this point no files are changed/created.
If you have unmatched roms you can match them by selecting the game name in the left data grid then selecting the corresponding rom file in the unmatched list then click match.
Once you’ve matched all the files you can, you can archive the unmatched files, they get moved to roms\orphans.
If you want to remove games from the list that do not have a rom, click the” remove games “button. This is only removing it from the data grid. To undo just click “import” again.
Once you have the list how you want it, click “Generate xml” This will create a backup of your current gamelist.xml and then create a new one.

You can download the program from here:
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release 2.zip?dl=0

The xml output looks like this.
<game>
<path>./10-Yard Fight (USA, Europe).zip</path>
<name>10-Yard Fight (USA, Europe)</name>
<cover>./media/covers/10-Yard Fight (USA, Europe).png</cover>
<image>./media/screenshots/10-Yard Fight (USA, Europe).png</image>
<marquee>./media/marquees/10-Yard Fight (USA, Europe).png</marquee>
<video>./media/videos/10-Yard Fight (USA, Europe).mp4</video>
<rating>0.9</rating>
<desc>10-Yard Fight (USA, Europe)</desc>
<releasedate>1985</releasedate>
<developer>Nintendo</developer>
<publisher>Nintendo</publisher>
<genre>Sports/Football</genre>
<players>2</players>
</game>

MotoGp

@bhakta said in RetroPi Media - gamelist.xml generator:

Can I give it a go?
I wanted to convert hyperspin and nfo files to pegasus but it's painfully slow to manually do it. even using scripts and trying to get everything right in regex.

TIA

Please give it a try and let me know if you have an issues, thanks

davidonabus

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

MotoGp

@davidonabus said in RetroPi Media - gamelist.xml generator:

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

When reading the gamelist.xml file, I added code to check to see how many attributes each game has. I found that some of the "scrapers" were inconsistant with what information it wrote for each game. So if you open your gamelist.xml and scoll down to the game 005.zip you will see that between the <game> & </game> there are only 4 nodes like <path><Image> etc. can you post the xml data from you game list for this particular game? I guess I could edit the code to still import it but you may only get the game name. Posting a snippet of your xml will be helpful. What scraper did you use?

rand0m

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;

MotoGp

@MotoGp said in RetroPi Media - gamelist.xml generator:

@davidonabus said in RetroPi Media - gamelist.xml generator:

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

When reading the gamelist.xml file, I added code to check to see how many attributes each game has. I found that some of the "scrapers" were inconsistant with what information it wrote for each game. So if you open your gamelist.xml and scoll down to the game 005.zip you will see that between the <game> & </game> there are only 4 nodes like <path><Image> etc. can you post the xml data from you game list for this particular game? I guess I could edit the code to still import it but you may only get the game name. Posting a snippet of your xml will be helpful. What scraper did you use?

I rewrote the code around this. Now if an attribute is missing like Genre, it will just leave it blank.
I updated the zip here.
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release 2.zip?dl=0

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;
that would be pretty easy to create.
So there is a file per system like nes.txt, atari2600.txt and in each txt it contains the all of the games? or do you select the games you want?

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;

Ok, I created this script for you. Where is the text file stored? In the root of the roms folder?

Here is a sample output for one system.

Aggressive Inline (USA);Aggressive Inline (USA);;;2002;Acclaim;Skating/Sports;;;;;;;;;;
AirForce Delta Storm (USA) (En,Ja,Fr,De);AirForce Delta Storm (USA) (En,Ja,Fr,De);;;2002;Konami;Shoot-'Em-Up;;;;;;;;;;
Aladdin (USA) (En,Fr,De,Es);Aladdin (USA) (En,Fr,De,Es);;;2004;Capcom;Platform;;;;;;;;;;
Alex Rider - Stormbreaker (USA);Alex Rider - Stormbreaker (USA);;;2006;THQ;Action;;;;;;;;;;
Alien Hominid (Europe) (En,Fr,De,Es,It);Alien Hominid (Europe) (En,Fr,De,Es,It);;;2006;Zoo Digital Publishing;Shooter;;;;;;;;;;
Alienators - Evolution Continues (USA, Europe);Alienators - Evolution Continues (USA, Europe);;;2001;Activision;Action;;;;;;;;;;
All Grown Up! - Express Yourself (USA, Europe);All Grown Up! - Express Yourself (USA, Europe);;;2004;THQ;Miscellaneous;;;;;;;;;;
All-Star Baseball 2003 (USA);All-Star Baseball 2003 (USA);;;2002;Acclaim;Baseball/Sports;;;;;;;;;;
All-Star Baseball 2004 (USA);All-Star Baseball 2004 (USA);;;2003;Acclaim;Baseball/Sports;;;;;;;;;;
Altered Beast - Guardian of the Realms (USA);Altered Beast - Guardian of the Realms (USA);;;2002;THQ;Beat-'Em-Up;;;;;;;;;;
Amazing Virtual Sea-Monkeys, The (USA);Amazing Virtual Sea-Monkeys, The (USA);;;2002;Conspiracy Entertainment;Action;;;;;;;;;;

rand0m

@MotoGp Thanks, every system has a separate list and only contains the items (roms) present in the rom directory ( given by user) if list is created through Attrcat-Mode. Otherwise a romlist can contain all items even those not present in a rom directory, users can then create a filter which only displays items which actually exist.

In PC the romlists are stored in Attrcat-Mode (root folder)\romlists. The sample is spot on.

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp Thanks, every system has a separate list and only contains the items (roms) present in the rom directory ( given by user) if list is created through Attrcat-Mode. Otherwise a romlist can contain all items even those not present in a rom directory, users can then create a filter which only displays items which actually exist.

In PC the romlists are stored in Attrcat-Mode (root folder)\romlists. The sample is spot on.

The latest release includes this function. It will only export games where the check box "include" is checked. The default is if the rom is found it is checked. So if there is a rom you don't want o include, you can just uncheck it. My only question is for the #name should it have the ROM extension?

Latest build is here 9/23/2019
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release.zip?dl=0

rand0m

@MotoGp #name does not include ROM extension, the rom "penny.bin" will be "penny" in #name. I checked out the app today, brilliant work. One thing I wish is that we also get the option to add rom folder (with options for including subfolders) then compare that to hyperlist file.

MotoGp

@rand0m I can add a checkbox to look in subfolders too. I'm planning to add where it will remember your settings.

For the rom folders I was thinking you could select the folder where are the systems are located then you just select the system from a dropdown list?

rand0m

@MotoGp It would be awesome if sub-folders could be added. For rom folders, drop down would definitely work for retropie but if it could be tweaked a little by adding an option for directly selecting a rom folder on hd (like between "retropie gamelist" and "select hyperlist") it would help a lot of people who run am/ es etc directly

MotoGp

@rand0m

I haven't forgotten this, the sub directories is a bit more code re-write than I thought. taking some time as quite a few functions have to be rewritten

miwasp

I love this kind of tools, so i did some tests. This is what i found so far:

The window is so large that even maximized on 1920x1080 you have scrollbars horizontally and vertically. This makes it difficult to handle.
Loading a full romset for fbneo (~6.000 roms) from a network drive is VERY slow (it takes ~15 minutes)
I have the folders "snap", "video" and "wheel" under every system folder in the roms dir, so i leave the field "Media folder" empty and set "Video folder" to "video" and so on. With these settings the entries in the gamelist.xml look like "<image>.//snap/..." (double slash).
A working gamelist entry like this:

  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <desc>Great 1000 Miles Rally © 1994 Kaneko.  A racing game featuring classic cars from Ferrari, Alfa Romeo, Mercedes Benz and more.</desc>
    <rating>0.4</rating>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>1-2</players>
    <image>./snap/gtmro.png</image>
    <video>./video/gtmro.mp4</video>
    <marquee>./wheel/gtmro.png</marquee>
  </game>

results after saving in this

  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <cover>.//covers/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</cover>
    <image>.//snap/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</image>
    <marquee>.//wheel/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</marquee>
    <video>.//video/1000 Miglia: Great 1000 Miles Rally (94/05/10).mp4</video>
    <rating>0.9</rating>
    <desc>1000 Miglia: Great 1000 Miles Rally (94/05/10)</desc>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>2</players>
  </game>

The content of <desc> is replaced with the content of <name>
The value of <rating> is wrong
The value of <players> "1-2" is replaced with "2"
The filenames for <cover>, <image>, <marquee> and <video> are generated based on <name>, not based on the name of the rom

MotoGp

@miwasp Thanks for the detail feedback...

@miwasp said in RetroPi Media - gamelist.xml generator:

I love this kind of tools, so i did some tests. This is what i found so far:

The window is so large that even maximized on 1920x1080 you have scrollbars horizontally and vertically. This makes it difficult to handle.
RESPONSE: Interesting as I use 1920x1080 and have no scrolls bars, are you sure you are set to 1920x1080?

Loading a full romset for fbneo (~6.000 roms) from a network drive is VERY slow (it takes ~15 minutes)
RESPONSE: This app wasn't designed to be utilized over a network as that can get very complicated. A full-blown enterprise program would have to deal with that somehow but that's beyond my coding ability, so this is not something I will even attempt to address. HD speed, network speed / traffic will all slow it down. I ran this against my MAME with ~7500 and it only took 14 seconds to populate the list and 8 min to search the roms not found. For each file in the folder all~7500 it searches the xml file to see if there is a match. I will take a look at this code and see if there is a different approach I can take to speed it up. I will make this search optional / manual run after the list populates. Some people won't care if there are orphan roms.

I have the folders "snap", "video" and "wheel" under every system folder in the roms dir, so i leave the field "Media folder" empty and set "Video folder" to "video" and so on. With these settings the entries in the gamelist.xml look like "<image>.//snap/..." (double slash).
RESPONSE: The "/" is hard coded in the output so I can make an adjustment for when the folder is left blank. The trick is in the Windows it uses "" and in the Pi it uses "/". I'll have to create a workaround for this.

A working gamelist entry like this:
  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <desc>Great 1000 Miles Rally © 1994 Kaneko.  A racing game featuring classic cars from Ferrari, Alfa Romeo, Mercedes Benz and more.</desc>
    <rating>0.4</rating>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>1-2</players>
    <image>./snap/gtmro.png</image>
    <video>./video/gtmro.mp4</video>
    <marquee>./wheel/gtmro.png</marquee>
  </game>
results after saving in this
  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <cover>.//covers/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</cover>
    <image>.//snap/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</image>
    <marquee>.//wheel/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</marquee>
    <video>.//video/1000 Miglia: Great 1000 Miles Rally (94/05/10).mp4</video>
    <rating>0.9</rating>
    <desc>1000 Miglia: Great 1000 Miles Rally (94/05/10)</desc>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>2</players>
  </game>
The content of <desc> is replaced with the content of <name>
RESPONSE: So if you import the gamelist.xml then the <desc> should be there and we can use it in the output. IF you use a hyperlist.xml it will not have the <desc> so we use the name again.

The value of <rating> is wrong
RESPONSE: same as the description. the rating can be used if there, hyperlist didn't use a numerical value, its is like "Other - NR (Not Rated)"

The value of <players> "1-2" is replaced with "2"
RESPONSE: similar to the <desc> issue, we can use the value if it came from the gamelist.xml otherwise either leave blank or use 2

The filenames for <cover>, <image>, <marquee> and <video> are generated based on <name>, not based on the name of the rom
RESPONSE: you are correct, i will fix

So right now I'm adding the ability to search subfolders which really took me off track as it is more work than I really wanted to do. Once this is finished, I'll tackle the things you mentioned.