RetroPi Media - gamelist.xml generator

bhakta

Can I give it a go?
I wanted to convert hyperspin and nfo files to pegasus but it's painfully slow to manually do it. even using scripts and trying to get everything right in regex.

TIA

MotoGp

Here is the first release. I added the Marquee files

As always try one system first, back up your data first.

There is nothing to install, just unzip the file and run RetroPieMedia.exe. Do not try to run from within the zip file. Keep all the files from the zip in the same folder. The folder name doesn’t matter.

RetroPie Media Basic Instructions:

Setup folders and extensions
Browse to rom Folder for the desired system.
Browse and select Hyperlist.xml file for system.
Click Import button. This process can take a little time depending on how many games are listed in the xml file. Wait for the “Success” message to appear.
Now the Hyperlist.xml will be displayed in the data grid. Anything in green means the file was located. Red means the file was not found. If all of your roms are red double check you entered the correct rom extension. Also all roms must have the same extension.
At this point no files are changed/created.
If you have unmatched roms you can match them by selecting the game name in the left data grid then selecting the corresponding rom file in the unmatched list then click match.
Once you’ve matched all the files you can, you can archive the unmatched files, they get moved to roms\orphans.
If you want to remove games from the list that do not have a rom, click the” remove games “button. This is only removing it from the data grid. To undo just click “import” again.
Once you have the list how you want it, click “Generate xml” This will create a backup of your current gamelist.xml and then create a new one.

You can download the program from here:
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release 2.zip?dl=0

The xml output looks like this.
<game>
<path>./10-Yard Fight (USA, Europe).zip</path>
<name>10-Yard Fight (USA, Europe)</name>
<cover>./media/covers/10-Yard Fight (USA, Europe).png</cover>
<image>./media/screenshots/10-Yard Fight (USA, Europe).png</image>
<marquee>./media/marquees/10-Yard Fight (USA, Europe).png</marquee>
<video>./media/videos/10-Yard Fight (USA, Europe).mp4</video>
<rating>0.9</rating>
<desc>10-Yard Fight (USA, Europe)</desc>
<releasedate>1985</releasedate>
<developer>Nintendo</developer>
<publisher>Nintendo</publisher>
<genre>Sports/Football</genre>
<players>2</players>
</game>

MotoGp

@bhakta said in RetroPi Media - gamelist.xml generator:

Can I give it a go?
I wanted to convert hyperspin and nfo files to pegasus but it's painfully slow to manually do it. even using scripts and trying to get everything right in regex.

TIA

Please give it a try and let me know if you have an issues, thanks

davidonabus

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

MotoGp

@davidonabus said in RetroPi Media - gamelist.xml generator:

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

When reading the gamelist.xml file, I added code to check to see how many attributes each game has. I found that some of the "scrapers" were inconsistant with what information it wrote for each game. So if you open your gamelist.xml and scoll down to the game 005.zip you will see that between the <game> & </game> there are only 4 nodes like <path><Image> etc. can you post the xml data from you game list for this particular game? I guess I could edit the code to still import it but you may only get the game name. Posting a snippet of your xml will be helpful. What scraper did you use?

rand0m

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;

MotoGp

@MotoGp said in RetroPi Media - gamelist.xml generator:

@davidonabus said in RetroPi Media - gamelist.xml generator:

@MotoGp I'm giving this a go. I pointed the app to my roms folder (arcade) and I received the following error:

Source File Error

Action cancelled.

Source File does not have correct number of attributes

ROM: ./005.zip Only has: 4

OK

When reading the gamelist.xml file, I added code to check to see how many attributes each game has. I found that some of the "scrapers" were inconsistant with what information it wrote for each game. So if you open your gamelist.xml and scoll down to the game 005.zip you will see that between the <game> & </game> there are only 4 nodes like <path><Image> etc. can you post the xml data from you game list for this particular game? I guess I could edit the code to still import it but you may only get the game name. Posting a snippet of your xml will be helpful. What scraper did you use?

I rewrote the code around this. Now if an attribute is missing like Genre, it will just leave it blank.
I updated the zip here.
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release 2.zip?dl=0

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;
that would be pretty easy to create.
So there is a file per system like nes.txt, atari2600.txt and in each txt it contains the all of the games? or do you select the games you want?

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp the file is txt named after the system like if the system is named genesis the filed created will be genesis.txt. The file uses ; as separator with every entry (game) being on a separate line and has following structure:

#Name;Title;Emulator;CloneOf;Year;Manufacturer;Category;Players;Rotation;Control;Status;DisplayCount;DisplayType;AltRomname;AltTitle;Extra;Buttons

Example:

Bank Heist (USA);Bank Heist;;;1983;Fox;Puzzle;;;;;;;;;;

Ok, I created this script for you. Where is the text file stored? In the root of the roms folder?

Here is a sample output for one system.

Aggressive Inline (USA);Aggressive Inline (USA);;;2002;Acclaim;Skating/Sports;;;;;;;;;;
AirForce Delta Storm (USA) (En,Ja,Fr,De);AirForce Delta Storm (USA) (En,Ja,Fr,De);;;2002;Konami;Shoot-'Em-Up;;;;;;;;;;
Aladdin (USA) (En,Fr,De,Es);Aladdin (USA) (En,Fr,De,Es);;;2004;Capcom;Platform;;;;;;;;;;
Alex Rider - Stormbreaker (USA);Alex Rider - Stormbreaker (USA);;;2006;THQ;Action;;;;;;;;;;
Alien Hominid (Europe) (En,Fr,De,Es,It);Alien Hominid (Europe) (En,Fr,De,Es,It);;;2006;Zoo Digital Publishing;Shooter;;;;;;;;;;
Alienators - Evolution Continues (USA, Europe);Alienators - Evolution Continues (USA, Europe);;;2001;Activision;Action;;;;;;;;;;
All Grown Up! - Express Yourself (USA, Europe);All Grown Up! - Express Yourself (USA, Europe);;;2004;THQ;Miscellaneous;;;;;;;;;;
All-Star Baseball 2003 (USA);All-Star Baseball 2003 (USA);;;2002;Acclaim;Baseball/Sports;;;;;;;;;;
All-Star Baseball 2004 (USA);All-Star Baseball 2004 (USA);;;2003;Acclaim;Baseball/Sports;;;;;;;;;;
Altered Beast - Guardian of the Realms (USA);Altered Beast - Guardian of the Realms (USA);;;2002;THQ;Beat-'Em-Up;;;;;;;;;;
Amazing Virtual Sea-Monkeys, The (USA);Amazing Virtual Sea-Monkeys, The (USA);;;2002;Conspiracy Entertainment;Action;;;;;;;;;;

rand0m

@MotoGp Thanks, every system has a separate list and only contains the items (roms) present in the rom directory ( given by user) if list is created through Attrcat-Mode. Otherwise a romlist can contain all items even those not present in a rom directory, users can then create a filter which only displays items which actually exist.

In PC the romlists are stored in Attrcat-Mode (root folder)\romlists. The sample is spot on.

MotoGp

@rand0m said in RetroPi Media - gamelist.xml generator:

@MotoGp Thanks, every system has a separate list and only contains the items (roms) present in the rom directory ( given by user) if list is created through Attrcat-Mode. Otherwise a romlist can contain all items even those not present in a rom directory, users can then create a filter which only displays items which actually exist.

In PC the romlists are stored in Attrcat-Mode (root folder)\romlists. The sample is spot on.

The latest release includes this function. It will only export games where the check box "include" is checked. The default is if the rom is found it is checked. So if there is a rom you don't want o include, you can just uncheck it. My only question is for the #name should it have the ROM extension?

Latest build is here 9/23/2019
https://www.dropbox.com/s/3bpa8435bfe6luv/RetroPieMedia Release.zip?dl=0

rand0m

@MotoGp #name does not include ROM extension, the rom "penny.bin" will be "penny" in #name. I checked out the app today, brilliant work. One thing I wish is that we also get the option to add rom folder (with options for including subfolders) then compare that to hyperlist file.

MotoGp

@rand0m I can add a checkbox to look in subfolders too. I'm planning to add where it will remember your settings.

For the rom folders I was thinking you could select the folder where are the systems are located then you just select the system from a dropdown list?

rand0m

@MotoGp It would be awesome if sub-folders could be added. For rom folders, drop down would definitely work for retropie but if it could be tweaked a little by adding an option for directly selecting a rom folder on hd (like between "retropie gamelist" and "select hyperlist") it would help a lot of people who run am/ es etc directly

MotoGp

@rand0m

I haven't forgotten this, the sub directories is a bit more code re-write than I thought. taking some time as quite a few functions have to be rewritten

miwasp

I love this kind of tools, so i did some tests. This is what i found so far:

The window is so large that even maximized on 1920x1080 you have scrollbars horizontally and vertically. This makes it difficult to handle.
Loading a full romset for fbneo (~6.000 roms) from a network drive is VERY slow (it takes ~15 minutes)
I have the folders "snap", "video" and "wheel" under every system folder in the roms dir, so i leave the field "Media folder" empty and set "Video folder" to "video" and so on. With these settings the entries in the gamelist.xml look like "<image>.//snap/..." (double slash).
A working gamelist entry like this:

  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <desc>Great 1000 Miles Rally © 1994 Kaneko.  A racing game featuring classic cars from Ferrari, Alfa Romeo, Mercedes Benz and more.</desc>
    <rating>0.4</rating>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>1-2</players>
    <image>./snap/gtmro.png</image>
    <video>./video/gtmro.mp4</video>
    <marquee>./wheel/gtmro.png</marquee>
  </game>

results after saving in this

  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <cover>.//covers/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</cover>
    <image>.//snap/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</image>
    <marquee>.//wheel/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</marquee>
    <video>.//video/1000 Miglia: Great 1000 Miles Rally (94/05/10).mp4</video>
    <rating>0.9</rating>
    <desc>1000 Miglia: Great 1000 Miles Rally (94/05/10)</desc>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>2</players>
  </game>

The content of <desc> is replaced with the content of <name>
The value of <rating> is wrong
The value of <players> "1-2" is replaced with "2"
The filenames for <cover>, <image>, <marquee> and <video> are generated based on <name>, not based on the name of the rom

MotoGp

@miwasp Thanks for the detail feedback...

@miwasp said in RetroPi Media - gamelist.xml generator:

I love this kind of tools, so i did some tests. This is what i found so far:

The window is so large that even maximized on 1920x1080 you have scrollbars horizontally and vertically. This makes it difficult to handle.
RESPONSE: Interesting as I use 1920x1080 and have no scrolls bars, are you sure you are set to 1920x1080?

Loading a full romset for fbneo (~6.000 roms) from a network drive is VERY slow (it takes ~15 minutes)
RESPONSE: This app wasn't designed to be utilized over a network as that can get very complicated. A full-blown enterprise program would have to deal with that somehow but that's beyond my coding ability, so this is not something I will even attempt to address. HD speed, network speed / traffic will all slow it down. I ran this against my MAME with ~7500 and it only took 14 seconds to populate the list and 8 min to search the roms not found. For each file in the folder all~7500 it searches the xml file to see if there is a match. I will take a look at this code and see if there is a different approach I can take to speed it up. I will make this search optional / manual run after the list populates. Some people won't care if there are orphan roms.

I have the folders "snap", "video" and "wheel" under every system folder in the roms dir, so i leave the field "Media folder" empty and set "Video folder" to "video" and so on. With these settings the entries in the gamelist.xml look like "<image>.//snap/..." (double slash).
RESPONSE: The "/" is hard coded in the output so I can make an adjustment for when the folder is left blank. The trick is in the Windows it uses "" and in the Pi it uses "/". I'll have to create a workaround for this.

A working gamelist entry like this:
  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <desc>Great 1000 Miles Rally © 1994 Kaneko.  A racing game featuring classic cars from Ferrari, Alfa Romeo, Mercedes Benz and more.</desc>
    <rating>0.4</rating>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>1-2</players>
    <image>./snap/gtmro.png</image>
    <video>./video/gtmro.mp4</video>
    <marquee>./wheel/gtmro.png</marquee>
  </game>
results after saving in this
  <game>
    <path>./gtmro.zip</path>
    <name>1000 Miglia: Great 1000 Miles Rally (94/05/10)</name>
    <cover>.//covers/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</cover>
    <image>.//snap/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</image>
    <marquee>.//wheel/1000 Miglia: Great 1000 Miles Rally (94/05/10).png</marquee>
    <video>.//video/1000 Miglia: Great 1000 Miles Rally (94/05/10).mp4</video>
    <rating>0.9</rating>
    <desc>1000 Miglia: Great 1000 Miles Rally (94/05/10)</desc>
    <releasedate>19940101T000000</releasedate>
    <developer>Kaneko</developer>
    <publisher>Kaneko</publisher>
    <genre>Rennen, Fahren-Rennen, Fahren / Rennen</genre>
    <players>2</players>
  </game>
The content of <desc> is replaced with the content of <name>
RESPONSE: So if you import the gamelist.xml then the <desc> should be there and we can use it in the output. IF you use a hyperlist.xml it will not have the <desc> so we use the name again.

The value of <rating> is wrong
RESPONSE: same as the description. the rating can be used if there, hyperlist didn't use a numerical value, its is like "Other - NR (Not Rated)"

The value of <players> "1-2" is replaced with "2"
RESPONSE: similar to the <desc> issue, we can use the value if it came from the gamelist.xml otherwise either leave blank or use 2

The filenames for <cover>, <image>, <marquee> and <video> are generated based on <name>, not based on the name of the rom
RESPONSE: you are correct, i will fix

So right now I'm adding the ability to search subfolders which really took me off track as it is more work than I really wanted to do. Once this is finished, I'll tackle the things you mentioned.

MotoGp

@MotoGp I've made some really good progress tonight. I should have a new release ready tomorrow. It will included subfolder search as default and the issues brought up by @miwasp except the performance piece although I did made the search for orphans a manual button so you can choose to just not run that part as it can take awhile on large lists.

miwasp

@MotoGp said in RetroPi Media - gamelist.xml generator:

The window is so large that even maximized on 1920x1080 you have scrollbars horizontally and vertically. This makes it difficult to handle.
RESPONSE: Interesting as I use 1920x1080 and have no scrolls bars, are you sure you are set to 1920x1080?

Yes, resolution is 1920x1080 but scaling is set to 125%. I can disable scaling for this app only in the compatibility settings, so this is not really a problem.

@MotoGp said in RetroPi Media - gamelist.xml generator:

@MotoGp I've made some really good progress tonight. I should have a new release ready tomorrow. It will included subfolder search as default and the issues brought up by @miwasp except the performance piece although I did made the search for orphans a manual button so you can choose to just not run that part as it can take awhile on large lists.

That sounds great, thank you very much.

MotoGp

OK here we go.

Here is a screenshot of the latest version. I moved a few things around.

By default it uses the gamelist.xml file but if you check the box it will use the Hyperlist file.

You can download the latest version from the first post

I've tested this on my system and everything appears to work properly, please backup your data before trying.
Please let me know how it works for you!

Version 1.004

Fixes:

For large rom sets, made the orphan tool optional / manual
fixed the "//" double slash issue if some folder settings are left blank
corrected the output XML file.
fixed the file names in xml to match the ROM name not the <desc>

added features

matching of ROMS will now included subfolders in match. This also applies to all videos, marquees, screenshots etc.
ability to save your settings
The rom list will now display all attributes that are imported from the gamelist.xml and/or the hyperlist.xml
added version number to title at top so we now if we are working with the latest version

example xml of output

<game>
    <path>./10yard.zip</path>
    <name>10-Yard Fight (World, set 1)</name>
    <cover>./media/covers/10yard.png</cover>
    <image>./media/screenshots/10yard.png</image>
    <marquee>./media/marquees/10yard.png</marquee>
    <video>./media/videos/10yard.mp4</video>
    <rating>0.9</rating>
    <desc>Football Game)</desc>
    <releasedate>1983</releasedate>
    <developer>Irem</developer>
    <publisher>Irem</publisher>
    <genre>Sports / Amer. Football</genre>
    <players>2</players>
  </game>