Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

Making premade gamelist.xml with xtra media collections


  • Global Moderator

    Almost everyone is accounted for except for that suspicious new moderator, @ColonelMustard.


  • Global Moderator

    @genericuserdude

    I'm going to take the liberty of unbanning your original account, as it has all the markings of an accident. Since every ban is added to your record, I also made a note that it seems to have been unintentional in some way.



  • @mediamogul Thanks to you and everyone else for looking into it.

    If you do find out why somebody did it and that it wasn't an accident, please let me know what I did and I will not do it again in the future.



  • @UDb23 I was in the middle of giving you a long reply about the program. It looks great. I tried 5-10 of the Atari 7800 txt files and there weren't any problems. I tried 5 NES files and 2 of them gave me errors. I had them all copied to send to you, but my laptop with all my pi work died on me. I think the motherboard or at least the GPU is fried. I don't know how long it's going to take me to get set up again. I don't have any of the work on this old PC I'm typing from and I can't afford a new laptop.

    I might have time over the next few days to try to at least round up the synopsis files from other systems to give you a list, but I'm kind of depressed now. I just lost a few months of work.



  • @Used2BeRX uh, that's bad! Really sorry about it.
    You should still be able to access the hd of your laptop. There are quite cheap adaptors to connect laptop hd's to a PC. Hope you can fix your laptop and/or recover your files.



  • @UDb23 said in Making premade gamelist.xml with xtra media collections:

    @Used2BeRX uh, that's bad! Really sorry about it.
    You should still be able to access the hd of your laptop. There are quite cheap adaptors to connect laptop hd's to a PC. Hope you can fix your laptop and/or recover your files.

    LOL... I don't think I'm going to be able to fix the laptop. It's now sitting in my garbage in about a thousand pieces :)

    I'm sure I can get the data off of it. The failure was either the motherboard or GPU. I'm just seriously limited on funds right now and need to get a job. I have to wait on my brother to get me a 2TB external drive to get back into any serious work on this stuff and to get the data off of that drive. He loaned it out to somebody else a while back but they're only using it for Wii games.

    This work is too important though and I didn't want to lose your interest by stalling so I pulled all the synopsis files for all of the systems so I could test this out again on another computer and give you initial feedback. I'm really impressed with it so far and I can't wait to see where this goes.

    Brace yourself.... I've got a LOT of feedback for you, and I'm going to have more info for you hopefully later today. Here we go........

    I started with the 7800 even though you made this with the 2600 in mind. It was a much smaller list of only 90 games so I was able to test every single file in about 10 minutes.

    7800 TEST RUN (EVERY GAME TXT FILE TESTED):

    I got an error and couldn't create the file for the following games:
    Beef Drop - 10 new levels.txt
    Beef Drop - More Beef Drop.txt
    bonQ.txt
    Centipede.txt
    FailSafe.txt
    Fatal Run.txt
    Meltdown.txt
    Ms Pac-Man.txt
    Pac-Man Collection.txt
    Robotron 2084.txt
    Snowday Jr Pac-Man.txt
    Tomcat F 14 Simulator.txt
    Tower Toppler.txt

    Here's the Error:

    Unhandled exception has occurred in your application.  If you click Continue, the application will ignore this error and attempt to continue.  If you click Quit, the application will close immediately.
    
    The given path's format is not supported.
    
    DETAILS:
    
    See the end of this message for details on invoking 
    just-in-time (JIT) debugging instead of this dialog box.
    
    ************** Exception Text **************
    System.NotSupportedException: The given path's format is not supported.
       at System.Security.Permissions.FileIOPermission.EmulateFileIOPermissionChecks(String fullPath)
       at System.Security.Permissions.FileIOPermission.QuickDemand(FileIOPermissionAccess access, String fullPath, Boolean checkForDuplicates, Boolean needFullPath)
       at Microsoft.VisualBasic.FileIO.FileSystem.NormalizePath(String Path)
       at Microsoft.VisualBasic.FileIO.FileSystem.CombinePath(String baseDirectory, String relativePath)
       at Microsoft.VisualBasic.MyServices.FileSystemProxy.CombinePath(String baseDirectory, String relativePath)
       at MetaX.Form1.ConvertButton_Click(Object sender, EventArgs e)
       at System.Windows.Forms.Control.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnClick(EventArgs e)
       at System.Windows.Forms.Button.OnMouseUp(MouseEventArgs mevent)
       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
       at System.Windows.Forms.Control.WndProc(Message& m)
       at System.Windows.Forms.ButtonBase.WndProc(Message& m)
       at System.Windows.Forms.Button.WndProc(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
    
    
    ************** Loaded Assemblies **************
    mscorlib
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1648.0 built by: NETFXREL3STAGE
        CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v4.0.30319/mscorlib.dll
    ----------------------------------------
    MetaX
        Assembly Version: 1.0.0.0
        Win32 Version: 1.0.0.0
        CodeBase: file:///C:/Users/The%20Dude/AppData/Local/Apps/2.0/Q5PQQAWH.RQH/9XWRQPVQ.KZM/meta..tion_9bae49a2533cd5b2_0001.0000_1813f733e3f252fe/MetaX.exe
    ----------------------------------------
    Microsoft.VisualBasic
        Assembly Version: 10.0.0.0
        Win32 Version: 14.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Microsoft.VisualBasic/v4.0_10.0.0.0__b03f5f7f11d50a3a/Microsoft.VisualBasic.dll
    ----------------------------------------
    System.Windows.Forms
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Windows.Forms/v4.0_4.0.0.0__b77a5c561934e089/System.Windows.Forms.dll
    ----------------------------------------
    System
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1647.0 built by: NETFXREL3STAGE
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System/v4.0_4.0.0.0__b77a5c561934e089/System.dll
    ----------------------------------------
    System.Drawing
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Drawing/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Drawing.dll
    ----------------------------------------
    System.Configuration
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Configuration/v4.0_4.0.0.0__b03f5f7f11d50a3a/System.Configuration.dll
    ----------------------------------------
    System.Core
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1647.0 built by: NETFXREL3STAGE
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Core/v4.0_4.0.0.0__b77a5c561934e089/System.Core.dll
    ----------------------------------------
    System.Xml
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Xml/v4.0_4.0.0.0__b77a5c561934e089/System.Xml.dll
    ----------------------------------------
    System.Runtime.Remoting
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/System.Runtime.Remoting/v4.0_4.0.0.0__b77a5c561934e089/System.Runtime.Remoting.dll
    ----------------------------------------
    Accessibility
        Assembly Version: 4.0.0.0
        Win32 Version: 4.6.1586.0 built by: NETFXREL2
        CodeBase: file:///C:/Windows/Microsoft.Net/assembly/GAC_MSIL/Accessibility/v4.0_4.0.0.0__b03f5f7f11d50a3a/Accessibility.dll
    ----------------------------------------
    
    ************** JIT Debugging **************
    To enable just-in-time (JIT) debugging, the .config file for this
    application or computer (machine.config) must have the
    jitDebugging value set in the system.windows.forms section.
    The application must also be compiled with debugging
    enabled.
    
    For example:
    
    <configuration>
        <system.windows.forms jitDebugging="true" />
    </configuration>
    
    When JIT debugging is enabled, any unhandled exception
    will be sent to the JIT debugger registered on the computer
    rather than be handled by this dialog box.
    

    I took a look at the failed files, and I think I spotted the potential problems with them.

    Beef Drop - 10 New Levels.txt, Beef Drop - More Beef Drop.txt, Robotron - 2084.txt, Snowday - Jr Pac-Man.txt, Tomcat F 14 Simulator.txt

    These have a ":" in the title. There are a LOT of games over all the systems that will have this character as I thought it looks a lot better in the romlist than using the " - " between the title and sub-title. For example "Beef Drop: 10 New Levels".

    (NOTE: Snowday Jr Pac-Man.txt also has a "." in the title, so probably would have failed for that as well)

    bonQ.txt
    The title has an asterisk in it. It is actually b**nQ. Would only be a problem with games like Q*bert and such.

    Centipede.txt
    A ";" in the Players line. Players: 1 or 2 VS; CO-OP
    I believe there are a lot of instances of this over all the systems.

    Ms Pac-Man.txt
    Maybe the "." in "Ms. Pac-Man"???

    Pac-Man Collection.txt
    Maybe the "!" in "Pac-Man Collection!" or the "(" and ")" in "Region: Homebrew (USA)"???

    Tower Toppler.txt
    Maybe the ":" in the description? There's a section toward the bottom that has hints and says "Hints: "

    I'm thinking that the program should avoid any colons while it's scraping text since a lot of the Game Titles will have them and they will probably creep up in other places like this example.

    This would also explain the failures for Meltdown.txt, Fatal Run.txt and FailSafe.txt

    I hope I'm right about all of these and saved you some time trying to figure out what went wrong.

    I also tested the first 25 text files for the NES and had 10 failures. I didn't check why they failed yet. Here's the list of games:

    10-Yard Fight.txt
    1942.txt
    1943.txt
    3-D Battles of WorldRunner - Part II
    720.txt
    75 Bingo.txt
    8 Eyes.txt
    A Boy and His Blob.txt
    A Nightmare On Elm Street.txt
    Abadox.txt

    Here's a full upload of the NES and 7800 synopsis files for you to check out:

    https://www.sendspace.com/file/ueaeig

    I will try to look at files from every system and get you a list of all of the possible tags that the program should look for. Most of them aren't currently used by any skins, but a few guys from our project have made their way over here and are currently working on their first skin, so might as well have all the info just in case it gets used down the road.

    A FEW OTHER NOTES:

    1. crcindex.txt = This was a file that had the crc's for all of the games. It was a necessary part of getting the synopsis files running quickly on another system. It should just be avoided by the program.

    2. skipped.txt = I'm not sure what this file was for, but it should be avoided by the program too.

    3. I'd like to automatically add a line in these XML files that would add a Marquee image. For example we'll use the Atari 7800 game "Ace of Aces".

    <marquee>/home/pi/RetroPie/Media/A7800/Artwork/Marquee/Ace Of Aces.jpg</marquee>

    1. I noticed that all "media" lines in the program are "/Media/A2600/". I know that you planned that at this stage. Just wanted to hilight that this is going to obviously be different from system to system and we have to decide whether we're actually searching for media in the program before making these media lines or just making them all like your program does now but we have to specify a different folder here per system at some point then.

    EDIT:
    I'll add other things I notice in this spot until you see the message and reply.

    I noticed on Ace of Aces.txt for the 7800 that it didn't carry over the "Genre" category.

    The .txt file had this line: "Genre: Flight Simulator - Action"

    The .xml file has this line: "<genre></genre>"

    Maybe it doesn't like the " - "?



  • @Used2BeRX You did quite extensive testing !
    The program uses original txt path & filename to write the xml: therefore if the original name includes "uncommon" characters it may generate the errors you got.
    It uses a specific VB streamwrite command so I have to check if, as it seems, special characters cannot be used and if there is a workaround.
    I'll come back to you on this when I have done specific tests on the files that actually generate errors.

    Concerning paths, as said, I'm going to add specific path fields the user can customize as needed.

    Will look also into the missing <genre> issue.



  • @UDb23 Yep. :) I want to be really kicking the tires on this and making it as awesome as I think it can be.

    I've got a question out to the guy who knows the most about our online synopsis system to see if there's an easy way to know for sure what fields are used for each system so I don't have to just open a bunch of games and try to find them. It looks as though the editor does not spit out empty fields, so it would be difficult to try to figure it out by hand. If I have to do that though, I will.

    I'm also going to spend some more time with this hopefully later today or by tomorrow to see if I have any more suggestions.

    Thanks again.



  • @Used2BeRX further testing on my side will take a couple of days; so no need to run to list all fields for you.



  • @UDb23 I have some free time the next few days so it's no problem. You're going to need that list eventually, so better to get it out of the way. The next few weeks are going to be a bit spotty for me after that, so I'll have less time to test this out, but it will be a top priority for me to get feedback to you ASAP so you can get back to it without waiting on me.

    EDIT:

    Looks like we're just going to make this easy (hopefully). I went into the synopsis system and from console to console the only fields they have for entering game info now are the following:

    Xtras Name: (Extras name in the system means the file name and extension we use for each game)
    Title: (Name of the game. The top line in all the TXT files)
    Original Title: (Only used for Hacks or Translated games)
    Alternate Title: (Only used for Hacks or Translated games)
    Release Year:
    Region:
    Genre:
    Gametype:
    Developer:
    Publisher:
    Players:
    Controller:
    Media:
    Hacked by:
    Translated by:
    Version: (Version of the hack/translation)

    I'd say just make sure the above fields are all accounted for, and leave in all of the fields you have currently. Any games they've added since they put the system online don't account for fields like musician or programmer, but it looks like if they were in there back when I was making this stuff by hand years ago that they've remained. No reason to take them out I guess.

    If any problems come up or if I notice things we can talk about that when they occur.



  • @Used2BeRX Thanks for the "items/tag" list; with this I can update the txt parsing function in the program to catch all these.
    I don't see the <genre> you mentioned previously; is it the same as <gametype> ?



  • @UDb23 Whoops. I forgot Genre. That's different than gametype. The gametype specifies if it's an official release or a hack or translation or prototype etc.

    Also, the "Xtras Name" won't be in any of the files. The txt file name IS the Xtras name 99.9% of the time. We're going to probably run into some problems down the road with this though since case sensitivity was not an issue on the other platform but will be here. Most of the time they're good, but for example a game like "Attack of the Mutant Camels.zip" might be "Attack Of The Mutant Camels.xxx" for one or more of the media pieces.



  • @Used2BeRX I can Camel Case (funny that you used Mutant Camels game as example!) the filenames of the xml if required. It's probably easier if you rename txt files and related media by using an utility like Bulk Rename to have all consistant camel cased names.



  • @UDb23 I thought Herb_Fergus was making a joke earlier in this thread or somewhere in another thread after I used that example. What does Camel Case mean?

    I suppose if need be I can do a Bulk Rename on everything. My initial intention was to be able to use the exact same packs as were originally created and have been floating around the internet for the last 7-8 years, but I've had to make drastic changes to stuff for a Pi Zero pack anyhow.

    What I'm really not looking forward to having to do is making romcenter datfiles with our naming conventions. I don't think that there is a way to make one file for each system that takes into account the subfolders. That would make life much easier, but I think I need to make a separate datfile for every sub-folder. Some of the systems like the NES have a crazy amount of nested folders.

    I guess it really doesn't matter about the case now. I could just make every single file all upper or lower case since the game name on the emu will show as whatever is in the gamelist.xml and look nice.

    When these sets were originally made, there were two things we had to deal with. None of the emulators on the other platform were set up initially to show anything other than the file name and extension (that was fixed years ago), and the other one was the filename limit was 42 characters including the extension.



  • @Used2BeRX Camel Case means having all first letters of words in a sentence in upper case;
    lower case: star wars: a new hope
    Camel Case:Star Wars: a New Hope
    So you can keep those rom names with the initial word capitals and make all the names consistant by applying "camel case" to all txt and media files using an utility like "Bulk rename".
    Also visual basic has a dedicated function to "camel case" a sentence.



  • @UDb23 Ah... well for the most part I've got it Camel Case already then. But your example brings up the problem with that. I would have named the rom "Star Wars - A New Hope", but maybe somebody helping me put the packs together might have named a piece or two of the artwork "Star Wars - a New Hope". Like I said, it doesn't happen very often, but it went completely overlooked since nothing was case sensitive.

    Too bad there isn't just a way to turn off case sensitivity. I know for sure everything matches up perfectly if that's not a factor.

    EDIT: I did finally take the time to re-size all of my artwork to around 240x180 for the Pi Zero I'm working on. I was having problem with larger images with how many I had. Having the gamelist.xml for just the 4 atari systems was enough to keep emulators from launching before. I reduced the VMM to only 50MB like somebody suggested in another thread and the smaller images seem to scroll a lot smoother on the list. Time will tell if I can use images with the Pi Zero once we start making the XML's for larger collections like the NES and Gameboy.

    I haven't re-"cased" anything yet. I'm kind of on a time frame to get this Pi back to my brother for a gift to his brother in law, so I don't want to be changing everything until I know one way or another whether he's going to get this Pi with artwork or not. I'll be getting a replacement to continue the work on when I give this one back though.



  • @UDb23 Case Sensitivity will not be an issue. I decided to make things easy and I just Bulk Renamed every file to all uppercase filenames and extensions. I haven't done that to the Atari systems that I've already made gamelist.xml files for already, but I'm sure I will once this program is working.

    Any luck ironing out any of those issues I found earlier?

    I don't know how hard it would be to do, but I was hoping that there could be a function in the program that would scan the roms directory and the sub-folders for the file match when it creates the line for where the rom exists. Every system I have comes with sub-folders that separate official titles from all sorts of hacks, translations, prototypes, homebrew and other categories. Putting them all in one folder would be a mess of a romlist that nobody would want to play, and having to go in and manually change almost 10,000 entries over all of the systems by hand would largely defeat the purpose of this program.



  • @Used2BeRX Just published new version of MetaX (.05). It's available in the same folder as previous.
    Changes:

    • Filename issues fixed (filenames for xml were based on game name inside the txt; now uses same filename as txt)

    • Genre issue fixed; is now detected and saved correctly

    • marquee tag added

    Next steps (coming soon):

    • custom media path (can be set by user)

    • convert all files in directory

    • create global gamelist.xml (converting all txt in folder)

    Your last request, to scan if actual rom exists and create xml only for those found, makes sense of course.
    The problem is related to names of the roms the user has; if they do not match exactly with the game names in the TXTs the "scan engine" would not find them correctly. Unless you're telling me that for these old console we can be almost sure roms have "standardized" names.

    Maybe @meleu can support us in this matter. He has written many really helpful scripts for retropie, including the installer for the overlays and game art.

    @meleu : what about adding a specific line in the info.txt like meta="mymetadata.xml" and providing the xml for that rom with complete metadata/tags generated with MetaX and @Used2BeRX txt data ?
    Would it then be possible for the your script to add (or replace) the game tags of that .xml into the gamelist.xml ?



  • @UDb23 Nice. I'll download it and re-test the *.txt files I tested last time and see if I get any errors this time. I won't test more than that for now and we'll just have to see if we get any other errors popping up when we start doing the global gamelist.xml.

    I can say for sure that at least with my sets the names of the roms and the synopsis.txt and any Xtras will be identical. I'm working on re-casing everything right now and I just got a new version of FlashFXP that has a function to compare file names and hide exact matches so I will be able to see if anything is not matching after the re-casing and fix those as well. My sets will be 100% match, that you can be sure of.

    If meleu has a way to do this where it doesn't have to be as perfect, that's great. It would allow people to not have to be so anal about doing things with their own sets. This is going to be a great tool, and that would make it more universal for other people.

    I will also get together an exact layout of my media structure for you to look over so you can see where media is going to be. It's going to be long as hell, so I'll put it in a "code window" here and you can copy and paste it into Notepad to see the structure for yourself.



  • @UDb23 ERROR REPORT:
    Atari 7800 (All 90 Games). Previous Errors: 13. Current Errors 7.
    NES (First 25 Games). Previous Errors: 10. Current Errors 9.

    Atari 7800 Errors:
    Centipede.txt
    FailSafe.txt
    Fatal Run.txt
    Meltdown.txt
    Ms Pac-Man.txt
    Pac-Man Collection.txt
    Tower Toppler.txt

    NES Errors:
    10-Yard Fight.txt
    1942.txt
    1943.txt
    720.txt
    75 Bingo.txt
    8 Eyes.txt
    A Boy and His Blob.txt
    A Nightmare On Elm Street.txt
    Abadox.txt

    I'm going to put any potential errors I see in the original text files underneath each game here for you to look at and decide if you think that's the problem.

    Atari7800:
    Centipede.txt
    Genre: Arcade - Shooter - Fixed
    Players: 1 or 2 VS; CO-OP
    "Hints:" in <desc>

    FailSafe.txt
    Genre: Action, Tank

    • There are ', /, -, :, ., ! , " characters in the <desc> on this file. I won't list this as an issue for any other files, but it should be expected that these characters will show up in the descriptions of a lot of other games in case they're currently a problem.

    Fatal Run.txt
    Genre: Racing - Cars

    • along with special characters in <desc> mentioned above, this one has "#" in it as well.

    Meltdown.txt
    Controller: XG-1 Lightgun

    Ms Pac-Man.txt
    [First line title]: Ms. Pac-Man
    Numbers "X." in <desc>

    Pac-Man Collection.txt
    [First line title]: Pac-Man Collection!

    Tower Toppler.txt
    Possibly one or more of the special characters mentioned above in <desc>.

    NES:
    10-Yard Fight.txt
    Genre: Sports - Football
    Special Characters in <desc>?

    1942.txt
    Genre: Shoot 'em Up - Vertical
    Players: 1 or 2 CO-OP
    Special Characters in <desc>?

    1943.txt
    Genre: Shoot 'em Up - Vertical
    Players: 1 or 2 CO-OP
    Special Characters in <desc>?

    720.txt
    Genre: Sports - X-Games
    Special Characters in <desc>?

    75 Bingo.txt
    Region: Unlicensed (Asia)
    Special Characters in <desc>?

    8 Eyes.txt
    Players: 1 or 2 CO-OP
    Special Characters in <desc>? There is also "+" and "=" in this one.

    A Boy and His Blob.txt
    [First line title]: A Boy and His Blob: Trouble on Blobolonia
    Special Characters in <desc>? There is also "(" and ")" in this one.

    A Nightmare On Elm Street.txt
    Players: 1 to 4 CO-OP
    Special Characters in <desc>?

    Abadox.txt
    Genre: Shoot 'em Up - Horizontal
    Special Characters in <desc>?

    Knocked out about half of the errors with this fix. Hopefully the stuff I pointed out will take care of the rest of them?

    I'll get to work on getting the file system put together for you.


 

Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.