RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Making premade gamelist.xml with xtra media collections

    Scheduled Pinned Locked Moved Ideas and Development
    gamelist.xmlpremade scrapeeasy config
    448 Posts 16 Posters 153.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Used2BeRXU
      Used2BeRX @UDb23
      last edited by Used2BeRX

      @UDb23 Well..... that was horrible. Just the 1st part took me about 5 1/2 hours, so hopefully that's the last system I'm doing that for with your help man.

      https://www.sendspace.com/file/modd8g

      I'll probably still be editing that a bit, but I did verify that all the boxart works with it. There were about 4 lines in there that I had to change because the artwork was a different case than the rom. Not too bad when it was over 700 entries. I didn't check any of the other artwork or videos yet.

      Let me know what questions you've got.

      1 Reply Last reply Reply Quote 0
      • UDb23U
        UDb23 @Used2BeRX
        last edited by

        @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

        I'm really OCD-ing out here and I'm manually doing things right now

        You shouldn't invest time in manually editing those files ! ;-)

        Excel 2007 is fine to run my VBA code.
        Let's proceed in steps with development:

        1. Create program to parse existing txt files and generate separate "per game" xml files + global gamelist.xml.
          Parsing requires source txt to have a defined structure.
          Looking at those TXTs the first part is ok (assigning specific fields/tags value).
          Then, after the"----------" there's the description. Last line always contains info source reference. Is that correct ? if so I will use these "rules" to parse the files.

        2. Add customizable folder paths (preference settings) that will be used during xml generation. In this way people can have supporting files in the folder they prefer.

        If I understood correctly for each game the rom name has to be added as it is not contained in the TXT file. All extras (boxart, video etc) are then named the same as the rom but, of course, with different extension and path.
        This can be generated too by the program.

        For existing XMLs (work you have already done) the program would read-in just the rom names and combine them with corresponding TXT data and generate XML.

        I suppose some games will not have some of the extras, so generating ALL EXTRA tags would actually create some "broken links" to missing data. Not sure if that would make ES "unhappy" and generate an error or if ES just ignores this.

        Concerning the case issue the program could simply generate all lower or all upper case.
        In this way you just need to "case convert" all actual extra files accordingly. I'm pretty sure free windows utilities exist to do this.

        I'll be away over the next days but can start working on it this weekend.

        Used2BeRXU 1 Reply Last reply Reply Quote 0
        • Used2BeRXU
          Used2BeRX @UDb23
          last edited by

          @UDb23 Hey man. Thanks again.

          ES doesn't freak about links to files that aren't there. Many of the games don't have all of the artwork, especially the title shots on systems like the 2600 that didn't have many titles. I don't actually put any of the other media on the Pi Zero I'm working on either, so there have been no errors popping up for missing videos or manuals, etc. Now is there memory being burned up using tags with no file? That's above my pay grade. :)

          Haha! Yeah, I know I shouldn't be manually making them. I thought that the first part was a lot quicker than it actually was. I couldn't believe I put 5 1/2 hours more into the 2600 until I checked the time when I was done.

          As for your "steps with development"

          1. A per game xml is great, even if we can't use it now. Hopefully somebody could change the code of ES somehow to make use of it. We had to break the synopsis down from one master file to individual games because on large systems the file was insanely huge and was causing memory issues. Hell... if you look at my 2600 xml it's almost 900kb and that's without any of the game data in them. My guess is just for 700 games that file is easily going to be 3MB when its done.

          Not every game is going to have every tag. My original synopsis file years ago had every tag for every game, but they made an online editor that the community could use and the individual game files that it creates spit out only the fields that had info in them. Sadly, the people who developed it are gone now so I can't change that.

          The last line "usually" contains an info source reference. I made that a big rule of mine when I was creating the system that if we took info from a website to have a link to the page it came from. I can't say for sure it was always followed, and sometimes there was no info found online and people wrote up their own stuff with no reference link.

          1. customizable folder paths would be good. Not only for me, but for people using this for their own stuff in the future too. For me it's good because the media/roms folders are going to be different from system to system and it would not require you to build alternate versions for different systems or me to try to go in and change things and probably break it.

          Yes, the rom name is not in the TXT file. The way our system worked was that it would find the matching "filename.txt" when you pulled up the game information.

          The problem with the Case Issue is that some games are named, for example "Attack of the Mutant Camels", but the artwork is named "Attack Of The Mutant Camels". It's usually not an either/or situation. If we can't correct for that, I think I'd rather just have it use the exact case of the game and I'll check that everything works by hand afterward and change the file accordingly. If possible, I want to maintain the existing file structure as is, so people don't have to re-download the entire thing or have two copies of it. I might change my mind on this part. I'll have to think about it while you're gone.

          Thanks for the help. Looking forward to seeing it in action. Let me know if I can clear anything else up.

          UDb23U 1 Reply Last reply Reply Quote 0
          • herb_fargusH
            herb_fargus administrators
            last edited by

            Camel casing articles is a very simple process

            If you read the documentation it will answer 99% of your questions: https://retropie.org.uk/docs/

            Also if you want a solution to your problems read this first: https://retropie.org.uk/forum/topic/3/read-this-first

            UDb23U 1 Reply Last reply Reply Quote 0
            • UDb23U
              UDb23 @herb_fargus
              last edited by

              @herb_fargus said in Making premade gamelist.xml with xtra media collections:

              Camel casing

              Didn't know that it's called that way. :-))

              1 Reply Last reply Reply Quote 0
              • UDb23U
                UDb23 @Used2BeRX
                last edited by

                @Used2BeRX Everything clear so far. Will keep you updated.

                Used2BeRXU 2 Replies Last reply Reply Quote 0
                • Used2BeRXU
                  Used2BeRX @UDb23
                  last edited by

                  @UDb23 Sweet.

                  There was a little miscommunication with the other guy that was helping me work something out and he created a python script for Linux that will make individual "game".xml files with tags for all of the game info and put the 10 file tags above it. It actually looks for the media if it's in the proper folders so it puts the proper extension too. Pretty cool, but I'm a Linux dummy and the only way I could really use it is in the Linux console with the Pi. The problem is I've got the Pi Zero with a small SD card so I couldn't really make use of it that way.

                  I've got really high hopes for what you're talking about doing here. Maybe if we iron out all of the kinks it could be actually turned into that executable you had mentioned as well. I'm thinking that what you're doing could be a valuable tool for anybody with Extra media for their games, even if they're not using my setup.

                  I've taken your advice and stepped away from this for a while, until you come back with something. I probably could have gotten the first half of the NES gamelist.xml done yesterday if I worked on it all day, but I'd be kicking myself for it once I had your program to work with I'm sure.

                  Let me know! Thanks.

                  1 Reply Last reply Reply Quote 0
                  • cafarellidigitalC
                    cafarellidigital
                    last edited by

                    I just wanted to show my support for this project, for all of the work that you've done so far, and I look forward to see where it goes in the future!

                    Console Stripe Video Splashscreen | Mushberry Splatter Splashscreen

                    Used2BeRXU 1 Reply Last reply Reply Quote 0
                    • Used2BeRXU
                      Used2BeRX @cafarellidigital
                      last edited by

                      @cafarellidigital Thanks man. I think this is going to be pretty sweet when it's finished.

                      1 Reply Last reply Reply Quote 0
                      • Used2BeRXU
                        Used2BeRX @UDb23
                        last edited by

                        @UDb23 hey man. Was just wondering if you had any time to work on this at all? I'm pretty busy with some other stuff at the moment, so I'm not rushing you or anything. Just don't want to lose touch with the only guy who seems to have a plan on how to do this.

                        UDb23U 1 Reply Last reply Reply Quote 0
                        • UDb23U
                          UDb23 @Used2BeRX
                          last edited by

                          @Used2BeRX Don't worry. ;-)
                          As promised I started working on it. In the end I decided to go directly for making an executable program (no need for excel..).
                          Downloaded and installed latest Visual Studio community 2017; took me some time to get acquainted with it.
                          So far I made a "skeleton" main program with functions to access file system and to do string manipulation. Tomorrow I hope to find time to write the code for the "conversion" from txt to xml (should be easy at this point).

                          Used2BeRXU 2 Replies Last reply Reply Quote 0
                          • Used2BeRXU
                            Used2BeRX @UDb23
                            last edited by

                            @UDb23 Sweet. Just let me know if you need any more input before making the first compile. Otherwise I should be around to take it for a spin whenever you've got it ready.

                            1 Reply Last reply Reply Quote 0
                            • Used2BeRXU
                              Used2BeRX @UDb23
                              last edited by

                              @UDb23 Since I haven't heard back from you yet I figure you're still putting it together. I was hoping you could add a field in the first 10 <marquee></marquee> I don't have any marquee images myself, but some guys I know that are making a skin are using that field and making marquee style images by cropping from title screens.

                              UDb23U 1 Reply Last reply Reply Quote 0
                              • UDb23U
                                UDb23 @Used2BeRX
                                last edited by

                                @Used2BeRX Sure I can add this additional tag.
                                Program development is progressing and I'll provide you a first working version over the weekend.

                                Used2BeRXU 1 Reply Last reply Reply Quote 0
                                • Used2BeRXU
                                  Used2BeRX @UDb23
                                  last edited by

                                  @UDb23 Nice.

                                  I'm going to ask around here and see if I can get some answers on memory issues. I actually "broke" the Pi Zero with just the hand made XML files for only the 4 Atari systems. EmulationStation was taking 2-3 minutes to load and some emulators stopped functioning altogether. I'm just wondering if loading pointers to data that doesn't exist is draining memory resources or not. And I'm also wondering if every single picture you load is draining memory at one time. Seems terribly inefficient to me. I'll be able to test our work here one system at a time, but then I have to rename the XML file so it doesn't get loaded when I go on to the next system.

                                  Anyways, maybe down the road there can be options made in your program to expand or limit exactly what fields the program adds to the XML file depending on what I find out about how things are loaded.

                                  UDb23U 1 Reply Last reply Reply Quote 0
                                  • UDb23U
                                    UDb23 @Used2BeRX
                                    last edited by

                                    @Used2BeRX First beta of the Application to convert your .txt to .xml is ready.
                                    You can find it here.
                                    Just unzip the MetaX.zip and launch Setup.exe.

                                    Use the "Browse" button in the App to select the folder that contains the txt files.
                                    Currently only one by one conversion works: select the file and click "convert" button.

                                    I found that the game metadata source txt files are not consistent meaning some info may or may not be included; also same info (e.g. "developer" can be in different line in one txt and in another line in a different txt).

                                    The application tries to find the specific "tags" regardless of line order; it also identifies the "description" by selecting the lines after the line with "________".

                                    Could you let me know the complete list of "items/tags" the App should:

                                    • look for in the source txt files
                                    • write to the .xml files (on top of items found in the .txt)

                                    Just want to first make sure everything is converted correctly as single files, and then implement "convert all" and "create global gamelist.xml" functionality.

                                    App requires .net libraries installed (created in VB with Visual Studio community 2017); I suppose it will ask during installation (can't test on another PC).

                                    note: if you convert the same txt multiple times, it will append lines to already existing corresponding .xml (we could call it a "known bug" ;-) ).

                                    Note 2: App does not do anything else as converting txt to xml so it should be totally harmless. In any case keep a backup of your original txt files.

                                    Used2BeRXU G 3 Replies Last reply Reply Quote 1
                                    • Used2BeRXU
                                      Used2BeRX @UDb23
                                      last edited by

                                      @UDb23 Awesome man. Thanks.

                                      I downloaded it, but haven't had time to use it. I should have some feedback for you tomorrow or Tuesday night at the latest. Thanks so much for this again.

                                      1 Reply Last reply Reply Quote 0
                                      • G
                                        genericuserdude @UDb23
                                        last edited by genericuserdude

                                        @UDb23 I somehow got banned earlier this morning with no reason. I think it was a mistake. Just wanted to post here that I'm working on trying to get to the bottom of this. If somebody has it out for me for some reason and I get banned with this ID as well, please reach out to me as I think we're on to something really good here and I don't have any way of keeping contact with you right now. I couldn't figure out a way to send a PM, so here's the hail mary.

                                        Please contact me on this online document if the ban doesn't get lifted until we come up with something better: https://docs.google.com/document/d/18cfXi3eO7Gj1piWbzans8zS5CQ2ATaUrxeM0jmdzal8/edit?usp=sharing

                                        UDb23U 1 Reply Last reply Reply Quote 0
                                        • UDb23U
                                          UDb23 @genericuserdude
                                          last edited by

                                          @genericuserdude ok.
                                          Maybe you should ask @Global-Moderators why this happened.

                                          G 1 Reply Last reply Reply Quote 0
                                          • G
                                            genericuserdude @UDb23
                                            last edited by

                                            @UDb23 said in Making premade gamelist.xml with xtra media collections:

                                            @Global-Moderators

                                            Okay. At least you and the other guy I have been working on another project has seen the link. Please save it in case I get banned again and they remove the posts so we can continue our work.

                                            @Global-Moderators Could somebody please tell me what is going on with the ban?

                                            mediamogulM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.