RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Making premade gamelist.xml with xtra media collections

    Scheduled Pinned Locked Moved Ideas and Development
    gamelist.xmlpremade scrapeeasy config
    448 Posts 16 Posters 163.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Used2BeRXU
      Used2BeRX @meleu
      last edited by Used2BeRX

      @meleu I forgot that I can't use that now since I cased everything in CAPS :)

      I'm having a problem though. I've removed everything except for the first entry for "3 Ninjas Kick Back". I'm positive I have the path and image correct, but it won't display in EmulationStation.

      I've never done a gamelist.xml for Genesis. Is there someplace special I have to have the gamelist.xml? I altered EmulationStation to display the "Sega Genesis" artwork instead of the Megadrive artwork. I'm not sure if that changes anything.

      EDIT: I FIGURED IT OUT!!!!

      Not sure if you are aware of this or not... actually, I'm not sure if it's because I changed the artwork or not...

      I had to create a shortcut called "genesis" in /opt/retropie/configs/all/emulationstation/gamelists that points to /opt/retropie/configs/all/emulationstation/gamelists/megadrive/

      I can't wait to try it out on my end. This is going to be great. Thanks again meleu. :)

      1 Reply Last reply Reply Quote 0
      • meleuM
        meleu @Used2BeRX
        last edited by

        @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

        I decided to try it out and none of them showed up. I found a bug. :(

        Man, that file is not supposed to be used on your system. I'm just showing you that my script is able to generate that kind of file. Those paths you see in that file were automatically detected by the script on my system.

        You have to run the script with your synopsis files on your own system in order to get a nice gamelist.xml with the right paths.

        I have some ideas for optimizations, but I really need to sleep now.

        See you later.

        • Useful topics
        • joystick-selection tool
        • rpie-art tool
        • achievements I made
        Used2BeRXU 2 Replies Last reply Reply Quote 1
        • Used2BeRXU
          Used2BeRX @meleu
          last edited by

          @meleu Yeah. I figured that out. The only thing that I couldn't figure was why I couldn't get anything to work at all and with genesis you need to have that shortcut to the megadrive folder and then the gamelist.xml works.

          Have a good sleep man. You've had a busy day. :)

          I'm going to keep working on getting everything prepared for this. I've got a lot of file comparing to do now since I changed so much in the media folders to make it comply to the retropie file system.

          1 Reply Last reply Reply Quote 0
          • Used2BeRXU
            Used2BeRX @meleu
            last edited by

            @meleu It took me almost a full day, but I got every single piece where it needs to be and everything is properly named. I also did this for the pack of things that cannot be used currently and/or cannot be used on a Pi Zero as well. (The only exception is CD based games which I think will bring their own challenges when we get to that point).

            Nemesis is working on a way to auto convert names of videos so we can have upgraded MP4 versions above the WMV8 ones we have now. Let me know when you've got something to test out.

            I had another idea about a 3rd option when running the script that would probably benefit a Pi Zero. I was thinking maybe a 3rd option would find and include ONLY the first line name from the game.txt files, the rom location/name and the "Box Front" location/name for every game, and it would leave out all other data, even the synopsis data.

            We may run into problems on a Pi Zero with all of this data in a large amount of gamelist.xml files, and perhaps with the smaller artwork if everything else was stripped out and a skin that didn't display metadata were used, you could still have a boxart or action screen displayed for every game with the improved names that are in the current game.txt files.

            meleuM 1 Reply Last reply Reply Quote 0
            • meleuM
              meleu @Used2BeRX
              last edited by

              @Used2BeRX The script already does what you're suggesting (create a gamelist.xml only with metadata useful for ES).

              Are you able to test the script?

              Please, see if @NeMesiS or other Linux bud from your crew can test my tool and give feedback/suggestions. No need to analyze the code if it seems obscure, just test it with all the options I provided (look the --help) and see if I can improve something.

              • Useful topics
              • joystick-selection tool
              • rpie-art tool
              • achievements I made
              Used2BeRXU 1 Reply Last reply Reply Quote 0
              • Used2BeRXU
                Used2BeRX @meleu
                last edited by Used2BeRX

                @meleu I wasn't aware that you had uploaded the new one. I thought you were still testing things and trying to tweak it so it didn't take so long to put a list together.

                I was talking about making an option that is completely bare bones. One that only has the following 3 things:

                1. First-Line game name from the game.txt
                2. Path to rom
                3. Path to Box Front for <image>

                EDIT: I just checked here: https://github.com/meleu/Used2BeTXT

                Unless you uploaded the script somewhere else, that's still the one you uploaded a few days ago. Can you tell me where the newest script is so I can test it out?

                meleuM 1 Reply Last reply Reply Quote 0
                • meleuM
                  meleu @Used2BeRX
                  last edited by

                  @Used2BeRX

                  @meleu I wasn't aware that you had uploaded the new one.

                  Ooops! My bad, sorry. I thought I uploaded it... I'll do it when I get home (in ~10 hours).

                  And then you will be able to get the latest version using the --update option.

                  I was talking about making an option that is completely bare bones. One that only has the following 3 things:

                  1. First-Line game name from the game.txt
                  2. Path to rom
                  3. Path to Box Front for <image>

                  Yep. The new (and not yet uploaded) script does it plus description in <desc>. Are you suggesting to add an option to omit the <desc>?

                  • Useful topics
                  • joystick-selection tool
                  • rpie-art tool
                  • achievements I made
                  Used2BeRXU 1 Reply Last reply Reply Quote 0
                  • Used2BeRXU
                    Used2BeRX @meleu
                    last edited by

                    @meleu Sounds good. I'll test it out as soon as I can.

                    Are you suggesting to add an option to omit the <desc>?

                    Yes. The Pi Zero is very low on performance, and some of those <desc> are reaaaaaaaaly long and make the gamelist.xml files really big.

                    I'm hoping that if we had a script that only did those 3 things and nothing else that combined with the shrunk jpg box images that I made that I can actually use this with all the games on the Pi Zero without crashing it.

                    meleuM 1 Reply Last reply Reply Quote 0
                    • meleuM
                      meleu @Used2BeRX
                      last edited by meleu

                      @Used2BeRX

                      I'm hoping that if we had a script that only did those 3 things and nothing else that combined with the shrunk jpg box images that I made that I can actually use this with all the games on the Pi Zero without crashing it.

                      Looks like you're very very proud of the images you have. Otherwise I would suggest the use of the Sseph's scraper. It's a very powerful tool with many options, as you can see in its wiki

                      EDIT
                      You can even run it on Windows: https://github.com/sselph/scraper/releases

                      EDIT 2
                      It has an option similar to what you talked about some posts above:

                      -gdb_img="b": Comma seperated order to prefer images, s=snapshot, b=boxart, f=fanart, a=banner, l=logo.
                      
                      • Useful topics
                      • joystick-selection tool
                      • rpie-art tool
                      • achievements I made
                      Used2BeRXU 1 Reply Last reply Reply Quote 0
                      • Used2BeRXU
                        Used2BeRX @meleu
                        last edited by Used2BeRX

                        @meleu I tried sselph's scraper and was never able to get it to work.

                        Our images are superior I'm guessing. I touched up over a thousand of them by hand myself and for every game that didn't have a boxart like prototypes and pirates and homebrew we had a talented graphic artist that made boxarts for them.

                        lol... why would I use a scraper after we've done all of this work?

                        meleuM 1 Reply Last reply Reply Quote 0
                        • meleuM
                          meleu @Used2BeRX
                          last edited by

                          @Used2BeRX

                          lol... why would I use a scraper after we've done all of this work?

                          haha. I'm doing this because I take it as a diversion. If a scraper can give you the results you want, please, use it! :-)

                          I'm a kind of data manipulation enthusiast. I love shell scripting, regular expressions, parsing files... With your request in this thread I saw a possiblity to improve my hability to parse files and manage xml files and this is already happening :-)

                          • Useful topics
                          • joystick-selection tool
                          • rpie-art tool
                          • achievements I made
                          Used2BeRXU 1 Reply Last reply Reply Quote 0
                          • Used2BeRXU
                            Used2BeRX @meleu
                            last edited by

                            @meleu Nah man. I got everything I want already if we get the code working.

                            Have you had a chance to upload that new script, or were you still doing some tweaks on it before I test it out?

                            meleuM 1 Reply Last reply Reply Quote 0
                            • meleuM
                              meleu @Used2BeRX
                              last edited by

                              @Used2BeRX

                              I've just uploaded the script. You can get it with the --update option.

                              I didn't implemented the option to create the gamelist.xml with no <desc>. And I'm going to be busy for a week. Will only have a chance to work on the script next week.

                              But you can put your feedback/suggestions here.

                              Cheers!

                              • Useful topics
                              • joystick-selection tool
                              • rpie-art tool
                              • achievements I made
                              Used2BeRXU 1 Reply Last reply Reply Quote 0
                              • Used2BeRXU
                                Used2BeRX @meleu
                                last edited by Used2BeRX

                                @meleu Got it. It works with the following command:

                                ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.*

                                Seems to be going through the files one by one and slowly adding them. It says "data has been added to "atari2600_gamelist.xml" after each file. Looks like it takes about 10 seconds for each entry.

                                I'm able to open atari2600_gamelist.xml right now and it looks to be working right for the roms.

                                It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

                                I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

                                meleuM 1 Reply Last reply Reply Quote 0
                                • meleuM
                                  meleu @Used2BeRX
                                  last edited by

                                  @Used2BeRX

                                  It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

                                  Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

                                  I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

                                  I noticed some fails but didn't fixed yet. Examples of games that the current script version is unable to find

                                  • mega man - wily wars (megadrive)
                                  • mickey mouse - fantasia (megadrive)

                                  Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

                                  • Useful topics
                                  • joystick-selection tool
                                  • rpie-art tool
                                  • achievements I made
                                  Used2BeRXU 2 Replies Last reply Reply Quote 0
                                  • Used2BeRXU
                                    Used2BeRX @meleu
                                    last edited by

                                    @meleu The script is still going after about 2 hours for the Atari 2600 (about 750 entries). I think it will have taken about 2 1/2 to 3 hours once it's done.

                                    Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

                                    Cool. Just making sure that the script wasn't designed to work on the folders yet.

                                    Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

                                    I noticed that the <xtrasname> field was added, but it's not being populated atm. That should be the synopsis.txt file name. If the script searched off of that it should be able to find things a lot quicker since every single piece of media and the rom itself is an exact match of that name.

                                    meleuM 1 Reply Last reply Reply Quote 0
                                    • Used2BeRXU
                                      Used2BeRX @meleu
                                      last edited by

                                      @meleu I've just checked it and the Script on the Atari 2600 had 108 erroneous entries out of 724.

                                      Almost 100% of these would be avoided if two things were done:

                                      1. Use only the SYNOPSIS.TXT name (<xtrasname>) to find the media.
                                      2. Fix the issue where "&" is turned into "&"

                                      Here is the list of the errors:

                                      M*A*S*H* entry wasn't even created.  (Filename MASH.TXT)
                                      
                                      TRACK AND FIELD ------ & character showed up as "&amp;" so title line was "Track &amp; Field"  (This happens every time a & was used in any part of the synopsis.txt)
                                      
                                      STEEPLE CHASE and STEEPLECHASE ------ Both of these fought for media/rom STEEPLECHASE instead of their separate ones.  US version STEEPLECHASE shows properly, EU version STEEPLE CHASE does not.
                                      
                                      TARG and TARG (ALT) both are called just "Targ" on the first line.  Only an entry for TARG.TXT was made and TARG (ALT).TXT was skipped.
                                      
                                      Specific errors:
                                      
                                      (1_1) US
                                      ATLANTIS I / Atlantis
                                      BUCK RODGERS / Buck Rogers: Planet of Zoom
                                      BUMP 'N JUMP / Bump 'n' Jump
                                      CIRCUS / Circus Atari
                                      COMBAT I / Combat
                                      DEFENDER I / Defender
                                      ENCOUNTER AT L5 / Encounter at L-5
                                      ET - THE EXTRA TERRESTRIAL / E.T.: The Extra Terrestrial
                                      FROGGER I / Frogger
                                      FROGGER II - THREEDEEP! / Frogger II: Threedeep!
                                      GHOSTBUSTERS I / Ghostbusters
                                      GI JOE - COBRA STRIKE / G.I. Joe: Cobra Strike
                                      HERO / H.E.R.O.
                                      HOME RUN BASEBALL / Home Run
                                      JAMES BOND 007 / James Bond: 007
                                      JOURNEY - ESCAPE / Journey: Escape
                                      JR PAC-MAN / Jr. Pac-Man
                                      KOOL AID MAN / Kool-Aid Man
                                      KUNG FU MASTER / Kung-Fu Master
                                      MAD / M.A.D.
                                      MARIO BROS / Mario Bros.
                                      MASTERS OF THE UNIVERSE / Masters of the Universe: The Power of He-Man
                                      MINER 2049ER - VOLUME II / Miner 2049er: Volume II
                                      MONTEZUMA'S REVENGE / Montezuma's Revenge featuring Panama Joe
                                      MR DO! / Mr. Do!
                                      MR DO! / Mr. Do!'s Castle
                                      MS PAC-MAN / Ms. Pac-Man
                                      MUSIC MACHINE / The Music Machine
                                      PITFALL! / Pitfall!
                                      PITFALL! II LOST CAVERNS / Pitfall! II: Lost Caverns
                                      QUICK STEP! / Quick Step
                                      RAMPAGE! / Rampage
                                      RIVER RAID I / River Raid
                                      SMURFS - RESCUE IN GARGAMEL'S CASTLE / Smurfs: Rescue in Gargamel's Castle
                                      SNEEK 'N PEEK / Sneak 'n Peak
                                      SPACE SHUTTLE - JOURNEY INTO SPACE / Space Shuttle: A Journey Into Space
                                      SPACEMASTER X-7 / Space Master X-7
                                      SPIDER MAZE / Spider-Maze
                                      STAR GUNNER / StarGunner
                                      STAR SHIP - OUTER SPACE / Star Ship
                                      STAR TREK - STRAGEGIC OPS SIMULATOR / Star Trek: Strategic Operations Simulator
                                      STAR WARS - DEATH STAR BATTLE / Star Wars: Return of the Jedi - Death Star Battle
                                      STAR WARS - JEDI ARENA / Star Wars: Jedi Arena
                                      STAR WARS - THE ARCADE GAME / Star Wars: The Arcade Game
                                      STAR WARS - THE EMPIRE STRIKES BACK / Star Wars: The Empire Strikes Back
                                      STRAWBERRRY SHORTCAKE - MUSICAL MATCH / Strawberry Shortcake: Musical Match-Ups
                                      SWORDFIGHT / Sword Fight
                                      SWORDQUEST - EARTHWORLD / SwordQuest: Earthworld
                                      SWORDQUEST - FIREWORLD / SwordQuest: Fireworld
                                      SWORDQUEST - WATERWORLD / SwordQuest: Waterworld
                                      TEXAS CHAINSAW MASSACRE / The Texas Chainsaw Massacre
                                      TOMCAT - THE F-14 FLIGHT SIMULATOR / Tomcat: The F-14 Flight Simulator
                                      TRON - DEADLY DISCS / Tron: Deadly Discs
                                      VENTURE I / Venture
                                      VIDEO CUBE / Atari Video Cube
                                      YAR'S REVENGE / Yars' Revenge
                                      
                                      (1_2) Europe (NTSC Conversions):
                                      STEEPLE CHASE / **** see specific notes
                                      SURFER'S PARADISE / Surfer's Paradise: But Danger Below!
                                      
                                      (1_3) Europe (Non-NTSC Conversions):
                                      MISSION 3000 AD / Mission 3000 A. D.
                                      MR POSTMAN / Mr. Postman
                                      SEA MONSTER - SEE MONSTER / Sea Monster
                                      
                                      (1_5) Cassettes
                                      FROGGER I (ALT) / The Official Frogger
                                      
                                      (1_7) Special Releases
                                      PEPSI INVADERS / Pepsi Invaders: Coke Wins!
                                      
                                      (2_1) Homebrew
                                      AMIGA BOING / Amiga Boing 2.0
                                      ANGRY VIDEO GAME NERD KO BOXING / Angry Video Game Nerd K.O. Boxing
                                      CRAZY VALET / Crazy Valet CGE
                                      EDTRIS 2600 / EDTris
                                      GINGERBREAD MAN / Gingerbread Man: One Tough Cookie
                                      INCOMING / Incoming!
                                      INVPLUS / INV+
                                      KO CRUISER / K.O. Cruiser
                                      LORD OF THE RINGS - FELLOWSHIP OT RING / Lord of the Rings: Fellowship of the Rings
                                      MERLIN'S WALLS / Merlin's Walls: Standard Edition
                                      METROID PROOF DEMO / Metroid (Proof Demo)
                                      PHANTOM TWO / Phantom II &amp; Pirate (adds that bad "&amp;")
                                      REINDEER RESCUE / 2005 AtariAge Holiday Cart: Reindeer Rescue
                                      SEAWOLF / Sea Wolf
                                      STARFIRE / Star Fire
                                      THRUST PLUS - PLATINUM / Thrust Plus: Platinum
                                      TOYSHOP TROUBLE / 2006: AtariAge Holiday Cart: Toyshop Trouble
                                      
                                      (2_2) Prototypes
                                      A-TEAM / The A-Team
                                      BALTHAZAR (AKA BABYLON 5) / Balthazar
                                      CAREBEARS / Care Bears
                                      COMBAT II / Combat Two
                                      DUKES OF HAZZARD [STUNT CYCLE VERSION] / Dukes of Hazzard (Stunt Cycle Version)
                                      ENTITY / The Entity
                                      LORD OF THE RINGS - RIVENDELL / Lord of the Rings: Journey to Rivendell
                                      MCDONALD'S / McDonalds
                                      RED VS BLUE / Red vs. Blue
                                      REVENGE OF THE JEDI - GAME I / Star Wars: Revenge of the Jedi: Game I - Ewok Adventure
                                      ROCKY & BULLWINKLE / Rockey &amp; Bullwinkle (adds that bad "&amp;")
                                      SNOW WHITE / Disney's Snow White
                                      SNOW WHITE (VERSION 2) / Disney's Snow White (Later Prototype)
                                      WINGS / Wings [6-3-1983] (NTSC Version)
                                      WINGS (ALT) / Wings [10-01-1983] (PAL Version)
                                      
                                      (2_3) Hacks
                                      ADVENTURE (NEW GRAPHICS) / Adventure ( New Graphics Hack)
                                      BATTLEZONE TC (2 JOYSTICK HACK) / Battlezone TC
                                      BERZERK - RENEGADE / Berzerk: Renegade
                                      MR PAC-MAN / Mr. Pac-Man
                                      MS HACK / Ms. Hack
                                      PAC-MAN (MS PAC-MAN HACK) / Pac-Man Hack
                                      PITFALL! I - UNLIMITED / Pitfall! Unlimited
                                      RED VS BLUE 2600 WARS / Red vs. Blue: The 2600 Wars
                                      RETURN OF MARIO BROS / Return of Mario Bros.
                                      STAR WARS - THE BATTLE OF ALDERAN / Star Wars: The Battle of Alderan
                                      TARG (ALT) / NO SYNOPSIS (SEE NOTES)
                                      TOMB RAIDER (CLASSIC) / Tomb Raider: Mystery of the Scarlet Dragon (Classic Outfit)
                                      TOMB RAIDER (UNDERWORLD) / Tomb Raider: Mystery of the Scarlet Dragon (Underworld Outfit)
                                      VENTURE II / Venture II: The Abysmal Abyss
                                      
                                      (3) Adult Games
                                      TERRANCE & PHILLIP / Terrance &amp; Phillip: Asses of Fire (adds that bad "&amp;")
                                      TERRANCE & PHILLIP 2 / Terrance &amp; Phillip 2: Not Without My Anus (adds that bad "&amp;")
                                      X MAN / X-Man
                                      
                                      (4_5) Kid's Controller
                                      MONSTERCISE / Monstersice
                                      (5) Non-Working Games
                                      SWEAT! - THE DECATHALON GAME / Sweat!: The Decathalon Game
                                      `
                                      meleuM 1 Reply Last reply Reply Quote 0
                                      • meleuM
                                        meleu @Used2BeRX
                                        last edited by

                                        @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

                                        That should be the synopsis.txt file name. (...) every single piece of media and the rom itself is an exact match of that name.

                                        d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

                                        Can you --update and run it again?

                                        • Useful topics
                                        • joystick-selection tool
                                        • rpie-art tool
                                        • achievements I made
                                        1 Reply Last reply Reply Quote 0
                                        • meleuM
                                          meleu @Used2BeRX
                                          last edited by

                                          @Used2BeRX

                                          1. Fix the issue where "&" is turned into "&"

                                          It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as &amp; and ES works just fine with it. Test it and see.

                                          Side note: you have many bizarre game names in the Adult games folder... :-)

                                          • Useful topics
                                          • joystick-selection tool
                                          • rpie-art tool
                                          • achievements I made
                                          Used2BeRXU 1 Reply Last reply Reply Quote 0
                                          • Used2BeRXU
                                            Used2BeRX @meleu
                                            last edited by Used2BeRX

                                            @meleu

                                            d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

                                            Can you --update and run it again?

                                            Sure. I'm going to do it right now.

                                            It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as & and ES works just fine with it. Test it and see.

                                            Okay. It just wasn't finding the media because it was looking for media with "&.amp;" in it :) I'll verify it works right after this run.

                                            Side note: you have many bizarre game names in the Adult games folder... :-)

                                            If you think the names are weird, you should try playing them. ;)

                                            meleuM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.