RetroPie forum home
    • Recent
    • Tags
    • Popular
    • Home
    • Docs
    • Register
    • Login

    Making premade gamelist.xml with xtra media collections

    Scheduled Pinned Locked Moved Ideas and Development
    gamelist.xmlpremade scrapeeasy config
    448 Posts 16 Posters 152.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Used2BeRXU
      Used2BeRX @meleu
      last edited by Used2BeRX

      @meleu Got it. It works with the following command:

      ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.*

      Seems to be going through the files one by one and slowly adding them. It says "data has been added to "atari2600_gamelist.xml" after each file. Looks like it takes about 10 seconds for each entry.

      I'm able to open atari2600_gamelist.xml right now and it looks to be working right for the roms.

      It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

      I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

      meleuM 1 Reply Last reply Reply Quote 0
      • meleuM
        meleu @Used2BeRX
        last edited by

        @Used2BeRX

        It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

        Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

        I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

        I noticed some fails but didn't fixed yet. Examples of games that the current script version is unable to find

        • mega man - wily wars (megadrive)
        • mickey mouse - fantasia (megadrive)

        Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

        • Useful topics
        • joystick-selection tool
        • rpie-art tool
        • achievements I made
        Used2BeRXU 2 Replies Last reply Reply Quote 0
        • Used2BeRXU
          Used2BeRX @meleu
          last edited by

          @meleu The script is still going after about 2 hours for the Atari 2600 (about 750 entries). I think it will have taken about 2 1/2 to 3 hours once it's done.

          Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

          Cool. Just making sure that the script wasn't designed to work on the folders yet.

          Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

          I noticed that the <xtrasname> field was added, but it's not being populated atm. That should be the synopsis.txt file name. If the script searched off of that it should be able to find things a lot quicker since every single piece of media and the rom itself is an exact match of that name.

          meleuM 1 Reply Last reply Reply Quote 0
          • Used2BeRXU
            Used2BeRX @meleu
            last edited by

            @meleu I've just checked it and the Script on the Atari 2600 had 108 erroneous entries out of 724.

            Almost 100% of these would be avoided if two things were done:

            1. Use only the SYNOPSIS.TXT name (<xtrasname>) to find the media.
            2. Fix the issue where "&" is turned into "&"

            Here is the list of the errors:

            M*A*S*H* entry wasn't even created.  (Filename MASH.TXT)
            
            TRACK AND FIELD ------ & character showed up as "&amp;" so title line was "Track &amp; Field"  (This happens every time a & was used in any part of the synopsis.txt)
            
            STEEPLE CHASE and STEEPLECHASE ------ Both of these fought for media/rom STEEPLECHASE instead of their separate ones.  US version STEEPLECHASE shows properly, EU version STEEPLE CHASE does not.
            
            TARG and TARG (ALT) both are called just "Targ" on the first line.  Only an entry for TARG.TXT was made and TARG (ALT).TXT was skipped.
            
            Specific errors:
            
            (1_1) US
            ATLANTIS I / Atlantis
            BUCK RODGERS / Buck Rogers: Planet of Zoom
            BUMP 'N JUMP / Bump 'n' Jump
            CIRCUS / Circus Atari
            COMBAT I / Combat
            DEFENDER I / Defender
            ENCOUNTER AT L5 / Encounter at L-5
            ET - THE EXTRA TERRESTRIAL / E.T.: The Extra Terrestrial
            FROGGER I / Frogger
            FROGGER II - THREEDEEP! / Frogger II: Threedeep!
            GHOSTBUSTERS I / Ghostbusters
            GI JOE - COBRA STRIKE / G.I. Joe: Cobra Strike
            HERO / H.E.R.O.
            HOME RUN BASEBALL / Home Run
            JAMES BOND 007 / James Bond: 007
            JOURNEY - ESCAPE / Journey: Escape
            JR PAC-MAN / Jr. Pac-Man
            KOOL AID MAN / Kool-Aid Man
            KUNG FU MASTER / Kung-Fu Master
            MAD / M.A.D.
            MARIO BROS / Mario Bros.
            MASTERS OF THE UNIVERSE / Masters of the Universe: The Power of He-Man
            MINER 2049ER - VOLUME II / Miner 2049er: Volume II
            MONTEZUMA'S REVENGE / Montezuma's Revenge featuring Panama Joe
            MR DO! / Mr. Do!
            MR DO! / Mr. Do!'s Castle
            MS PAC-MAN / Ms. Pac-Man
            MUSIC MACHINE / The Music Machine
            PITFALL! / Pitfall!
            PITFALL! II LOST CAVERNS / Pitfall! II: Lost Caverns
            QUICK STEP! / Quick Step
            RAMPAGE! / Rampage
            RIVER RAID I / River Raid
            SMURFS - RESCUE IN GARGAMEL'S CASTLE / Smurfs: Rescue in Gargamel's Castle
            SNEEK 'N PEEK / Sneak 'n Peak
            SPACE SHUTTLE - JOURNEY INTO SPACE / Space Shuttle: A Journey Into Space
            SPACEMASTER X-7 / Space Master X-7
            SPIDER MAZE / Spider-Maze
            STAR GUNNER / StarGunner
            STAR SHIP - OUTER SPACE / Star Ship
            STAR TREK - STRAGEGIC OPS SIMULATOR / Star Trek: Strategic Operations Simulator
            STAR WARS - DEATH STAR BATTLE / Star Wars: Return of the Jedi - Death Star Battle
            STAR WARS - JEDI ARENA / Star Wars: Jedi Arena
            STAR WARS - THE ARCADE GAME / Star Wars: The Arcade Game
            STAR WARS - THE EMPIRE STRIKES BACK / Star Wars: The Empire Strikes Back
            STRAWBERRRY SHORTCAKE - MUSICAL MATCH / Strawberry Shortcake: Musical Match-Ups
            SWORDFIGHT / Sword Fight
            SWORDQUEST - EARTHWORLD / SwordQuest: Earthworld
            SWORDQUEST - FIREWORLD / SwordQuest: Fireworld
            SWORDQUEST - WATERWORLD / SwordQuest: Waterworld
            TEXAS CHAINSAW MASSACRE / The Texas Chainsaw Massacre
            TOMCAT - THE F-14 FLIGHT SIMULATOR / Tomcat: The F-14 Flight Simulator
            TRON - DEADLY DISCS / Tron: Deadly Discs
            VENTURE I / Venture
            VIDEO CUBE / Atari Video Cube
            YAR'S REVENGE / Yars' Revenge
            
            (1_2) Europe (NTSC Conversions):
            STEEPLE CHASE / **** see specific notes
            SURFER'S PARADISE / Surfer's Paradise: But Danger Below!
            
            (1_3) Europe (Non-NTSC Conversions):
            MISSION 3000 AD / Mission 3000 A. D.
            MR POSTMAN / Mr. Postman
            SEA MONSTER - SEE MONSTER / Sea Monster
            
            (1_5) Cassettes
            FROGGER I (ALT) / The Official Frogger
            
            (1_7) Special Releases
            PEPSI INVADERS / Pepsi Invaders: Coke Wins!
            
            (2_1) Homebrew
            AMIGA BOING / Amiga Boing 2.0
            ANGRY VIDEO GAME NERD KO BOXING / Angry Video Game Nerd K.O. Boxing
            CRAZY VALET / Crazy Valet CGE
            EDTRIS 2600 / EDTris
            GINGERBREAD MAN / Gingerbread Man: One Tough Cookie
            INCOMING / Incoming!
            INVPLUS / INV+
            KO CRUISER / K.O. Cruiser
            LORD OF THE RINGS - FELLOWSHIP OT RING / Lord of the Rings: Fellowship of the Rings
            MERLIN'S WALLS / Merlin's Walls: Standard Edition
            METROID PROOF DEMO / Metroid (Proof Demo)
            PHANTOM TWO / Phantom II &amp; Pirate (adds that bad "&amp;")
            REINDEER RESCUE / 2005 AtariAge Holiday Cart: Reindeer Rescue
            SEAWOLF / Sea Wolf
            STARFIRE / Star Fire
            THRUST PLUS - PLATINUM / Thrust Plus: Platinum
            TOYSHOP TROUBLE / 2006: AtariAge Holiday Cart: Toyshop Trouble
            
            (2_2) Prototypes
            A-TEAM / The A-Team
            BALTHAZAR (AKA BABYLON 5) / Balthazar
            CAREBEARS / Care Bears
            COMBAT II / Combat Two
            DUKES OF HAZZARD [STUNT CYCLE VERSION] / Dukes of Hazzard (Stunt Cycle Version)
            ENTITY / The Entity
            LORD OF THE RINGS - RIVENDELL / Lord of the Rings: Journey to Rivendell
            MCDONALD'S / McDonalds
            RED VS BLUE / Red vs. Blue
            REVENGE OF THE JEDI - GAME I / Star Wars: Revenge of the Jedi: Game I - Ewok Adventure
            ROCKY & BULLWINKLE / Rockey &amp; Bullwinkle (adds that bad "&amp;")
            SNOW WHITE / Disney's Snow White
            SNOW WHITE (VERSION 2) / Disney's Snow White (Later Prototype)
            WINGS / Wings [6-3-1983] (NTSC Version)
            WINGS (ALT) / Wings [10-01-1983] (PAL Version)
            
            (2_3) Hacks
            ADVENTURE (NEW GRAPHICS) / Adventure ( New Graphics Hack)
            BATTLEZONE TC (2 JOYSTICK HACK) / Battlezone TC
            BERZERK - RENEGADE / Berzerk: Renegade
            MR PAC-MAN / Mr. Pac-Man
            MS HACK / Ms. Hack
            PAC-MAN (MS PAC-MAN HACK) / Pac-Man Hack
            PITFALL! I - UNLIMITED / Pitfall! Unlimited
            RED VS BLUE 2600 WARS / Red vs. Blue: The 2600 Wars
            RETURN OF MARIO BROS / Return of Mario Bros.
            STAR WARS - THE BATTLE OF ALDERAN / Star Wars: The Battle of Alderan
            TARG (ALT) / NO SYNOPSIS (SEE NOTES)
            TOMB RAIDER (CLASSIC) / Tomb Raider: Mystery of the Scarlet Dragon (Classic Outfit)
            TOMB RAIDER (UNDERWORLD) / Tomb Raider: Mystery of the Scarlet Dragon (Underworld Outfit)
            VENTURE II / Venture II: The Abysmal Abyss
            
            (3) Adult Games
            TERRANCE & PHILLIP / Terrance &amp; Phillip: Asses of Fire (adds that bad "&amp;")
            TERRANCE & PHILLIP 2 / Terrance &amp; Phillip 2: Not Without My Anus (adds that bad "&amp;")
            X MAN / X-Man
            
            (4_5) Kid's Controller
            MONSTERCISE / Monstersice
            (5) Non-Working Games
            SWEAT! - THE DECATHALON GAME / Sweat!: The Decathalon Game
            `
            meleuM 1 Reply Last reply Reply Quote 0
            • meleuM
              meleu @Used2BeRX
              last edited by

              @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

              That should be the synopsis.txt file name. (...) every single piece of media and the rom itself is an exact match of that name.

              d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

              Can you --update and run it again?

              • Useful topics
              • joystick-selection tool
              • rpie-art tool
              • achievements I made
              1 Reply Last reply Reply Quote 0
              • meleuM
                meleu @Used2BeRX
                last edited by

                @Used2BeRX

                1. Fix the issue where "&" is turned into "&"

                It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as &amp; and ES works just fine with it. Test it and see.

                Side note: you have many bizarre game names in the Adult games folder... :-)

                • Useful topics
                • joystick-selection tool
                • rpie-art tool
                • achievements I made
                Used2BeRXU 1 Reply Last reply Reply Quote 0
                • Used2BeRXU
                  Used2BeRX @meleu
                  last edited by Used2BeRX

                  @meleu

                  d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

                  Can you --update and run it again?

                  Sure. I'm going to do it right now.

                  It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as & and ES works just fine with it. Test it and see.

                  Okay. It just wasn't finding the media because it was looking for media with "&.amp;" in it :) I'll verify it works right after this run.

                  Side note: you have many bizarre game names in the Adult games folder... :-)

                  If you think the names are weird, you should try playing them. ;)

                  meleuM 1 Reply Last reply Reply Quote 0
                  • meleuM
                    meleu @Used2BeRX
                    last edited by

                    @Used2BeRX I've just added the --no-desc option you asked. Update the script again. ;-)

                    • Useful topics
                    • joystick-selection tool
                    • rpie-art tool
                    • achievements I made
                    Used2BeRXU 1 Reply Last reply Reply Quote 0
                    • Used2BeRXU
                      Used2BeRX @meleu
                      last edited by Used2BeRX

                      @meleu It says I have the latest version. Nothing changed. Maybe I have to wait a few minutes? nevermind. got it now.

                      About folders... How do you suppose we should go about that?

                      The <image> for the folder is going to be named exactly the same as the folder name and the folder.txt file, and it's going to be in /home/pi/RetroPie/Media/{system_name}/Artwork/Folders/

                      meleuM 1 Reply Last reply Reply Quote 0
                      • meleuM
                        meleu @Used2BeRX
                        last edited by

                        @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

                        @meleu It says I have the latest version. Nothing changed. Maybe I have to wait a few minutes?

                        maybe yes. Just a few minutes...

                        About folders... How do you suppose we should go about that?

                        I'm taking a look at it.

                        The <image> for the folder is going to be named exactly the same as the folder name and the folder.txt file, and it's going to be in /home/pi/RetroPie/Media/{system_name}/Artwork/Folders/

                        Good. I'll stick with this standard.

                        • Useful topics
                        • joystick-selection tool
                        • rpie-art tool
                        • achievements I made
                        Used2BeRXU 1 Reply Last reply Reply Quote 0
                        • Used2BeRXU
                          Used2BeRX @meleu
                          last edited by

                          @meleu I ran "--no-desc" after the command and it's running now but still adding the description. Overall, as well, I'm not noticing that the speed has picked up at all yet.

                          I ran the command like this ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.* --no-desc

                          Should I run it like this instead?

                          ./Used2BeTxt.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/*.*

                          meleuM 1 Reply Last reply Reply Quote 0
                          • meleuM
                            meleu @Used2BeRX
                            last edited by meleu

                            @Used2BeRX

                            I ran the command like this ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.* --no-desc

                            Usage:
                            ./Used2BeTXT.sh [OPTIONS] synopsis1.txt [synopsisN.txt ...]
                            

                            Yes, the options needs to be the first argument

                            • Useful topics
                            • joystick-selection tool
                            • rpie-art tool
                            • achievements I made
                            meleuM Used2BeRXU 2 Replies Last reply Reply Quote 0
                            • meleuM
                              meleu @meleu
                              last edited by

                              @Used2BeRX by the way, if you want to interrupt the script in execution, hit ctrl+c.

                              • Useful topics
                              • joystick-selection tool
                              • rpie-art tool
                              • achievements I made
                              1 Reply Last reply Reply Quote 0
                              • Used2BeRXU
                                Used2BeRX @meleu
                                last edited by Used2BeRX

                                @meleu Got it. Will re-run it now. BTW.... I checked the entry for "Adventure (New Graphics)" which previously didn't work and it works now! :)

                                by the way, if you want to interrupt the script in execution, hit ctrl+c.

                                I accidentally figured that out when I stopped the script 10 minutes ago for your new update. Sometimes I get lucky and Linux behaves like DOS did. ;)

                                EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

                                meleuM 1 Reply Last reply Reply Quote 0
                                • meleuM
                                  meleu @Used2BeRX
                                  last edited by meleu

                                  @Used2BeRX

                                  EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

                                  Here's a tip to measure this kind of thing: Use time before a command.

                                  Examples:

                                  with --no-desc

                                  meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --no-desc ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
                                  "../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".
                                  
                                  real    0m0.242s
                                  user    0m0.124s
                                  sys     0m0.084s
                                  

                                  default use (no options)

                                  meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT
                                  "../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".
                                  
                                  real    0m0.257s
                                  user    0m0.148s
                                  sys     0m0.072s
                                  

                                  with --full

                                  meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --full ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
                                  "../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_FULL_gamelist.xml".
                                  
                                  real    0m1.200s
                                  user    0m0.664s
                                  sys     0m0.312s
                                  

                                  If you are curious about the meaning of real, user and sys, this is a good reading: https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1

                                  • Useful topics
                                  • joystick-selection tool
                                  • rpie-art tool
                                  • achievements I made
                                  Used2BeRXU 1 Reply Last reply Reply Quote 0
                                  • Used2BeRXU
                                    Used2BeRX @meleu
                                    last edited by Used2BeRX

                                    @meleu Good to know. Thanks. I will use that going forward.

                                    Right now it's 4:49, so we're at 1 hour and 7 minutes and we're only at "Miner 2049er", which is number 359 out of 745.

                                    I didn't time it last time, but I think it took just over 3 hours to make the list. The way this is going now, I think it will be about 2 1/2 hours. Looks like changing the way it searches for files like you did today may have cut down the time about 15-20%, but I can't be sure. Some of this time savings might also be because it's not doing the <desc>

                                    I'm thinking that if you can write the code so it looks for specific media in specific folders instead of all of the folders that we could put a serious dent in the time it takes to create these lists.

                                    I didn't check all of them so far, but a cursory "find" on the document shows that every entry that I looked at that wasn't working before is working now. I will let you know what our percentage of good files was when the list completes and I can scroll through the games in the romlist in EmulationStation.

                                    Also, the file is extremely smaller than it was with the description like I wanted it to be. I'm hoping this helps with a Pi Zero. The file I ran yesterday was over 1.2MB when it was finished. With the file nearly halfway done right now, the file is only 168kb. I think that "--no-desc" option is going to work out great for the Pi Zero when coupled with a skin that has no metadata like "Carbon No-Meta"

                                    meleuM 1 Reply Last reply Reply Quote 0
                                    • meleuM
                                      meleu @Used2BeRX
                                      last edited by

                                      @Used2BeRX I was working on the folder path finding and then the kids demanded my atention. I uploaded what I've done so far but didn't make enough tests. If you wanna test, give it a try. There's no need to parse every synopsis file again, only the folders ones. The script updates the file accordingly.

                                      Now I have to play with kids! :-)

                                      • Useful topics
                                      • joystick-selection tool
                                      • rpie-art tool
                                      • achievements I made
                                      Used2BeRXU 1 Reply Last reply Reply Quote 0
                                      • Used2BeRXU
                                        Used2BeRX @meleu
                                        last edited by

                                        @meleu So you're saying that even though the folder entries are already in the gamelist.xml it will automatically make the updates to them? If that's the case, that's sweet. :)

                                        When this is done I'm going to verify which games if any still have problems and let you know how close to 100% we came, then I'll re-run the script to see if we get the folders working too.

                                        Have fun with the kids. :)

                                        meleuM 1 Reply Last reply Reply Quote 0
                                        • meleuM
                                          meleu @Used2BeRX
                                          last edited by

                                          @Used2BeRX said in Making premade gamelist.xml with xtra media collections:

                                          So you're saying that even though the folder entries are already in the gamelist.xml it will automatically make the updates to them?

                                          The script updates an existing .xml file instead of creating a new one since the first version. I did this way for a case you get a new synopsis.txt for a game, there's no need to parse every single file again. Just run the script with the new synopsis file and the metadata will be added to the existing gamelist.xml.

                                          • Useful topics
                                          • joystick-selection tool
                                          • rpie-art tool
                                          • achievements I made
                                          Used2BeRXU 2 Replies Last reply Reply Quote 0
                                          • Used2BeRXU
                                            Used2BeRX @meleu
                                            last edited by Used2BeRX

                                            @meleu Nice.

                                            Does it alter the existing entry when you do this, or does it create a second entry?

                                            If I run the same command with the *.* will it take a few hours to complete again, or will it skip all of the entries that are complete with the given parameter of --no-desc?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post

                                            Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

                                            Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.