Making premade gamelist.xml with xtra media collections

meleu

lol... why would I use a scraper after we've done all of this work?

haha. I'm doing this because I take it as a diversion. If a scraper can give you the results you want, please, use it! :-)

I'm a kind of data manipulation enthusiast. I love shell scripting, regular expressions, parsing files... With your request in this thread I saw a possiblity to improve my hability to parse files and manage xml files and this is already happening :-)

Used2BeRX

@meleu Nah man. I got everything I want already if we get the code working.

Have you had a chance to upload that new script, or were you still doing some tweaks on it before I test it out?

meleu

@Used2BeRX

I've just uploaded the script. You can get it with the --update option.

I didn't implemented the option to create the gamelist.xml with no <desc>. And I'm going to be busy for a week. Will only have a chance to work on the script next week.

But you can put your feedback/suggestions here.

Cheers!

Used2BeRX

@meleu Got it. It works with the following command:

./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.*

Seems to be going through the files one by one and slowly adding them. It says "data has been added to "atari2600_gamelist.xml" after each file. Looks like it takes about 10 seconds for each entry.

I'm able to open atari2600_gamelist.xml right now and it looks to be working right for the roms.

It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

meleu

@Used2BeRX

It's not working for the folder entries though. It is making folder entries and adding the correct <name></name>, but it is not putting anything for <path></path>

Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

I'm pretty sure that this is going to have all of the roms themselves working once its' done though. I'll let you know.

I noticed some fails but didn't fixed yet. Examples of games that the current script version is unable to find

mega man - wily wars (megadrive)
mickey mouse - fantasia (megadrive)

Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

Used2BeRX

@meleu The script is still going after about 2 hours for the Atari 2600 (about 750 entries). I think it will have taken about 2 1/2 to 3 hours once it's done.

Yeah. I've spent my last free time learning find tricks to optimize the processing time and didn't work on finding folder's path yet. And I'm still learning advanced find options...

Cool. Just making sure that the script wasn't designed to work on the folders yet.

Currently the script tries to find the files based on the game name (first line of a synopsis.txt). I'll make it try to find based on the synopsis file name too.

I noticed that the <xtrasname> field was added, but it's not being populated atm. That should be the synopsis.txt file name. If the script searched off of that it should be able to find things a lot quicker since every single piece of media and the rom itself is an exact match of that name.

Used2BeRX

@meleu I've just checked it and the Script on the Atari 2600 had 108 erroneous entries out of 724.

Almost 100% of these would be avoided if two things were done:

Use only the SYNOPSIS.TXT name (<xtrasname>) to find the media.
Fix the issue where "&" is turned into "&"

Here is the list of the errors:

M*A*S*H* entry wasn't even created.  (Filename MASH.TXT)

TRACK AND FIELD ------ & character showed up as "&amp;" so title line was "Track &amp; Field"  (This happens every time a & was used in any part of the synopsis.txt)

STEEPLE CHASE and STEEPLECHASE ------ Both of these fought for media/rom STEEPLECHASE instead of their separate ones.  US version STEEPLECHASE shows properly, EU version STEEPLE CHASE does not.

TARG and TARG (ALT) both are called just "Targ" on the first line.  Only an entry for TARG.TXT was made and TARG (ALT).TXT was skipped.

Specific errors:

(1_1) US
ATLANTIS I / Atlantis
BUCK RODGERS / Buck Rogers: Planet of Zoom
BUMP 'N JUMP / Bump 'n' Jump
CIRCUS / Circus Atari
COMBAT I / Combat
DEFENDER I / Defender
ENCOUNTER AT L5 / Encounter at L-5
ET - THE EXTRA TERRESTRIAL / E.T.: The Extra Terrestrial
FROGGER I / Frogger
FROGGER II - THREEDEEP! / Frogger II: Threedeep!
GHOSTBUSTERS I / Ghostbusters
GI JOE - COBRA STRIKE / G.I. Joe: Cobra Strike
HERO / H.E.R.O.
HOME RUN BASEBALL / Home Run
JAMES BOND 007 / James Bond: 007
JOURNEY - ESCAPE / Journey: Escape
JR PAC-MAN / Jr. Pac-Man
KOOL AID MAN / Kool-Aid Man
KUNG FU MASTER / Kung-Fu Master
MAD / M.A.D.
MARIO BROS / Mario Bros.
MASTERS OF THE UNIVERSE / Masters of the Universe: The Power of He-Man
MINER 2049ER - VOLUME II / Miner 2049er: Volume II
MONTEZUMA'S REVENGE / Montezuma's Revenge featuring Panama Joe
MR DO! / Mr. Do!
MR DO! / Mr. Do!'s Castle
MS PAC-MAN / Ms. Pac-Man
MUSIC MACHINE / The Music Machine
PITFALL! / Pitfall!
PITFALL! II LOST CAVERNS / Pitfall! II: Lost Caverns
QUICK STEP! / Quick Step
RAMPAGE! / Rampage
RIVER RAID I / River Raid
SMURFS - RESCUE IN GARGAMEL'S CASTLE / Smurfs: Rescue in Gargamel's Castle
SNEEK 'N PEEK / Sneak 'n Peak
SPACE SHUTTLE - JOURNEY INTO SPACE / Space Shuttle: A Journey Into Space
SPACEMASTER X-7 / Space Master X-7
SPIDER MAZE / Spider-Maze
STAR GUNNER / StarGunner
STAR SHIP - OUTER SPACE / Star Ship
STAR TREK - STRAGEGIC OPS SIMULATOR / Star Trek: Strategic Operations Simulator
STAR WARS - DEATH STAR BATTLE / Star Wars: Return of the Jedi - Death Star Battle
STAR WARS - JEDI ARENA / Star Wars: Jedi Arena
STAR WARS - THE ARCADE GAME / Star Wars: The Arcade Game
STAR WARS - THE EMPIRE STRIKES BACK / Star Wars: The Empire Strikes Back
STRAWBERRRY SHORTCAKE - MUSICAL MATCH / Strawberry Shortcake: Musical Match-Ups
SWORDFIGHT / Sword Fight
SWORDQUEST - EARTHWORLD / SwordQuest: Earthworld
SWORDQUEST - FIREWORLD / SwordQuest: Fireworld
SWORDQUEST - WATERWORLD / SwordQuest: Waterworld
TEXAS CHAINSAW MASSACRE / The Texas Chainsaw Massacre
TOMCAT - THE F-14 FLIGHT SIMULATOR / Tomcat: The F-14 Flight Simulator
TRON - DEADLY DISCS / Tron: Deadly Discs
VENTURE I / Venture
VIDEO CUBE / Atari Video Cube
YAR'S REVENGE / Yars' Revenge

(1_2) Europe (NTSC Conversions):
STEEPLE CHASE / **** see specific notes
SURFER'S PARADISE / Surfer's Paradise: But Danger Below!

(1_3) Europe (Non-NTSC Conversions):
MISSION 3000 AD / Mission 3000 A. D.
MR POSTMAN / Mr. Postman
SEA MONSTER - SEE MONSTER / Sea Monster

(1_5) Cassettes
FROGGER I (ALT) / The Official Frogger

(1_7) Special Releases
PEPSI INVADERS / Pepsi Invaders: Coke Wins!

(2_1) Homebrew
AMIGA BOING / Amiga Boing 2.0
ANGRY VIDEO GAME NERD KO BOXING / Angry Video Game Nerd K.O. Boxing
CRAZY VALET / Crazy Valet CGE
EDTRIS 2600 / EDTris
GINGERBREAD MAN / Gingerbread Man: One Tough Cookie
INCOMING / Incoming!
INVPLUS / INV+
KO CRUISER / K.O. Cruiser
LORD OF THE RINGS - FELLOWSHIP OT RING / Lord of the Rings: Fellowship of the Rings
MERLIN'S WALLS / Merlin's Walls: Standard Edition
METROID PROOF DEMO / Metroid (Proof Demo)
PHANTOM TWO / Phantom II &amp; Pirate (adds that bad "&amp;")
REINDEER RESCUE / 2005 AtariAge Holiday Cart: Reindeer Rescue
SEAWOLF / Sea Wolf
STARFIRE / Star Fire
THRUST PLUS - PLATINUM / Thrust Plus: Platinum
TOYSHOP TROUBLE / 2006: AtariAge Holiday Cart: Toyshop Trouble

(2_2) Prototypes
A-TEAM / The A-Team
BALTHAZAR (AKA BABYLON 5) / Balthazar
CAREBEARS / Care Bears
COMBAT II / Combat Two
DUKES OF HAZZARD [STUNT CYCLE VERSION] / Dukes of Hazzard (Stunt Cycle Version)
ENTITY / The Entity
LORD OF THE RINGS - RIVENDELL / Lord of the Rings: Journey to Rivendell
MCDONALD'S / McDonalds
RED VS BLUE / Red vs. Blue
REVENGE OF THE JEDI - GAME I / Star Wars: Revenge of the Jedi: Game I - Ewok Adventure
ROCKY & BULLWINKLE / Rockey &amp; Bullwinkle (adds that bad "&amp;")
SNOW WHITE / Disney's Snow White
SNOW WHITE (VERSION 2) / Disney's Snow White (Later Prototype)
WINGS / Wings [6-3-1983] (NTSC Version)
WINGS (ALT) / Wings [10-01-1983] (PAL Version)

(2_3) Hacks
ADVENTURE (NEW GRAPHICS) / Adventure ( New Graphics Hack)
BATTLEZONE TC (2 JOYSTICK HACK) / Battlezone TC
BERZERK - RENEGADE / Berzerk: Renegade
MR PAC-MAN / Mr. Pac-Man
MS HACK / Ms. Hack
PAC-MAN (MS PAC-MAN HACK) / Pac-Man Hack
PITFALL! I - UNLIMITED / Pitfall! Unlimited
RED VS BLUE 2600 WARS / Red vs. Blue: The 2600 Wars
RETURN OF MARIO BROS / Return of Mario Bros.
STAR WARS - THE BATTLE OF ALDERAN / Star Wars: The Battle of Alderan
TARG (ALT) / NO SYNOPSIS (SEE NOTES)
TOMB RAIDER (CLASSIC) / Tomb Raider: Mystery of the Scarlet Dragon (Classic Outfit)
TOMB RAIDER (UNDERWORLD) / Tomb Raider: Mystery of the Scarlet Dragon (Underworld Outfit)
VENTURE II / Venture II: The Abysmal Abyss

(3) Adult Games
TERRANCE & PHILLIP / Terrance &amp; Phillip: Asses of Fire (adds that bad "&amp;")
TERRANCE & PHILLIP 2 / Terrance &amp; Phillip 2: Not Without My Anus (adds that bad "&amp;")
X MAN / X-Man

(4_5) Kid's Controller
MONSTERCISE / Monstersice
(5) Non-Working Games
SWEAT! - THE DECATHALON GAME / Sweat!: The Decathalon Game
`

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

That should be the synopsis.txt file name. (...) every single piece of media and the rom itself is an exact match of that name.

d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

Can you --update and run it again?

meleu

@Used2BeRX

Fix the issue where "&" is turned into "&"

It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as & and ES works just fine with it. Test it and see.

Side note: you have many bizarre game names in the Adult games folder... :-)

Used2BeRX

@meleu

d'uh! I wasn't aware about it. I slightly modified the script to search things this way and indeed, it's faster.

Can you --update and run it again?

Sure. I'm going to do it right now.

It's not an issue! I did it intentionally. the & character alone is an invalid XML char. It needs to be encoded as & and ES works just fine with it. Test it and see.

Okay. It just wasn't finding the media because it was looking for media with "&.amp;" in it :) I'll verify it works right after this run.

Side note: you have many bizarre game names in the Adult games folder... :-)

If you think the names are weird, you should try playing them. ;)

meleu

@Used2BeRX I've just added the --no-desc option you asked. Update the script again. ;-)

Used2BeRX

@meleu ~~It says I have the latest version. Nothing changed. Maybe I have to wait a few minutes?~~ nevermind. got it now.

About folders... How do you suppose we should go about that?

The <image> for the folder is going to be named exactly the same as the folder name and the folder.txt file, and it's going to be in /home/pi/RetroPie/Media/{system_name}/Artwork/Folders/

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu It says I have the latest version. Nothing changed. Maybe I have to wait a few minutes?

maybe yes. Just a few minutes...

About folders... How do you suppose we should go about that?

I'm taking a look at it.

The <image> for the folder is going to be named exactly the same as the folder name and the folder.txt file, and it's going to be in /home/pi/RetroPie/Media/{system_name}/Artwork/Folders/

Good. I'll stick with this standard.

Used2BeRX

@meleu I ran "--no-desc" after the command and it's running now but still adding the description. Overall, as well, I'm not noticing that the speed has picked up at all yet.

I ran the command like this ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.* --no-desc

Should I run it like this instead?

./Used2BeTxt.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synoipsis/*.*

meleu

@Used2BeRX

I ran the command like this ./Used2BeTxt.sh /home/pi/RetroPie/Media/atari2600/Synoipsis/*.* --no-desc

Usage:
./Used2BeTXT.sh [OPTIONS] synopsis1.txt [synopsisN.txt ...]

Yes, the options needs to be the first argument

meleu

@Used2BeRX by the way, if you want to interrupt the script in execution, hit ctrl+c.

Used2BeRX

@meleu Got it. Will re-run it now. BTW.... I checked the entry for "Adventure (New Graphics)" which previously didn't work and it works now! :)

by the way, if you want to interrupt the script in execution, hit ctrl+c.

I accidentally figured that out when I stopped the script 10 minutes ago for your new update. Sometimes I get lucky and Linux behaves like DOS did. ;)

EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

meleu

@Used2BeRX

EDIT: Start time was 3:42 by me. I'll try to gauge when it will finish so I can get an exact time to see how long it takes to do 745 entries with "--no-desc"

Here's a tip to measure this kind of thing: Use time before a command.

Examples:

with --no-desc

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --no-desc ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".

real    0m0.242s
user    0m0.124s
sys     0m0.084s

default use (no options)

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_gamelist.xml".

real    0m0.257s
user    0m0.148s
sys     0m0.072s

with --full

meleu@mlenovo ~/src/Used2BeTXT $ time ./Used2BeTXT.sh --full ../XBox\ Synopsis\ Files/GENESIS/synopsis/MEGA\ MAN\ -\ THE\ WILY\ WARS.TXT 
"../XBox Synopsis Files/GENESIS/synopsis/MEGA MAN - THE WILY WARS.TXT" data has been added to "megadrive_FULL_gamelist.xml".

real    0m1.200s
user    0m0.664s
sys     0m0.312s

If you are curious about the meaning of real, user and sys, this is a good reading: https://stackoverflow.com/questions/556405/what-do-real-user-and-sys-mean-in-the-output-of-time1

Used2BeRX

@meleu Good to know. Thanks. I will use that going forward.

Right now it's 4:49, so we're at 1 hour and 7 minutes and we're only at "Miner 2049er", which is number 359 out of 745.

I didn't time it last time, but I think it took just over 3 hours to make the list. The way this is going now, I think it will be about 2 1/2 hours. Looks like changing the way it searches for files like you did today may have cut down the time about 15-20%, but I can't be sure. Some of this time savings might also be because it's not doing the <desc>

I'm thinking that if you can write the code so it looks for specific media in specific folders instead of all of the folders that we could put a serious dent in the time it takes to create these lists.

I didn't check all of them so far, but a cursory "find" on the document shows that every entry that I looked at that wasn't working before is working now. I will let you know what our percentage of good files was when the list completes and I can scroll through the games in the romlist in EmulationStation.

Also, the file is extremely smaller than it was with the description like I wanted it to be. I'm hoping this helps with a Pi Zero. The file I ran yesterday was over 1.2MB when it was finished. With the file nearly halfway done right now, the file is only 168kb. I think that "--no-desc" option is going to work out great for the Pi Zero when coupled with a skin that has no metadata like "Carbon No-Meta"

meleu

@Used2BeRX I was working on the folder path finding and then the kids demanded my atention. I uploaded what I've done so far but didn't make enough tests. If you wanna test, give it a try. There's no need to parse every synopsis file again, only the folders ones. The script updates the file accordingly.

Now I have to play with kids! :-)