Making premade gamelist.xml with xtra media collections

meleu

CURRENT ISSUES:

None of the `<path/> were written for Atari 5200/800. This might be because it's a special case. Of all of the systems I have for the RetroPie, this is the only one that you can't ZIP the files. All 5200/800 images I have are in BIN format.

Yep. The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

Folders for other systems are being added to "atari2600_gamelist.xml". (I know why this is happening. All of my folders.txt files still say the information for the Atari 2600).

Exactly. The synopsis.txt files must be accurate. In this case, the problem is in the Platform: field.

With the Atari 5200/800, I'd like for them to all write to atari5200_gamelist.xml. I don't include a lot of 800 games in my collection and rather than have two entries for it in the system I have all of the roms and media in the 5200 file structure, so I was wondering if we could maybe do somthing about putting those two scripts together. If that's too much of a hassle, no worries. I can just manually combine them when I'm done.

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

I had a few ideas, such as creating all of the gamelist.xmls for all existing systems one after another without having to be there to put in the next command.(or from a list I could create in a file).

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

I was also thinking that we could do a --no-desc but with all of the Artwork folders, and then there could be a script that would easily change all of the entries to point to a different piece of artwork (ie: displaying action shots for all of the games instead of Boxart).

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

And the generated gamelist.xml will have the <image> pointing to the 3D Box art.

Edit: Run --help to see the --image details.

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

Used2BeRX

@meleu

The script was only trying to find .zip. I've made it search for .bin too. Do you have other extensions?

This won't increase search time, will it? All consoles have .zip except for .bin in 5200. I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

The script puts every entry on its own platform gamelist.xml. Do they use the same emulator in RetroPie? Anyway, if you want to see atari 800 games in the atari 5200 you'll have to combine manually, as you said.

Yeah. They both use the same emulator.

You can give everything in only one command line. Like this:

./Used2BeTXT /path/to/atari2600/synopsis/*.* /path/to/nes/synopsis/*.* /path/to/megadrive/synopsis/*.*

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

I've added the --image option. Then you can do:

./Used2BeTXT --no-desc --image 3dbox /path/to/atari2600/synopsis/*.*

What does --image alone do? Does it add all of the image file fields, but default to Box Front? And then it would change it to 3D Box art or other types if I put the second command after it?

It's not fast if you just wanna change this field on an existing gamelist.xml. But if you just wanna do that, you don't need a script. You just need a basic knowledge about sed command. I'll leave this task with you. ;-)

The reason I wanted the --image option to add all 5 types of artwork is because before I had any help writing scripts I did this switch between artwork types manually myself with a few search and replace commands in Notepad++.

I was hoping that if the --image option wrote all 5 fields for all 5 pieces of artwork, that a quick script could be written that would automatically do this. For instance, if you wanted to show Title shots, but it was currently showing Box Front, the script would first replace all <image></image> lines to be <boxfront></boxfront> and then replace all <title></title> lines to be <image></image>.

But... It would have to do this ONLY in <game> entries. The script would break the <folder> if it changed the <image> fields for the folders.

Used2BeRX

@meleu BTW dude... I've been learning some cool stuff in a great photo manipulation program called paint.net. Never did any graphic arts before and I always thought Photoshop was too complicated to bother learning...

Check these folder images out. :)

ATARI 5200 PROTOTYPES / HOMEBREW
alt text

US LICENSED GAME GEAR:
alt text

EUROPE LICENSED GAME GEAR:
alt text

JAPAN LICENSED GAME GEAR:
alt text

Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems. I just started the script for all included systems so far and this is what it looked like:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/atari2600/Synopsis/*.* /home/pi/RetroPie/Media/atari5200/Synopsis/*.* /home/pi/RetroPie/Media/atari7800/Synopsis/*.* /home/pi/RetroPie/Media/atarilynx/Synopsis/*.* /home/pi/RetroPie/Media/gamegear/Synopsis/*.* /home/pi/RetroPie/Media/gb/Synopsis/*.* /home/pi/RetroPie/Media/gba/Synopsis/*.* /home/pi/RetroPie/Media/gbc/Synopsis/*.* /home/pi/RetroPie/Media/mastersystem/Synopsis/*.* /home/pi/RetroPie/Media/megadrive/Synopsis/*.* /home/pi/RetroPie/Media/nes/Synopsis/*.* /home/pi/RetroPie/Media/ngp/Synopsis/*.* /home/pi/RetroPie/Media/pcengine/Synopsis/*.* /home/pi/RetroPie/Media/sega32x/Synopsis/*.* /home/pi/RetroPie/Media/sg-1000/Synopsis/*.* /home/pi/RetroPie/Media/snes/Synopsis/*.* /home/pi/RetroPie/Media/videopac/Synopsis/*.* /home/pi/RetroPie/Media/virtualboy/Synopsis/*.* /home/pi/RetroPie/Media/wonderswan/Synopsis/*.*

Not fun.... :(

Anyway....

I completed editing EVERY folder synopsis file, so once this finishes running I should be able to compile a full list of any problems with the script for every system except for the CD based systems.

I'll let you know more as soon as I know.

Thanks again.

meleu

@Used2BeRX

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

This won't increase search time, will it?

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

I haven't tried any CD consoles yet, but .bin will probably be an issue there. I'm pretty sure that with SegaCD and TurboGrafxCD it will need to look for .cue and actually NOT look for .bin

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Is it possible for me to write the command once and save it so I can just type a quick command to run it?

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

What does --image alone do?

gives an error message and exit.

I'm planning add more options for <image> soon.

meleu

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:

time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*

Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Used2BeRX

@meleu

I've just noticed that there's a smallbug in the code when some metadata has & and you re-run the script with the same synopsis. I'm trying to fix it now. In the first run (creating a new entry in the gamelist.xml) this bug doesn't happen.

No worries. I'm creating all new gamelist.xml files right now.

Maybe some miliseconds. But the script must be able to find those files, doesn't it? :-)

Milliseconds on your PC probably adds a lot on the Pi Zero. I don't have a time for just Atari 2600 since I was doing all of them, but I think it's taking 3-4 seconds each entry compared to the 2 seconds it was taking before. Maybe a separate script for the Atari 5200 is in order?

OK, I've made the script search for .cue if Media: CD, no matter what platform it is.

Nice... I won't be able to test this until I have a Pi 3 though, unfortunately :(

Yes, you can create an alias. But if you prefer to create a list in a file, I can show you some tricks on how to use this file (nothing to be changed in the script).

I'd be interesting in learning more about this. I gave you the full command above... what would I do with that to run it without typing 1,000 characters next time?

I'm planning add more options for <image> soon.

Sweet.

The script is still running. I started it over 5 hours ago and right now it's on the megadrive. Still got quite a lot to go through with some smaller systems as well as the TurboGrafx and SNES.

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:
time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*
Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Ah... didn't notice you posted this before I replied.

Still have some possible issues that I mentioned though. I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

Used2BeRX

@meleu I'm still making pics for the folders on about 3 more systems before I test this out, but the script finally finished!!! Just wanted to give you the time.

673m44s, 420m10s, 197m50s.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

The searching for bin occurs only when platform is atari5200 or atari800.

Used2BeRX

@meleu 1st run through is done and the results are great overall.

100% HITS FOR THE FOLLOWING SYSTEMS!!!:
Atari 2600
Atari 7800
Atari LYNX
Game Gear
Gameboy Advance
Gameboy Color
Neo Geo Pocket/Neo Geo Pocket Color
Sega 32x
Sega SG-1000
Wonderswan/Wonderswan Color

MAJOR FULL-SYSTEM PROBLEMS:

SNES did not work. 338kb gamelist.xml file was created, but not a single rom or image was entered in the fields. Instead of writing to "snes_gamelist.xml" it wrote to "-supernintendoentertainmentsystem_gamelist.xml".
Odyssey 2 did not work. It makes magnavoxodyssey2_gamelist.xml, but RetroPie is expecting videopac for the folder, so it didn't find any roms or media.
Virtual Boy didn't work. It makes gamelist.xml with no other identifiers. RetroPie is expecting virtualboy for the folder.

FIXES TO SYNOPIS I MADE:
Changed "Platform: " on the following.
All "Atari 800" to "Atari 5200"
All "Sega Net" to "Sega Genesis / Mega Drive"
All "The Famicom Disk System" to "Nintendo Entertainment System"

Big thing to note is that you cannot have two 1st line synopsis.txt descriptions in the same gamelist.xml. It confuses the script. So I made the following changes by system to get the rest of the working systems up to 100%

TurboGrafx:
(2) Translated Hu-Card Games
(2) Translated CD Games
Bonk 3: Bonk's Big Adventure (CD)
Altered Beast (CD)

NES:
Metroid (Famicom Disk System)
Clu Clu Land (Famicom Disk System)
Gyruss (Famicom Disk System)
Kid Icarus (Famicom Disk System)
Super Mario Bros (Famicom Disk System)
Super Mario Bros 2 (Famicom Disk System)
Vs Excite Bike (Famicom Disk System)
AD&D Dragons of the Flame (Translated)
Battleship (Unlicensed)
Contra (Translated)
Maniac Mansion [Uncensored]
Ms. Pac-Man (Unlicensed)
Rainbow Islands (Europe)
Fantasy Zone I (Japan)
Super Mario Bros. 2 (Japan)
Othello (Australia)
Airwolf (Translated)
Contra (Translated)
Squoon (Translated)
Super Mario Bros. 2 (Prototype)
"Adult Games" 2/3rds of them were "Famicom Disk System" and showed up after changing this to "Nintendo Entertainment System"

GAME BOY:
Bomberman GB (Japan)

GENESIS:
Dynamite Headdy (Japan)
Rings of Power: The Hands of Fate (Hack)

MASTER SYSTEM:
Pit Pot..... accidentially had it listed as Game Gear.
Assault City (Light Phaser Version)

ATARI 5200:
Centipede (Atari 800)
Choplifter! (Atari 800)
Crystal Castles (Atari 800)
Mario Bros. (Atari 800)
Qix (Atari 800)
Rally Speedway (Atari 800)

Used2BeRX

@meleu Any ideas about SNES, Virtualboy and Odyssey 2, meleu?

I'll have to look deeper into those individual game errors on the other systems, but if you look at my last post you'll see a section for "Major Problems" detailing why I can't get any of those gamelist.xml files to work with the script.

It looks like based off of the Platform: field, it's not looking in the correct folders for roms or media.

meleu

@Used2BeRX I'll look into it when I have a chance, I think I know what's happening... No need to change txt files on your end.

meleu

@Used2BeRX I've fixed the snes issue and then the kids dragged me to play. :-)

Please test the snes synopsis conversion.

Used2BeRX

@meleu I already made all the synopsis changes on my end for the individual things above. Most notably changing all "Atari 800" to "Atari 5200", "Famicom Disk System" to "Nintendo Entertainment System" and "Sega Net" to "Sega Genesis / Mega Drive".

I'm actually in the process of re-running all of the gamelists right now. After making those changes to all of the games with the same name, I also had to delete a lot of entries from the original gamelists because they were bad. When it was getting confused, sometimes it was putting the Japanese or Translated version paths for the US version. But when I deleted that entry and re-ran the system, it was finding the other rom and not putting the US one in.

I don't know if I actually fixed anything by changing the name because of this, so I decided to re-run everything from scratch and then re-check the results and see where I land after all of these changes.

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

meleu

@Used2BeRX

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

I'm afraid that the wisest decision would be to cancel this execution, --update and run it again. :-)

By the way, in the pack of synopsis you sent me doesn't have synopsis for Virtual Boy games. Can you upload it somewhere?

Used2BeRX

@meleu Did you make any changes other than SNES? If not, then I'll let this run finish first, and then do the SNES after the update. I renamed the synopsis folders for SNES, Odyssey 2 and Virtual Boy to "Synopsis2" so it will skip them during this run.

If there is a specific reason otherwise that you think I should cancel though, I will do so.

Here's virtual boy synopsis: https://www.sendspace.com/file/29fay6

EDIT:

Yeah... actually it's probably a lot closer to finishing than I thought with those 3 folders removed. It just finished up nes, so all that is left is the smaller libraries of ngpc, pcengine, sega32x and wonderswan now. Should be done in 30-45 minutes tops.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Did you make any changes other than SNES?

I've just fixed SNES, Virtual Boy and Odissey2 platform issue. Test it when you have a chance.

Let me know if you have any new issue. I'm assuming those problems with specific games you solved on your end, right?

Used2BeRX

@meleu Oh cool. You got to all three of the others. I should be able to test them very soon.

I believe I solved the individual game problems, but using the command to update the xml files wasn't working for me since the original problem was making totally erroneous entries, so I started over.

I should have that answer for you later today. I just have to go through every game again and see if any errors pop up.

Used2BeRX

@meleu Hey bud. Great update there. SNES, Odyssey 2 and Virtual Boy script works now. :)

I'm posting my current results here just for the record. I don't believe at this time that there is anything to do on your end, but I have a lot of individual things to go through. I noticed that some games on some systems are using artwork for other systems. I need to go in and see if this is a mistake in the artwork packs themselves or just a wrong entry in the synopsis entries for Platform: .

The only possible thing at this point that I think we may need to address to finally get 100% on this is on the Odyssey 2. ZIP files are included in the es_systems.cfg file since they show up on the romlist, but I don't believe the system will actually play zipped roms since none of them work and I know they're good roms. I will need to research that sometime this week and get back to you about it.

Anyway.... Here's my current results.

6-25-17 script test:

SYSTEMS WITH 100% NO ERRORS:

GAME BOY COLOR (0 errors of 596 roms)
SEGA 32X (0 errors out of 39 roms)
SEGA MASTER SYSTEM (0 errors out of 319 roms)
GAME GEAR (0 errors out of 308 roms)
SEGA SG-1000 (0 errors out of 65 roms)
ATARI 2600 (0 errors out of 724 roms)
ATARI 5200 (0 errors out of 263 roms)
ATARI 7800 (0 errors out of 86 roms)
ATARI LYNX (0 errors out of 86 roms)
NEO GEO POCKET/COLOR (0 errors out of 49 roms)
VIRTUAL BOY (0 errors out of 27 roms)
WONDERSWAN/COLOR (0 errors out of 23 games)

NES: (2 errors out of 1,756 roms)
FOLDERS: 100%

in (1) US: LICENSED NES GAMES
BATTLESHIP

in (2) TRANSLATED GAMES
CONTRA [STARDUST CRUSADERS]

SNES: (25 errors out of 1,144 roms)
FOLDERS 100%

in (1) US: LICENSED SUPER NINTENDO GAMES
ACTRAISER I
BATMAN FOREVER - GB
BATTLETOADS & DOUBLE DRAGON: THE ULTIMATE TEAM - GB
BUBSY II (HAS GAME BOY ARTWORK?)
CAPTAIN AMERICA AND THE AVENGERS (HAS GAME BOY ARTWORK?)
CASPER (HAS GAMEBOY ARTWORK?)
CHAMPIONSHIP POOL (GB ARTWORK?)
CHOPLIFTER III: RESCUE SURVIVE (GB ARTWORK?)
CHUCK ROCK (GB ARTWORK?)
CLIFFHANGER (GB ARTWORK?)
COOL SPOT (GB ARTWORK?)
COOL WORLD (GB ARTWORK?)
CUTTHROAT ISLAND (GB ARTWORK?)
DAFFY DUCK: THE MARVIN MISSIONS - GB
DENNIS THE MENACE -GB
DESERT STRIKE: RETURN TO THE GULF - GB
ELITE SOCCER - GB
FACEBALL 2000 - GB
FIFA INTERNATIONAL SOCCER - GB
FOREMAN FOR REAL - GB
FRANK THOMAS BIG HURT BASEBALL - GB
THE CHESSMASTER - GB

in (3) EUROPE: LICENSED SUPER NINTENDO GAMES
ASTERIX - GB

in (2) TRANSLATED GAMES
BREATH OF FIRE II - SHIMEI NO KO

in (02) PROTOTYPE GAMES
BATMAN - GB

GB: (11 errors out of 668 roms)
FOLDERS 100%

in (1) US: LICENSED GAME BOY GAMES
DISNEY'S TAILSPIN (TURBOGRAFX 16 BOX?)
HATRIS (TG16)
HIT THE ICE - TG
JIMMY CONNORS' PRO TENNIS TOUR (JAPAN BOX?)
KID DRACULA - TG
KLAX - TG
SHANGHAI - TG
SPOT'S COOL ADVENTURE - (IS THIS A GAMEBOY BOX?)
TURRICAN - TG

in (2) EUROPE: LICENSED GAME BOY GAMES
PARASOL STARS: RAINBOW ISLANDS II - TG
POPULUS - TG

GBA: (3 errors out of 1,119 roms)
FOLDERS: 100%

in (1) US: LICENSED GAME BOY ADVANCED GAMES
OVER THE HEDGE - NINTENDO DS BOXART?
SCURGE: HIVE (DS BOXART)

in (2) EUROPE: LICENSED GAME BOY ADVANCED GAMES
MY VET PRACTICE (DS BOXART)

SEGA GENESIS: (3 errors out of 959 roms)
FOLDERS: 100%

in (1) US: LICENSED SEGA GENESIS GAMES
DYNAMITE HEADDY

in (3) EUROPE: LICENSED SEGA MEGADRIVE GAMES
TECMO CUP FOOTBALL GAME (NES BOXART???)

in (4) GAME HACKS
RINGS OF POWER - THE HANDS OF NEXUS

TURBO GRAFX 16/CD: (1 error out of 220 roms)
FOLDERS: 100%

in (1) US: LICENSED TURBOGRAFX 16 GAMES
BONK 3: BONK'S BIG ADVENTURE (TURBO CD BOXART?)

ODYSSEY 2: (1 error out of 91 roms)
FOLDER ISSUES:
(3) HOMEBREW & HACKS (No artwork)

ROM ISSUES: NONE

****NOTE: Might have to unzip these games. They will not play on RetroPie as-is. Need more research on this.