Making premade gamelist.xml with xtra media collections

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX

@meleu Yes... I definitely think we need a way to simplify a command that makes a gamelist.xml for all of the systems.

I definitely think you need to invest some time to learn the basics of Linux shell. :-)

That huge command can be simplified to:
time ./Used2BeTXT.sh --no-desc /home/pi/RetroPie/Media/*/Synopsis/*.*
Also, don't know if you are aware, but you can auto-complete some commands using TAB key.

I've updated the script a few minutes ago. Small bugfixes and added the --only-image option (see --help to get details).

It was fun, I learned some find and xmlstarlet tricks, but I think I'm done with it. Time to move on...

Ah... didn't notice you posted this before I replied.

Still have some possible issues that I mentioned though. I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

Used2BeRX

@meleu I'm still making pics for the folders on about 3 more systems before I test this out, but the script finally finished!!! Just wanted to give you the time.

673m44s, 420m10s, 197m50s.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I think we effectively doubled the search time by adding the BIN files to the search for the 5200.

The searching for bin occurs only when platform is atari5200 or atari800.

Used2BeRX

@meleu 1st run through is done and the results are great overall.

100% HITS FOR THE FOLLOWING SYSTEMS!!!:
Atari 2600
Atari 7800
Atari LYNX
Game Gear
Gameboy Advance
Gameboy Color
Neo Geo Pocket/Neo Geo Pocket Color
Sega 32x
Sega SG-1000
Wonderswan/Wonderswan Color

MAJOR FULL-SYSTEM PROBLEMS:

SNES did not work. 338kb gamelist.xml file was created, but not a single rom or image was entered in the fields. Instead of writing to "snes_gamelist.xml" it wrote to "-supernintendoentertainmentsystem_gamelist.xml".
Odyssey 2 did not work. It makes magnavoxodyssey2_gamelist.xml, but RetroPie is expecting videopac for the folder, so it didn't find any roms or media.
Virtual Boy didn't work. It makes gamelist.xml with no other identifiers. RetroPie is expecting virtualboy for the folder.

FIXES TO SYNOPIS I MADE:
Changed "Platform: " on the following.
All "Atari 800" to "Atari 5200"
All "Sega Net" to "Sega Genesis / Mega Drive"
All "The Famicom Disk System" to "Nintendo Entertainment System"

Big thing to note is that you cannot have two 1st line synopsis.txt descriptions in the same gamelist.xml. It confuses the script. So I made the following changes by system to get the rest of the working systems up to 100%

TurboGrafx:
(2) Translated Hu-Card Games
(2) Translated CD Games
Bonk 3: Bonk's Big Adventure (CD)
Altered Beast (CD)

NES:
Metroid (Famicom Disk System)
Clu Clu Land (Famicom Disk System)
Gyruss (Famicom Disk System)
Kid Icarus (Famicom Disk System)
Super Mario Bros (Famicom Disk System)
Super Mario Bros 2 (Famicom Disk System)
Vs Excite Bike (Famicom Disk System)
AD&D Dragons of the Flame (Translated)
Battleship (Unlicensed)
Contra (Translated)
Maniac Mansion [Uncensored]
Ms. Pac-Man (Unlicensed)
Rainbow Islands (Europe)
Fantasy Zone I (Japan)
Super Mario Bros. 2 (Japan)
Othello (Australia)
Airwolf (Translated)
Contra (Translated)
Squoon (Translated)
Super Mario Bros. 2 (Prototype)
"Adult Games" 2/3rds of them were "Famicom Disk System" and showed up after changing this to "Nintendo Entertainment System"

GAME BOY:
Bomberman GB (Japan)

GENESIS:
Dynamite Headdy (Japan)
Rings of Power: The Hands of Fate (Hack)

MASTER SYSTEM:
Pit Pot..... accidentially had it listed as Game Gear.
Assault City (Light Phaser Version)

ATARI 5200:
Centipede (Atari 800)
Choplifter! (Atari 800)
Crystal Castles (Atari 800)
Mario Bros. (Atari 800)
Qix (Atari 800)
Rally Speedway (Atari 800)

Used2BeRX

@meleu Any ideas about SNES, Virtualboy and Odyssey 2, meleu?

I'll have to look deeper into those individual game errors on the other systems, but if you look at my last post you'll see a section for "Major Problems" detailing why I can't get any of those gamelist.xml files to work with the script.

It looks like based off of the Platform: field, it's not looking in the correct folders for roms or media.

meleu

@Used2BeRX I'll look into it when I have a chance, I think I know what's happening... No need to change txt files on your end.

meleu

@Used2BeRX I've fixed the snes issue and then the kids dragged me to play. :-)

Please test the snes synopsis conversion.

Used2BeRX

@meleu I already made all the synopsis changes on my end for the individual things above. Most notably changing all "Atari 800" to "Atari 5200", "Famicom Disk System" to "Nintendo Entertainment System" and "Sega Net" to "Sega Genesis / Mega Drive".

I'm actually in the process of re-running all of the gamelists right now. After making those changes to all of the games with the same name, I also had to delete a lot of entries from the original gamelists because they were bad. When it was getting confused, sometimes it was putting the Japanese or Translated version paths for the US version. But when I deleted that entry and re-ran the system, it was finding the other rom and not putting the US one in.

I don't know if I actually fixed anything by changing the name because of this, so I decided to re-run everything from scratch and then re-check the results and see where I land after all of these changes.

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

meleu

@Used2BeRX

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

I'm afraid that the wisest decision would be to cancel this execution, --update and run it again. :-)

By the way, in the pack of synopsis you sent me doesn't have synopsis for Virtual Boy games. Can you upload it somewhere?

Used2BeRX

@meleu Did you make any changes other than SNES? If not, then I'll let this run finish first, and then do the SNES after the update. I renamed the synopsis folders for SNES, Odyssey 2 and Virtual Boy to "Synopsis2" so it will skip them during this run.

If there is a specific reason otherwise that you think I should cancel though, I will do so.

Here's virtual boy synopsis: https://www.sendspace.com/file/29fay6

EDIT:

Yeah... actually it's probably a lot closer to finishing than I thought with those 3 folders removed. It just finished up nes, so all that is left is the smaller libraries of ngpc, pcengine, sega32x and wonderswan now. Should be done in 30-45 minutes tops.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Did you make any changes other than SNES?

I've just fixed SNES, Virtual Boy and Odissey2 platform issue. Test it when you have a chance.

Let me know if you have any new issue. I'm assuming those problems with specific games you solved on your end, right?

Used2BeRX

@meleu Oh cool. You got to all three of the others. I should be able to test them very soon.

I believe I solved the individual game problems, but using the command to update the xml files wasn't working for me since the original problem was making totally erroneous entries, so I started over.

I should have that answer for you later today. I just have to go through every game again and see if any errors pop up.

Used2BeRX

@meleu Hey bud. Great update there. SNES, Odyssey 2 and Virtual Boy script works now. :)

I'm posting my current results here just for the record. I don't believe at this time that there is anything to do on your end, but I have a lot of individual things to go through. I noticed that some games on some systems are using artwork for other systems. I need to go in and see if this is a mistake in the artwork packs themselves or just a wrong entry in the synopsis entries for Platform: .

The only possible thing at this point that I think we may need to address to finally get 100% on this is on the Odyssey 2. ZIP files are included in the es_systems.cfg file since they show up on the romlist, but I don't believe the system will actually play zipped roms since none of them work and I know they're good roms. I will need to research that sometime this week and get back to you about it.

Anyway.... Here's my current results.

6-25-17 script test:

SYSTEMS WITH 100% NO ERRORS:

GAME BOY COLOR (0 errors of 596 roms)
SEGA 32X (0 errors out of 39 roms)
SEGA MASTER SYSTEM (0 errors out of 319 roms)
GAME GEAR (0 errors out of 308 roms)
SEGA SG-1000 (0 errors out of 65 roms)
ATARI 2600 (0 errors out of 724 roms)
ATARI 5200 (0 errors out of 263 roms)
ATARI 7800 (0 errors out of 86 roms)
ATARI LYNX (0 errors out of 86 roms)
NEO GEO POCKET/COLOR (0 errors out of 49 roms)
VIRTUAL BOY (0 errors out of 27 roms)
WONDERSWAN/COLOR (0 errors out of 23 games)

NES: (2 errors out of 1,756 roms)
FOLDERS: 100%

in (1) US: LICENSED NES GAMES
BATTLESHIP

in (2) TRANSLATED GAMES
CONTRA [STARDUST CRUSADERS]

SNES: (25 errors out of 1,144 roms)
FOLDERS 100%

in (1) US: LICENSED SUPER NINTENDO GAMES
ACTRAISER I
BATMAN FOREVER - GB
BATTLETOADS & DOUBLE DRAGON: THE ULTIMATE TEAM - GB
BUBSY II (HAS GAME BOY ARTWORK?)
CAPTAIN AMERICA AND THE AVENGERS (HAS GAME BOY ARTWORK?)
CASPER (HAS GAMEBOY ARTWORK?)
CHAMPIONSHIP POOL (GB ARTWORK?)
CHOPLIFTER III: RESCUE SURVIVE (GB ARTWORK?)
CHUCK ROCK (GB ARTWORK?)
CLIFFHANGER (GB ARTWORK?)
COOL SPOT (GB ARTWORK?)
COOL WORLD (GB ARTWORK?)
CUTTHROAT ISLAND (GB ARTWORK?)
DAFFY DUCK: THE MARVIN MISSIONS - GB
DENNIS THE MENACE -GB
DESERT STRIKE: RETURN TO THE GULF - GB
ELITE SOCCER - GB
FACEBALL 2000 - GB
FIFA INTERNATIONAL SOCCER - GB
FOREMAN FOR REAL - GB
FRANK THOMAS BIG HURT BASEBALL - GB
THE CHESSMASTER - GB

in (3) EUROPE: LICENSED SUPER NINTENDO GAMES
ASTERIX - GB

in (2) TRANSLATED GAMES
BREATH OF FIRE II - SHIMEI NO KO

in (02) PROTOTYPE GAMES
BATMAN - GB

GB: (11 errors out of 668 roms)
FOLDERS 100%

in (1) US: LICENSED GAME BOY GAMES
DISNEY'S TAILSPIN (TURBOGRAFX 16 BOX?)
HATRIS (TG16)
HIT THE ICE - TG
JIMMY CONNORS' PRO TENNIS TOUR (JAPAN BOX?)
KID DRACULA - TG
KLAX - TG
SHANGHAI - TG
SPOT'S COOL ADVENTURE - (IS THIS A GAMEBOY BOX?)
TURRICAN - TG

in (2) EUROPE: LICENSED GAME BOY GAMES
PARASOL STARS: RAINBOW ISLANDS II - TG
POPULUS - TG

GBA: (3 errors out of 1,119 roms)
FOLDERS: 100%

in (1) US: LICENSED GAME BOY ADVANCED GAMES
OVER THE HEDGE - NINTENDO DS BOXART?
SCURGE: HIVE (DS BOXART)

in (2) EUROPE: LICENSED GAME BOY ADVANCED GAMES
MY VET PRACTICE (DS BOXART)

SEGA GENESIS: (3 errors out of 959 roms)
FOLDERS: 100%

in (1) US: LICENSED SEGA GENESIS GAMES
DYNAMITE HEADDY

in (3) EUROPE: LICENSED SEGA MEGADRIVE GAMES
TECMO CUP FOOTBALL GAME (NES BOXART???)

in (4) GAME HACKS
RINGS OF POWER - THE HANDS OF NEXUS

TURBO GRAFX 16/CD: (1 error out of 220 roms)
FOLDERS: 100%

in (1) US: LICENSED TURBOGRAFX 16 GAMES
BONK 3: BONK'S BIG ADVENTURE (TURBO CD BOXART?)

ODYSSEY 2: (1 error out of 91 roms)
FOLDER ISSUES:
(3) HOMEBREW & HACKS (No artwork)

ROM ISSUES: NONE

****NOTE: Might have to unzip these games. They will not play on RetroPie as-is. Need more research on this.

meleu

@Used2BeRX please post the problematic gamelist.xml files on ghostbin.com or something so I can take a look.

Used2BeRX

@meleu Will do. No problems on your end yet. I have some preparing to do for company tomorrow so I may or may not have a chance to get around to in-depth testing right now. I'm thinking that at least 90% of the issues I'm experiencing right now are problems on my end. If I get a chance to look at it tonight I will let you know more then, otherwise it might be a day or two before I can get back to it.

Thanks again man. This is freakin awesome!

Used2BeRX

@meleu Well I got the final results and it's all good news man! All of the issues were on my end and they've been resolved so any future runs should be issue free. :)

SYSTEMS WITH 100% NO ERRORS: ALL TESTED SYSTEMS!!!!

NES: (0 errors out of 1,756 roms)
SNES: (0 errors out of 1,144 roms)
GAME BOY: (0 errors out of 668 roms)
GAME BOY ADVANCE: (0 errors out of 1,119 roms)
GAME BOY COLOR: (0 errors out of 596 roms)
SEGA 32X: (0 errors out of 39 roms)
SEGA MASTER SYSTEM: (0 errors out of 319 roms)
GAME GEAR: (0 errors out of 308 roms)
SEGA SG-1000: (0 errors out of 65 roms)
ATARI 2600: (0 errors out of 724 roms)
ATARI 5200: (0 errors out of 263 roms)
ATARI 7800: (0 errors out of 86 roms)
ATARI LYNX: (0 errors out of 86 roms)
TURBOGRAFX 16: (0 errors out of 220 roms)
NEO GEO POCKET/COLOR: (0 errors out of 49 roms)
ODYSSEY 2: (0 error out of 91 roms)
VIRTUAL BOY: (0 errors out of 27 roms)
WONDERSWAN/COLOR: (0 errors out of 23 games)

TOTAL: (0 errors out of 7,583)

I can't thank you enough meleu. My bro is going to be here tomorrow and I'll get to show him everything that's been upgraded on his Pi. He gave it to me months ago to just take a look at and see if I could figure out how to get the NES and SNES games running properly, but he's getting a lot more than that now.

I'm hoping that you'll be around to make some upgrades to your badass script now that we've got it running perfectly on a --no-desc run.

Things I'd like to work on in the future.

Making sure that videos work with the script when I get a Pi 3 to test this all out on.
Making sure that CD based games like SegaCD, TurboGrafxCD and PSX work with the script.
Making options to quickly convert the artwork that is displayed in RetroPie after the gamelist.xml files have been created by replacing which image type has the <image> tags (but only alter these in <game> entries and not the <folder> entries). This last option would also be a very quick way for me to verify that all other types of artwork are working with the script, and not just the Box Front.

This really is so awesome man. Thank you, and thanks again to UDb23 for getting the ball rolling on this. You guys are the best!!!

cafarellidigital

Hey everyone, I started following this in the beginning, but I've been away and busy the last few weeks, and lost track of where this is at now. It seems it's been successful, but not sure what that means.

Are there now premade gamelist.xml files along with images and videos? Are these available for download? If not now, will they be eventually?

Used2BeRX

@cafarellidigital Right now, it's more of a "proof of concept" that has worked.

The only thing that I have tested that works for sure is the option of the script that just includes the rom path and the path to the Box Front image. Still untested are all other types of artwork, videos and other media that is not currently even usable in EmulationStation and would need to have functions coded in to make use of.

What I have right now is pretty much perfect for the Pi Zero. Low-res boxart and minimal gamelist.xml files that do not have any of the game descriptions in them to keep their size as small as possible.

The great news though is that I have now proven that you can have gamelist entries for nearly 8,000 games over nearly 20 console systems and display small boxart without making RetroPie hang, and without breaking any of the emulator functions.

When I was using artwork that hadn't been resized (to 180x240, 240x180 or 210x210), and I had full XMV files with the game description and all of the media being pointed to for each game, the EmulationStation was seriously lagging on startup, I would have major slow downs while scrolling through games, and the Atari 5200 and SNES emulators even refused to launch any games.

We're still quite a way from having anything uploaded and ready to go. I'm open to doing that, but I'd need to know a place here where literally 100,000 to 200,000 or more pieces could be hosted.

Given that I cannot share roms here, there would also have to be some sort of way to get people to have the roms they provide in the correct folders, since there are easily over 100-150 sub folders throughout the different console systems included. To my knowledge, there is no way to create datfiles that have sub-folders for the roms. (If anybody knows a way to do this, please let me know).

Without these sub-folders for the games, you would have a very messy romlist that included the original official games, games from other countries, translated games, prototypes, pirates, game hacks and others all in a single folder. For the NES that would be a single folder of nearly 1,800 games to scroll through of all different varieties.

Right now I am working with a guy to "upgrade" our sets to the no-intro standard as much as possible. Many of our roms are not in no-intro like game hacks and translations and such, but the goal is to at least have a no-intro rom for all of the official releases by region. This isn't for me, since I know the roms I have work and are personally tested by me on another platform. This is to make it easier for people using a datfile to match their rom names to ours so they would work with the sets.

There's still a ton of work to be done, and a ton more getting these things available someplace online where they won't be taken down and setting it up in such a way where it's not still a huge chore for the end user.

I'm open to any suggestions you may have about any of that.

mediamogul

@Used2BeRX

getting these things available someplace online where they won't be taken down

Why would anything be taken down?

Used2BeRX

@mediamogul I can't think of any specific reason since there wouldn't be roms. Torrent sites over the years had a lot of it, but they're all shut down now. We used to upload to places like Megaupload back in the day too and all of the sudden links would disappear. It takes a lot of effort to make stuff like this publicly available only to see it disappear after the fact. I'd need to be relatively confident that they weren't going anywhere for a long while before I put all that effort into something again.

EDIT: Believe it or not, providing this stuff without roms is much more difficult for me than with them, so there is a lot of extra work that goes into providing this stuff without them.

I could just dump it all in zips somewhere, but what good is that if none of it is easily usable. Without spending tons of hours making the datfiles that people could actually use their roms with our stuff, they would have to find and rename nearly 10,000 roms themselves by hand.

That's a big reason why we're working on at least getting the offical releases upgraded to no-intro since many of them were not part of no-intro, tosec or goodsets. That right there is proving to be a HUGE effort that is of no benefit to me personally since I know the roms we have are fine.

Unless somebody knows of a way to make datfiles that will read roms in sub folders and place them in sub-folders for the end user, I would also have to create a dat-file for every single sub-folder, and that is not a quick or easy process in and of itself.

EDIT2: I'm not bitching or anything. Just explaining that this is going to be a major project, and letting anybody interested know not to expect anything to happen soon. We're a long way from there. It's still in a conceptual state. I'm not even sure what the best way to go about it would be.

Of course the easiest way would be to just make an image of my setup when it's completed that somebody could just burn to an SD card and be done with it, but even if that were possible here I'm still a long way off from having even just that.