Making premade gamelist.xml with xtra media collections

meleu

@Used2BeRX I'll look into it when I have a chance, I think I know what's happening... No need to change txt files on your end.

meleu

@Used2BeRX I've fixed the snes issue and then the kids dragged me to play. :-)

Please test the snes synopsis conversion.

Used2BeRX

@meleu I already made all the synopsis changes on my end for the individual things above. Most notably changing all "Atari 800" to "Atari 5200", "Famicom Disk System" to "Nintendo Entertainment System" and "Sega Net" to "Sega Genesis / Mega Drive".

I'm actually in the process of re-running all of the gamelists right now. After making those changes to all of the games with the same name, I also had to delete a lot of entries from the original gamelists because they were bad. When it was getting confused, sometimes it was putting the Japanese or Translated version paths for the US version. But when I deleted that entry and re-ran the system, it was finding the other rom and not putting the US one in.

I don't know if I actually fixed anything by changing the name because of this, so I decided to re-run everything from scratch and then re-check the results and see where I land after all of these changes.

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

meleu

@Used2BeRX

I won't probably be able to test out the SNES for at least 6 hours, but I'll let you know.

I'm afraid that the wisest decision would be to cancel this execution, --update and run it again. :-)

By the way, in the pack of synopsis you sent me doesn't have synopsis for Virtual Boy games. Can you upload it somewhere?

Used2BeRX

@meleu Did you make any changes other than SNES? If not, then I'll let this run finish first, and then do the SNES after the update. I renamed the synopsis folders for SNES, Odyssey 2 and Virtual Boy to "Synopsis2" so it will skip them during this run.

If there is a specific reason otherwise that you think I should cancel though, I will do so.

Here's virtual boy synopsis: https://www.sendspace.com/file/29fay6

EDIT:

Yeah... actually it's probably a lot closer to finishing than I thought with those 3 folders removed. It just finished up nes, so all that is left is the smaller libraries of ngpc, pcengine, sega32x and wonderswan now. Should be done in 30-45 minutes tops.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu Did you make any changes other than SNES?

I've just fixed SNES, Virtual Boy and Odissey2 platform issue. Test it when you have a chance.

Let me know if you have any new issue. I'm assuming those problems with specific games you solved on your end, right?

Used2BeRX

@meleu Oh cool. You got to all three of the others. I should be able to test them very soon.

I believe I solved the individual game problems, but using the command to update the xml files wasn't working for me since the original problem was making totally erroneous entries, so I started over.

I should have that answer for you later today. I just have to go through every game again and see if any errors pop up.

Used2BeRX

@meleu Hey bud. Great update there. SNES, Odyssey 2 and Virtual Boy script works now. :)

I'm posting my current results here just for the record. I don't believe at this time that there is anything to do on your end, but I have a lot of individual things to go through. I noticed that some games on some systems are using artwork for other systems. I need to go in and see if this is a mistake in the artwork packs themselves or just a wrong entry in the synopsis entries for Platform: .

The only possible thing at this point that I think we may need to address to finally get 100% on this is on the Odyssey 2. ZIP files are included in the es_systems.cfg file since they show up on the romlist, but I don't believe the system will actually play zipped roms since none of them work and I know they're good roms. I will need to research that sometime this week and get back to you about it.

Anyway.... Here's my current results.

6-25-17 script test:

SYSTEMS WITH 100% NO ERRORS:

GAME BOY COLOR (0 errors of 596 roms)
SEGA 32X (0 errors out of 39 roms)
SEGA MASTER SYSTEM (0 errors out of 319 roms)
GAME GEAR (0 errors out of 308 roms)
SEGA SG-1000 (0 errors out of 65 roms)
ATARI 2600 (0 errors out of 724 roms)
ATARI 5200 (0 errors out of 263 roms)
ATARI 7800 (0 errors out of 86 roms)
ATARI LYNX (0 errors out of 86 roms)
NEO GEO POCKET/COLOR (0 errors out of 49 roms)
VIRTUAL BOY (0 errors out of 27 roms)
WONDERSWAN/COLOR (0 errors out of 23 games)

NES: (2 errors out of 1,756 roms)
FOLDERS: 100%

in (1) US: LICENSED NES GAMES
BATTLESHIP

in (2) TRANSLATED GAMES
CONTRA [STARDUST CRUSADERS]

SNES: (25 errors out of 1,144 roms)
FOLDERS 100%

in (1) US: LICENSED SUPER NINTENDO GAMES
ACTRAISER I
BATMAN FOREVER - GB
BATTLETOADS & DOUBLE DRAGON: THE ULTIMATE TEAM - GB
BUBSY II (HAS GAME BOY ARTWORK?)
CAPTAIN AMERICA AND THE AVENGERS (HAS GAME BOY ARTWORK?)
CASPER (HAS GAMEBOY ARTWORK?)
CHAMPIONSHIP POOL (GB ARTWORK?)
CHOPLIFTER III: RESCUE SURVIVE (GB ARTWORK?)
CHUCK ROCK (GB ARTWORK?)
CLIFFHANGER (GB ARTWORK?)
COOL SPOT (GB ARTWORK?)
COOL WORLD (GB ARTWORK?)
CUTTHROAT ISLAND (GB ARTWORK?)
DAFFY DUCK: THE MARVIN MISSIONS - GB
DENNIS THE MENACE -GB
DESERT STRIKE: RETURN TO THE GULF - GB
ELITE SOCCER - GB
FACEBALL 2000 - GB
FIFA INTERNATIONAL SOCCER - GB
FOREMAN FOR REAL - GB
FRANK THOMAS BIG HURT BASEBALL - GB
THE CHESSMASTER - GB

in (3) EUROPE: LICENSED SUPER NINTENDO GAMES
ASTERIX - GB

in (2) TRANSLATED GAMES
BREATH OF FIRE II - SHIMEI NO KO

in (02) PROTOTYPE GAMES
BATMAN - GB

GB: (11 errors out of 668 roms)
FOLDERS 100%

in (1) US: LICENSED GAME BOY GAMES
DISNEY'S TAILSPIN (TURBOGRAFX 16 BOX?)
HATRIS (TG16)
HIT THE ICE - TG
JIMMY CONNORS' PRO TENNIS TOUR (JAPAN BOX?)
KID DRACULA - TG
KLAX - TG
SHANGHAI - TG
SPOT'S COOL ADVENTURE - (IS THIS A GAMEBOY BOX?)
TURRICAN - TG

in (2) EUROPE: LICENSED GAME BOY GAMES
PARASOL STARS: RAINBOW ISLANDS II - TG
POPULUS - TG

GBA: (3 errors out of 1,119 roms)
FOLDERS: 100%

in (1) US: LICENSED GAME BOY ADVANCED GAMES
OVER THE HEDGE - NINTENDO DS BOXART?
SCURGE: HIVE (DS BOXART)

in (2) EUROPE: LICENSED GAME BOY ADVANCED GAMES
MY VET PRACTICE (DS BOXART)

SEGA GENESIS: (3 errors out of 959 roms)
FOLDERS: 100%

in (1) US: LICENSED SEGA GENESIS GAMES
DYNAMITE HEADDY

in (3) EUROPE: LICENSED SEGA MEGADRIVE GAMES
TECMO CUP FOOTBALL GAME (NES BOXART???)

in (4) GAME HACKS
RINGS OF POWER - THE HANDS OF NEXUS

TURBO GRAFX 16/CD: (1 error out of 220 roms)
FOLDERS: 100%

in (1) US: LICENSED TURBOGRAFX 16 GAMES
BONK 3: BONK'S BIG ADVENTURE (TURBO CD BOXART?)

ODYSSEY 2: (1 error out of 91 roms)
FOLDER ISSUES:
(3) HOMEBREW & HACKS (No artwork)

ROM ISSUES: NONE

****NOTE: Might have to unzip these games. They will not play on RetroPie as-is. Need more research on this.

meleu

@Used2BeRX please post the problematic gamelist.xml files on ghostbin.com or something so I can take a look.

Used2BeRX

@meleu Will do. No problems on your end yet. I have some preparing to do for company tomorrow so I may or may not have a chance to get around to in-depth testing right now. I'm thinking that at least 90% of the issues I'm experiencing right now are problems on my end. If I get a chance to look at it tonight I will let you know more then, otherwise it might be a day or two before I can get back to it.

Thanks again man. This is freakin awesome!

Used2BeRX

@meleu Well I got the final results and it's all good news man! All of the issues were on my end and they've been resolved so any future runs should be issue free. :)

SYSTEMS WITH 100% NO ERRORS: ALL TESTED SYSTEMS!!!!

NES: (0 errors out of 1,756 roms)
SNES: (0 errors out of 1,144 roms)
GAME BOY: (0 errors out of 668 roms)
GAME BOY ADVANCE: (0 errors out of 1,119 roms)
GAME BOY COLOR: (0 errors out of 596 roms)
SEGA 32X: (0 errors out of 39 roms)
SEGA MASTER SYSTEM: (0 errors out of 319 roms)
GAME GEAR: (0 errors out of 308 roms)
SEGA SG-1000: (0 errors out of 65 roms)
ATARI 2600: (0 errors out of 724 roms)
ATARI 5200: (0 errors out of 263 roms)
ATARI 7800: (0 errors out of 86 roms)
ATARI LYNX: (0 errors out of 86 roms)
TURBOGRAFX 16: (0 errors out of 220 roms)
NEO GEO POCKET/COLOR: (0 errors out of 49 roms)
ODYSSEY 2: (0 error out of 91 roms)
VIRTUAL BOY: (0 errors out of 27 roms)
WONDERSWAN/COLOR: (0 errors out of 23 games)

TOTAL: (0 errors out of 7,583)

I can't thank you enough meleu. My bro is going to be here tomorrow and I'll get to show him everything that's been upgraded on his Pi. He gave it to me months ago to just take a look at and see if I could figure out how to get the NES and SNES games running properly, but he's getting a lot more than that now.

I'm hoping that you'll be around to make some upgrades to your badass script now that we've got it running perfectly on a --no-desc run.

Things I'd like to work on in the future.

Making sure that videos work with the script when I get a Pi 3 to test this all out on.
Making sure that CD based games like SegaCD, TurboGrafxCD and PSX work with the script.
Making options to quickly convert the artwork that is displayed in RetroPie after the gamelist.xml files have been created by replacing which image type has the <image> tags (but only alter these in <game> entries and not the <folder> entries). This last option would also be a very quick way for me to verify that all other types of artwork are working with the script, and not just the Box Front.

This really is so awesome man. Thank you, and thanks again to UDb23 for getting the ball rolling on this. You guys are the best!!!

cafarellidigital

Hey everyone, I started following this in the beginning, but I've been away and busy the last few weeks, and lost track of where this is at now. It seems it's been successful, but not sure what that means.

Are there now premade gamelist.xml files along with images and videos? Are these available for download? If not now, will they be eventually?

Used2BeRX

@cafarellidigital Right now, it's more of a "proof of concept" that has worked.

The only thing that I have tested that works for sure is the option of the script that just includes the rom path and the path to the Box Front image. Still untested are all other types of artwork, videos and other media that is not currently even usable in EmulationStation and would need to have functions coded in to make use of.

What I have right now is pretty much perfect for the Pi Zero. Low-res boxart and minimal gamelist.xml files that do not have any of the game descriptions in them to keep their size as small as possible.

The great news though is that I have now proven that you can have gamelist entries for nearly 8,000 games over nearly 20 console systems and display small boxart without making RetroPie hang, and without breaking any of the emulator functions.

When I was using artwork that hadn't been resized (to 180x240, 240x180 or 210x210), and I had full XMV files with the game description and all of the media being pointed to for each game, the EmulationStation was seriously lagging on startup, I would have major slow downs while scrolling through games, and the Atari 5200 and SNES emulators even refused to launch any games.

We're still quite a way from having anything uploaded and ready to go. I'm open to doing that, but I'd need to know a place here where literally 100,000 to 200,000 or more pieces could be hosted.

Given that I cannot share roms here, there would also have to be some sort of way to get people to have the roms they provide in the correct folders, since there are easily over 100-150 sub folders throughout the different console systems included. To my knowledge, there is no way to create datfiles that have sub-folders for the roms. (If anybody knows a way to do this, please let me know).

Without these sub-folders for the games, you would have a very messy romlist that included the original official games, games from other countries, translated games, prototypes, pirates, game hacks and others all in a single folder. For the NES that would be a single folder of nearly 1,800 games to scroll through of all different varieties.

Right now I am working with a guy to "upgrade" our sets to the no-intro standard as much as possible. Many of our roms are not in no-intro like game hacks and translations and such, but the goal is to at least have a no-intro rom for all of the official releases by region. This isn't for me, since I know the roms I have work and are personally tested by me on another platform. This is to make it easier for people using a datfile to match their rom names to ours so they would work with the sets.

There's still a ton of work to be done, and a ton more getting these things available someplace online where they won't be taken down and setting it up in such a way where it's not still a huge chore for the end user.

I'm open to any suggestions you may have about any of that.

mediamogul

@Used2BeRX

getting these things available someplace online where they won't be taken down

Why would anything be taken down?

Used2BeRX

@mediamogul I can't think of any specific reason since there wouldn't be roms. Torrent sites over the years had a lot of it, but they're all shut down now. We used to upload to places like Megaupload back in the day too and all of the sudden links would disappear. It takes a lot of effort to make stuff like this publicly available only to see it disappear after the fact. I'd need to be relatively confident that they weren't going anywhere for a long while before I put all that effort into something again.

EDIT: Believe it or not, providing this stuff without roms is much more difficult for me than with them, so there is a lot of extra work that goes into providing this stuff without them.

I could just dump it all in zips somewhere, but what good is that if none of it is easily usable. Without spending tons of hours making the datfiles that people could actually use their roms with our stuff, they would have to find and rename nearly 10,000 roms themselves by hand.

That's a big reason why we're working on at least getting the offical releases upgraded to no-intro since many of them were not part of no-intro, tosec or goodsets. That right there is proving to be a HUGE effort that is of no benefit to me personally since I know the roms we have are fine.

Unless somebody knows of a way to make datfiles that will read roms in sub folders and place them in sub-folders for the end user, I would also have to create a dat-file for every single sub-folder, and that is not a quick or easy process in and of itself.

EDIT2: I'm not bitching or anything. Just explaining that this is going to be a major project, and letting anybody interested know not to expect anything to happen soon. We're a long way from there. It's still in a conceptual state. I'm not even sure what the best way to go about it would be.

Of course the easiest way would be to just make an image of my setup when it's completed that somebody could just burn to an SD card and be done with it, but even if that were possible here I'm still a long way off from having even just that.

cafarellidigital

@Used2BeRX ahh ok, thanks for that great explanation!

Regarding the usability of this "database", I understand what you mean when you say that it's easier to provide it with the roms than without, but isn't this mostly due to file and folder naming conventions? And wouldn't it be easier to rename my rom files to accommodate your conventions?

For instance, I only have about 50 NES roms. That's not that many to rename. If you have a gamelist.xml with perfect consistency and images/videos/marquees of consistent size and quality, I think it would be infinitely easier to just rename my roms to match, no?

Used2BeRX

@cafarellidigital It would definately be easier for you to rename your roms in any case, but with only 50 games it wouldn't be hard at all.... In theory.

The problem comes in the file locations for your roms and media. If you had the media from any future packs and put them in the right spot, the media would not be a problem, but you would likely want to go in after the fact and delete a ton of it to not waste space on your Pi. The artwork really doesn't take up all that much space, but without checking I would say that the videos for the NES collection alone would probably be somewhere in the area of 10-20GB for all 1,700+ games, and since you'd only be using 50 of them that's a ton of wasted space.

The other problem comes in the file structure for your roms. Because of the nature of the gamelist.xml files, if your names aren't a perfect case match in the EXACT folders that mine were in, they will just show the generic rom name in the list without benefiting from any of the media.

For example, if you just had 50 roms in the /home/pi/RetroPie/roms/nes/ directory, you would not see any of the work. If they were all US roms, they would need to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/. It gets even more complicated if some of the games you had were from Europe, or were translations. Europe games would have to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_4) Europe Licensed/, and translations would need to be in /home/pi/RetroPie/roms/nes/(2) Translated/`.

I hope that paints a better picture of how difficult this will be to try to organize and present in a way that is worth the time for the end user when it needs to be done without roms included. :)

Used2BeRX

@meleu Hey meleu. Did you say that the script is set to look for CUE files when doing CD systems like Sega CD, Turbografx CD and PSX?

I just discovered that the Pi Zero amazingly seems to play Sega CD games perfectly (at least the few that I tried). There is no way I could fit them all on the small SD card I had, but I'm going to copy over a bunch of txt files that are renamed to the exact CD/synopsis names I have with a .CUE extension and run your script to get the gamelist.xml entries and the path and box front info in them.

Then I was going to put a few games on there at a time and test out that they're all working right. :)

Used2BeRX

@Used2BeRX Wow.... I don't believe it, but every single game for both the SegaCD and the TurboGrafx CD that I've tried works just fine on a Pi Zero! I've had this thing for months and didn't even bother testing either of those systems. Mainly because I didn't have the proper images (mine were ISO/MP3/CUE).

Also, since the Pi Zero struggles with NES and SNES emulation unless you're using quicknes and pisnes, I didn't think there was any way that a 500MB+ CD based game was going to have a chance on this system.

So... Once I get everything properly renamed, here's what I'm going to do....

I'm going to copy all of the CUE sheets to a mirrored folder layout.
I'm going to copy these mirrored layouts to the proper place that the folder synopsis TXT files are assuming the games will be.
If the script is set up right and it will look for .CUE files on SegaCD and Turbo Grafx CD, the gamelist.xml should add all of these games even if the BIN or ISO is not there.
Then I'll be able to do groups of games to test to make sure they're all working.

As far as I know, there is no compatibility spreadsheet for the Pi Zero and Sega CD/TurboGrafx CD.... correct guys?

Let me know if you're interested. I'll make sure the spreadsheet doesn't have anything linking to any roms or anything like that on it. I copy and pasted this stuff from an old spreadsheet of ours from about a decade ago, so there may be some identifying things in there that I will make sure to remove before giving this to you.

I'd say if a game is marked as working on the Pi Zero that there shouldn't be any problems with it on any Pi that's more powerful. :)

meleu

@Used2BeRX please, consider learning the basics of github and then create a repository for your synopsis text files there. I think it would be useful for the community (and would make me recover the will to work on that script :) ).