Making premade gamelist.xml with xtra media collections

Used2BeRX

@UDb23 I got pretty much everything re-cased and compared now for both the PiZero and Pi3 setups. I'm still waiting on changing the highest quality artwork to JPG to see if you can make the program recognize what type of file is in the media folders since the original PNG files would look better.

I'm ready to start making those synopsis.txt files for all of the folders whenever you guys have directions for me. Turns out there is going to be quite a bit of rain the next 3-4 days so the gig I had is going to be put off for a few more days at least.

I also had a very long list I put together of about 500-600 romlist names that I wanted to change in those synopsis files a few months back that I finally got around to doing today. I'm not going to say that the names are perfect across nearly 10,000 games, but they're pretty damn close. I tried to make them exactly how they are displayed on the boxart for the games. I've been through that list about 5 or 6 times over the years and I'm really happy with how it looks now.

Looking forward to being able to test more out and see if having all this artwork is going to break my Pi Zero or not. :)

meleu

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).
the art files.
where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).
please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

mediamogul

@meleu

I'm ready to start.
Here's what I need...

alt text

phulshof

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

lilbud

@mediamogul I love GOTG

meleu

@phulshof said in Making premade gamelist.xml with xtra media collections:

What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?

I kinda have the same curiosity...

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).

the art files.

where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).

please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

The gamelists I made before by hand won't work with the artwork now because I made everything upper case. For test purposes you could download one of the 7800 packs I put up, but be aware that everything that will be happening in the future is designed with all uppercase file names and extensions in mind. I could upload a quick pack with the uppercase, but it would not work with the gamelist.xml.

The art files would be in ~/RetroPie/Media/{SYSTEM_NAME}/Artwork/
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/3D Boxart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Action
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Box Front
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Cart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Console
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Titles

(/Artwork/Console/ is a folder that will have the artwork meant to be displayed when resting on a folder or sub-folder. I still need to know how I should make the synopsis.txt files for folders)

You won't need me to rename the systems for a while, since you're just looking at the Atari 7800 pack right now. I thought I had everything prepared for this until I read that. That's going to be a big ass pain for me to do since they were all set up that way for the other system. Quite a few "systems" we had are mixed with other systems because the emulators played several different systems. For example Master System/Game Gear/SG-1000 are all grouped together, Genesis/32x/Sega CD are all grouped together, etc. And I'll get to do it twice because there is a low def set and a high def set. Fun times.

Anyway. Here's the link: https://mega.nz/#!QKxmlCYL!z5ck_O7L7pNzL19Z4HdMZ64FikmYGC7J1mpfXOGvEsk

EDIT:

Renaming the folders themselves to make them match the RetroPie wont' be a big deal. That being said, I thought about it and I will not be separating any of the media. That introduces far too many possibilities for there being problems in a setup that I know is 100% correct at the moment.

So all media for
Gameboy, Gameboy Advance and Gameboy Color will be in /Media/gb/
Genesis, 32x, Sega CD will be in /Media/megadrive/
Master System, Game Gear, SG-1000 will be in /Media/mastersystem/
Neo Geo Pocket, Neo Geo Pocket color will be in /Media/ngp/
Turbografx, TurboCD will be in /Media/pcengine/

If I absolutely need to make these name changes for the media folders, that's the best I can do.

Used2BeRX

@phulshof said in Making premade gamelist.xml with xtra media collections:

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

It will be downloadable somewhere. I can't give the link out here though.
None of this will have anything to do with MAME. I am unaware what a MAME software list is.
I dunno. I like it.

meleu

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.
Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".
Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).
Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.

Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".

Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).

Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

It's data. We never used xml though. Our system also relied on an individual games crc values and the file name for all extras and data so we didn't have any reason to load every single piece of information for every single game when the system booted. The pi seems rather inefficient that way, but I'm sure since it's still getting upgrades that eventually that won't be a problem.
Yes, although a factor of 10 may have slightly overstated it. I was not happy with the in-menu scraper or the other not so intuitive method compared to what we have. As stated above I'm not a fan of the XML necessity at all, but I'm trying my best here to integrate the work we've already done. It's like having a bunch of kids and moving to a new country and making them all learn to speak the language.
Yes. UDb23 already made that happen on a 1-to-1 basis, as well as adding in pointers to media (In the folders I already have, not the ones you're requesting I switch everything to).
Yes.
"meleu, can you help us to______________?"

I don't know the answer to that. I don't know what UDb23 is asking you to do either.

phulshof

@Used2BeRX

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@phulshof said in Making premade gamelist.xml with xtra media collections:

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

It will be downloadable somewhere. I can't give the link out here though.

None of this will have anything to do with MAME. I am unaware what a MAME software list is.

I dunno. I like it.

I guess my reason for asking is that the general problem people encounter is to find the games with matching database information and matching art work. Apparently you have a set, which, if made available, people would have to match with their games.

As an alternative, HyperSpin has created XML files with metadata for many systems, with corresponding artwork from their own site and EmuMovies, where matching games can easily be found. These XML files have been filtered for duplicate games and non-playable games (to us non-Japanese speaking gamers).

MAME also supports consoles/home computers/handhelds via their software lists, which contains meta information, and for which matching games can easily be found. Artwork is handled via MAME EXTRAs, though is severely lacking (I spent a lot of time creating thousands of screenshots/titleshots for MSX emulation for instance).

Can you give an indication of the number of games per system you have meta information on? It would give an indication of how it compares to the HyperSpin and MAME alternatives. I personally use the HyperSpin lists at the moment, since finding matching artwork is a huge pain for the alternatives.

Used2BeRX

@phulshof Here's the count of the systems currently on my RetroPie. This does not include any arcade games or CD systems (segaCD, TurboCD, PSX).

NES = 1,756
SNES = 1, 144
Game Boy = 668
Game Boy Advance = 1,118
Game Boy Color = 596
Sega Genesis = 959
Sega 32x = 39
Sega Master System = 319
Sega Game Gear = 308
SG-1000 = 66
Atari 2600 = 715
Atari 5200/800 = 263
Atari 7800 = 86
Atari LYNX = 86
TurboGrafx 16 = 220
Neo Geo Pocket/Color = 49

There are a few other systems like Wonderswan and Supervision and Virtual Boy I haven't bothered setting up yet.

Every one of these systems is completely clone free. Priority of games was US, then EU, then Japan (other). Any Japanese games that are included that were not translations are games that were in english or had very little text and are completely playable.

There are a TON of Translations, Hacks, Prototypes, Unlicensed and other games outside of the official releases. Every game has boxarts. For games that did not have boxart, our team had a talented graphic artist that made system specific "generic boxart" that looked just like an official box for the system. All games that have a title screen have a title shot (almost 100% with all of the newer systems had one... not so much with old systems like Atari 2600). Every game has an action shot. There are a lot of carts/CDs/Disks, but that's hit or miss. Every boxart also has a PNG image "3D Boxart". They look nice, but they're not "real". Somebody took all of our Boxarts and tilted them at an angle on a template that makes them look like a DVD case.

Also with our system comes GameFAQs ripped from the site and renamed exactly as the roms (and all of the files in these zips were renamed by hand so they weren't 250 characters long and you can actually see what they are). Not usable currently in RetroPie, but works awesome on our setup when you want to pull up a gamefaq in the middle of a game.

Tons of manuals as well as rips of VGMaps, all matched to the rom names.

(I'm not sure if I can post any gamefaqs or vgmaps rips here. That is probably against forum rules. Can't make any use of them currently with RetroPie anyhow).

If we can get this program to the point where it can make a perfect gamelist.xml file with our file structure without having to do any manual edits after it's done, then any new games, improved videos, artwork, manuals could be put in place of the old stuff and a new gamelist.xml could be created and FTPd to the Pi and it would reflect any changes. At least, that's what I hope to accomplish here.

phulshof

Well, since I'm relatively new here I'm not sure what is or is not allowed on these forums, but feel free to drop me an email so I can get you in touch with some other people working on similar projects. It would be nice to combine all this work, and provide this material as downloads to people, and I can certainly be of assistance there. :) I can also handle conversion to other front-ends that do support some of these missing features, and help build a cross-front-end set of material this way (with matching XML and/or dat files).

Used2BeRX

@phulshof I'm interested. I don't want to give out my email here in the threads though, so how would we best get in touch?

phulshof

@Used2BeRX

My email is in my profile. 😀

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu, can you help us to______________?"

I don't know the answer to that. I don't know what UDb23 is asking you to do either.

OK. Let's try to summarize @Used2BeRX needs, check who can best support him to do what and if what we can do is worth the effort vs using or integrating with other existing metadata.

About the needs (@Used2BeRX please confirm):
A) convert the txt files to XML with specific parsing and specific items/tags generation in the xml (A1- one to one conversion txt to xml A2-generate global gamelist.xml based on all files in a folder)

I wrote a windows program (MetaX) to do this; currently finalizing it to include all tags and fix minor conversion issues.

B) As @Used2BeRX wrote

When the program reads "1ST ULTIMATE SUPER MARIO BROS.TXT", I'd like to see it be able to recognize what sub folder the rom "1ST ULTIMATE SUPER MARIO BROS.ZIP" is in and add that path to the <path></path> tag, instead of just putting "./" before the file name.

For this example, using the table I had posted above, this line should read as:
<path>/home/pi/RetroPie/roms/nes/(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP</path>
(I think ./(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP would be the same thing???)
Right now this is how it's reading:
<path>./Frank's First Ultimate Super Mario Bros..zip</path>

This is something I think @meleu can do with a script, that would modify the single xml files or even the global gamelist.xml (both generated by MetaX).

C) An additional nice idea would be to generate system specific gamelist.xml based on A) and B) but including only the ROMs that are actually used/installed on the Pi (avoiding to load a huge xml with tons of info related to roms that are not installed; and probably slowing down ES). @Meleu: can this be done ?

meleu

@UDb23 I think I got it. Will try to sketch something this evening.

Used2BeRX

@UDb23 @meleu Yes UDb23. That seems to be exactly what I was looking for.

Assuming the script works that will locate the rom (with an exact match for the name), I was also hoping that we could take that a step further so that instead of just creating static Media locations as it does now, it would actually find the media with the exact match for the file name, and it would A) only add media tags for existing media and B) put the exact extension of the file match.

Right now, all of the media I have will be exactly as we discussed for those static directories MetaX creates for each entry, but I'm hoping if we get this set up right that anybody at any time could upgrade media and not be concerned with what the extension is as long as the file name is an exact match. This would be important, for instance, when the videos get HD upgrades. Our old system was only SD and the videos currently are in WMV8 format. I have some people right now who claim they are beginning to create MP4 versions of much higher quality, and I was hoping this could recognize that change and create the gamelist.xml accordingly.

Also, I want to create synopsis entries for every one of my folders that point to a single picture, or possibly even a video that would be picked up by MetaX and added to the gamelist.xml (I posted that Atari 7800 example a day or two ago). But we can discuss that part later, and what I would need to do to prepare those.

Used2BeRX

@phulshof I know it shows up when you look at your profile, but emails are blocked to other users. I don't remember where I read that, but I'm not seeing yours.

phulshof

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@phulshof I know it shows up when you look at your profile, but emails are blocked to other users. I don't remember where I read that, but I'm not seeing yours.

No worries; I can always use a bit more spam. 😀