Making premade gamelist.xml with xtra media collections

Used2BeRX

@UDb23 It works without crashing now. I did the Atari 7800 (90) for myself and I also did the NES (1,756 files!). I'm pretty sure that with a group that large not spitting out any errors that we're good on that now.

Two issues I noticed so far. (I'll study the output files and see if I find any more).

Zip file romname is showing the proper case now, but the extension is a lowercase ".zip". Can your program recognize that and case the extension accordingly? For my purposes, every file involved now will be all uppercase to avoid any confusion, so forcing it is also acceptable, but that will limit the use of the program for others not using my collection in the future.
All other media is still showing the game title instead of the romname, so the lines are wrong and the case is incorrect. Everything in the <artwork-type><movies><manual><vgmap><marquee>, etc. should be the exact same as the file name in the <path> tag (the extension will be different though of course).

Actual path were the ROM resides will then be found by @meleu 's new script, that will also automagically modify the path tag in the .xml accordingly.

I'd like to test this out. Was this implemented in this change, or you're going to put it in a future update? If it's in there now, how do I have to have the file structures set up since there is never a question about where the roms are found?

Future ideas:

If this script works, perhaps we can do even more upgrades to the program that would be similar to this one. For instance, a person might have artwork that is mixed between .JPG and .PNG. Using a similar script the exact file names could be put on each line of the tags (and also you could have the ability to completely skip tags if you didn't have the particular artwork).

I may or may not have somebody upgrading a lot of the videos we have. The old videos were .WMV and the new videos would be MP4. That would be awesome to have the program determine what extension to put in there.

UDb23

Glad to hear it works and was able to manage that huge number of files :-)

1.Can your program recognize that and case the extension accordingly?

Should be able to detect actual case of extension of original file and apply to .zip accordingly.

2 All other media is still showing the game title .....should be the exact same as the file name in the <path> tag (the extension

I've overseen this; sure I can do. Very easy fix

Actual path were the ROM resides...

Current idea is that my program creates the xml files (eventually with some user customizable path for roms and media; to be set by user).

@meleu very kindly agreed to write a new specific script that would "scrape" (check) where actual roms are and automatically modify the required xml files. I'll wait for his feedback and proposal on this, including where xml file should be available to be accessed thru the web from his script (e.g. github). This would avoid user the need to download ALL xml files locally just to use/install only the ones he has roms for.

UDb23

btw I have complete metadata for VIc20 roms I got thru a combination of an own written scraper (mobygames based) and other sources found on the web. Also have most cover art. Will try to find a way to generate the related xml for Retropie like I did for your txt files.
Likely it will be an additional convert command/option of MetaX.

Used2BeRX

@UDb23 Sounds good man.

I know you got a lot on your plate with this now and I don't want to bog you down with requests on it, but I was thinking about how to handle the sub-folders. Currently, I had written entries for the folders on the Atari gamelists that have information about the console itself as well as pointers to a .PNG file that shows a nice pic of the system.

If you look at the NES sub-folders, there are a TON of different ones. I'd like to make some cool artwork to display for each of these folders as well as a short description for what's inside each of them for the end user.

If I were to make TXT files for every sub-folder for the program to use, how would you suggest I name these, and how would you suggest I put the data in for your program to scrape? There wouldn't be much to add on these as far as media. Currently I'm only putting one picture from a folder called /Artwork/Console/.

I was thinking that for naming them I could start every folder entry with "FOLDER xxxxxxxxx.TXT"?

No need to give an answer on that right away. Id' like to see what you think about it.

I'm hoping that eventually we can get the entire process automated so that only existing media is added to the XML files and if any media is upgraded/added that you could simply re-run the program and get a completely working gamelist.xml file that contains the upgrades without having to worry about re-testing that everything still works and/or going through and making a lot of changes by hand.

UDb23

@Used2BeRX Just a quick answer from my side as @meleu is the master in this area.
I read a post somewhere that folder images can be used in ES but can't remember how this is done. To provide necessary information about suggested folders you may create a info_folder_xxx.txt file or even just a single folderInfo.txt file that includes all necessary folder info.
Please check how we managed this for overlays here.; it may give you some idea. on how to structure it.

Used2BeRX

@UDb23 I already know how to make the folder images to show up. I was asking how to make the "FOLDERxxxxxx.TXT" files work in your program. I haven't made any txt files to run through your program, but here's an example of what a working <folder></folder> entry looks like in my gamelist.xmls for the Atari systems:

<folder>
		<path>./(1) Licensed</path>
		<image>/home/pi/RetroPie/Media/A7800/Artwork/Console/7800.png</image>
		<name>(1) Licensed Atari 7800 ProSystem Games</name>
		<releasedate>1982</releasedate>
		<developer>Atari Inc.</developer>
		<publisher>Ateri Inc.</publisher>
		<desc>LICENSED ATARI 7800 PROSYSTEM GAMES
==========================================
The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986. It is almost fully backward-compatible with the Atari 2600, the first console to have backward compatibility without the use of additional modules. It was considered affordable at a price of US$140.

The 7800 has significantly improved graphics hardware over the 2600, but uses the same audio chip. It also shipped with a different model of joystick than the 2600-standard CX40.

The 1986 launch is sometimes referred to as a "re-release" or "relaunch" because the Atari 7800 had originally been announced on May 21, 1984, to replace Atari Inc.'s Atari 5200, but a general release was shelved due to the sale of the company. A few units were released to test markets in June 1984 though.
               </desc>
</folder>

My question to you was how would you like these individual sub-folder synopsis.txt files created?

Used2BeRX

@UDb23 It took a couple of days work, but I now have 100% confirmed filename/case matches for every rom, synopsis.txt and low def Pi-Zero artwork. Everything for the Pi-Zero is now completely prepared for testing when the program is ready.

To Do List:
100% match for the highest quality artwork I have for use with a greater Pi.
100% match for videos, manual scans, vgmap zips, gamefaq zips for use with a greater Pi. *

(* No current use for manuals, vgmaps or gamefaqs, but hopefully somebody will find a way to use them in EmulationStation like we can use them on the other platform)

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

here's an example of what a working <folder></folder> entry looks like

nice, thanks.

how would you like these individual sub-folder synopsis.txt files

Let's wait for @meleu 's considerations on this; meanwhile I finalize the program.

Used2BeRX

@UDb23 I got pretty much everything re-cased and compared now for both the PiZero and Pi3 setups. I'm still waiting on changing the highest quality artwork to JPG to see if you can make the program recognize what type of file is in the media folders since the original PNG files would look better.

I'm ready to start making those synopsis.txt files for all of the folders whenever you guys have directions for me. Turns out there is going to be quite a bit of rain the next 3-4 days so the gig I had is going to be put off for a few more days at least.

I also had a very long list I put together of about 500-600 romlist names that I wanted to change in those synopsis files a few months back that I finally got around to doing today. I'm not going to say that the names are perfect across nearly 10,000 games, but they're pretty damn close. I tried to make them exactly how they are displayed on the boxart for the games. I've been through that list about 5 or 6 times over the years and I'm really happy with how it looks now.

Looking forward to being able to test more out and see if having all this artwork is going to break my Pi Zero or not. :)

meleu

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).
the art files.
where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).
please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

mediamogul

@meleu

I'm ready to start.
Here's what I need...

alt text

phulshof

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

lilbud

@mediamogul I love GOTG

meleu

@phulshof said in Making premade gamelist.xml with xtra media collections:

What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?

I kinda have the same curiosity...

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).

the art files.

where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).

please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

The gamelists I made before by hand won't work with the artwork now because I made everything upper case. For test purposes you could download one of the 7800 packs I put up, but be aware that everything that will be happening in the future is designed with all uppercase file names and extensions in mind. I could upload a quick pack with the uppercase, but it would not work with the gamelist.xml.

The art files would be in ~/RetroPie/Media/{SYSTEM_NAME}/Artwork/
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/3D Boxart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Action
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Box Front
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Cart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Console
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Titles

(/Artwork/Console/ is a folder that will have the artwork meant to be displayed when resting on a folder or sub-folder. I still need to know how I should make the synopsis.txt files for folders)

You won't need me to rename the systems for a while, since you're just looking at the Atari 7800 pack right now. I thought I had everything prepared for this until I read that. That's going to be a big ass pain for me to do since they were all set up that way for the other system. Quite a few "systems" we had are mixed with other systems because the emulators played several different systems. For example Master System/Game Gear/SG-1000 are all grouped together, Genesis/32x/Sega CD are all grouped together, etc. And I'll get to do it twice because there is a low def set and a high def set. Fun times.

Anyway. Here's the link: https://mega.nz/#!QKxmlCYL!z5ck_O7L7pNzL19Z4HdMZ64FikmYGC7J1mpfXOGvEsk

EDIT:

Renaming the folders themselves to make them match the RetroPie wont' be a big deal. That being said, I thought about it and I will not be separating any of the media. That introduces far too many possibilities for there being problems in a setup that I know is 100% correct at the moment.

So all media for
Gameboy, Gameboy Advance and Gameboy Color will be in /Media/gb/
Genesis, 32x, Sega CD will be in /Media/megadrive/
Master System, Game Gear, SG-1000 will be in /Media/mastersystem/
Neo Geo Pocket, Neo Geo Pocket color will be in /Media/ngp/
Turbografx, TurboCD will be in /Media/pcengine/

If I absolutely need to make these name changes for the media folders, that's the best I can do.

Used2BeRX

@phulshof said in Making premade gamelist.xml with xtra media collections:

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

It will be downloadable somewhere. I can't give the link out here though.
None of this will have anything to do with MAME. I am unaware what a MAME software list is.
I dunno. I like it.

meleu

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.
Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".
Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).
Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.

Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".

Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).

Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

It's data. We never used xml though. Our system also relied on an individual games crc values and the file name for all extras and data so we didn't have any reason to load every single piece of information for every single game when the system booted. The pi seems rather inefficient that way, but I'm sure since it's still getting upgrades that eventually that won't be a problem.
Yes, although a factor of 10 may have slightly overstated it. I was not happy with the in-menu scraper or the other not so intuitive method compared to what we have. As stated above I'm not a fan of the XML necessity at all, but I'm trying my best here to integrate the work we've already done. It's like having a bunch of kids and moving to a new country and making them all learn to speak the language.
Yes. UDb23 already made that happen on a 1-to-1 basis, as well as adding in pointers to media (In the folders I already have, not the ones you're requesting I switch everything to).
Yes.
"meleu, can you help us to______________?"

I don't know the answer to that. I don't know what UDb23 is asking you to do either.

phulshof

@Used2BeRX

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@phulshof said in Making premade gamelist.xml with xtra media collections:

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

It will be downloadable somewhere. I can't give the link out here though.

None of this will have anything to do with MAME. I am unaware what a MAME software list is.

I dunno. I like it.

I guess my reason for asking is that the general problem people encounter is to find the games with matching database information and matching art work. Apparently you have a set, which, if made available, people would have to match with their games.

As an alternative, HyperSpin has created XML files with metadata for many systems, with corresponding artwork from their own site and EmuMovies, where matching games can easily be found. These XML files have been filtered for duplicate games and non-playable games (to us non-Japanese speaking gamers).

MAME also supports consoles/home computers/handhelds via their software lists, which contains meta information, and for which matching games can easily be found. Artwork is handled via MAME EXTRAs, though is severely lacking (I spent a lot of time creating thousands of screenshots/titleshots for MSX emulation for instance).

Can you give an indication of the number of games per system you have meta information on? It would give an indication of how it compares to the HyperSpin and MAME alternatives. I personally use the HyperSpin lists at the moment, since finding matching artwork is a huge pain for the alternatives.

Used2BeRX

@phulshof Here's the count of the systems currently on my RetroPie. This does not include any arcade games or CD systems (segaCD, TurboCD, PSX).

NES = 1,756
SNES = 1, 144
Game Boy = 668
Game Boy Advance = 1,118
Game Boy Color = 596
Sega Genesis = 959
Sega 32x = 39
Sega Master System = 319
Sega Game Gear = 308
SG-1000 = 66
Atari 2600 = 715
Atari 5200/800 = 263
Atari 7800 = 86
Atari LYNX = 86
TurboGrafx 16 = 220
Neo Geo Pocket/Color = 49

There are a few other systems like Wonderswan and Supervision and Virtual Boy I haven't bothered setting up yet.

Every one of these systems is completely clone free. Priority of games was US, then EU, then Japan (other). Any Japanese games that are included that were not translations are games that were in english or had very little text and are completely playable.

There are a TON of Translations, Hacks, Prototypes, Unlicensed and other games outside of the official releases. Every game has boxarts. For games that did not have boxart, our team had a talented graphic artist that made system specific "generic boxart" that looked just like an official box for the system. All games that have a title screen have a title shot (almost 100% with all of the newer systems had one... not so much with old systems like Atari 2600). Every game has an action shot. There are a lot of carts/CDs/Disks, but that's hit or miss. Every boxart also has a PNG image "3D Boxart". They look nice, but they're not "real". Somebody took all of our Boxarts and tilted them at an angle on a template that makes them look like a DVD case.

Also with our system comes GameFAQs ripped from the site and renamed exactly as the roms (and all of the files in these zips were renamed by hand so they weren't 250 characters long and you can actually see what they are). Not usable currently in RetroPie, but works awesome on our setup when you want to pull up a gamefaq in the middle of a game.

Tons of manuals as well as rips of VGMaps, all matched to the rom names.

(I'm not sure if I can post any gamefaqs or vgmaps rips here. That is probably against forum rules. Can't make any use of them currently with RetroPie anyhow).

If we can get this program to the point where it can make a perfect gamelist.xml file with our file structure without having to do any manual edits after it's done, then any new games, improved videos, artwork, manuals could be put in place of the old stuff and a new gamelist.xml could be created and FTPd to the Pi and it would reflect any changes. At least, that's what I hope to accomplish here.