Making premade gamelist.xml with xtra media collections

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

No path, but the other problem I noticed is that it's taking the altered game name instead of using the rom file name. In this example that means the file name is incorrect and the Case is wrong because the case of the TXT file and ZIP file and all Xtras will be all Caps "XXXXXX.XXX"

I modified the program: now the exact TXT filename (with .zip extension) will be used for the <path> tag.
So if TXT filename corresponds to ROM filename (case included) the xml will inlcude the correct name for the rom file.

Actual path were the ROM resides will then be found by @meleu 's new script, that will also automagically modify the path tag in the .xml accordingly.

UDb23

@Used2BeRX New version (06) uploaded.

No more errors (as far as I could check) with txt files import.
Error was due to max number of lines limit I used (was not expecting that long descriptions). Now basically unlimited.
ROM name exactly as txt filename (as mentioned previously)
"convert all" checkbox allows to convert all txt files in folder (tested on 7800 TXTs, converted 90 files without any issue).

Next to be implemented:

implement all items/tags you requested (not yet finalized)
custom media path (can be set by user)
create global gamelist.xml (converting all txt in folder)
fix append issue (currently appends to existing xml with same name, if file already exists)

Used2BeRX

@UDb23 It works without crashing now. I did the Atari 7800 (90) for myself and I also did the NES (1,756 files!). I'm pretty sure that with a group that large not spitting out any errors that we're good on that now.

Two issues I noticed so far. (I'll study the output files and see if I find any more).

Zip file romname is showing the proper case now, but the extension is a lowercase ".zip". Can your program recognize that and case the extension accordingly? For my purposes, every file involved now will be all uppercase to avoid any confusion, so forcing it is also acceptable, but that will limit the use of the program for others not using my collection in the future.
All other media is still showing the game title instead of the romname, so the lines are wrong and the case is incorrect. Everything in the <artwork-type><movies><manual><vgmap><marquee>, etc. should be the exact same as the file name in the <path> tag (the extension will be different though of course).

Actual path were the ROM resides will then be found by @meleu 's new script, that will also automagically modify the path tag in the .xml accordingly.

I'd like to test this out. Was this implemented in this change, or you're going to put it in a future update? If it's in there now, how do I have to have the file structures set up since there is never a question about where the roms are found?

Future ideas:

If this script works, perhaps we can do even more upgrades to the program that would be similar to this one. For instance, a person might have artwork that is mixed between .JPG and .PNG. Using a similar script the exact file names could be put on each line of the tags (and also you could have the ability to completely skip tags if you didn't have the particular artwork).

I may or may not have somebody upgrading a lot of the videos we have. The old videos were .WMV and the new videos would be MP4. That would be awesome to have the program determine what extension to put in there.

UDb23

Glad to hear it works and was able to manage that huge number of files :-)

1.Can your program recognize that and case the extension accordingly?

Should be able to detect actual case of extension of original file and apply to .zip accordingly.

2 All other media is still showing the game title .....should be the exact same as the file name in the <path> tag (the extension

I've overseen this; sure I can do. Very easy fix

Actual path were the ROM resides...

Current idea is that my program creates the xml files (eventually with some user customizable path for roms and media; to be set by user).

@meleu very kindly agreed to write a new specific script that would "scrape" (check) where actual roms are and automatically modify the required xml files. I'll wait for his feedback and proposal on this, including where xml file should be available to be accessed thru the web from his script (e.g. github). This would avoid user the need to download ALL xml files locally just to use/install only the ones he has roms for.

UDb23

btw I have complete metadata for VIc20 roms I got thru a combination of an own written scraper (mobygames based) and other sources found on the web. Also have most cover art. Will try to find a way to generate the related xml for Retropie like I did for your txt files.
Likely it will be an additional convert command/option of MetaX.

Used2BeRX

@UDb23 Sounds good man.

I know you got a lot on your plate with this now and I don't want to bog you down with requests on it, but I was thinking about how to handle the sub-folders. Currently, I had written entries for the folders on the Atari gamelists that have information about the console itself as well as pointers to a .PNG file that shows a nice pic of the system.

If you look at the NES sub-folders, there are a TON of different ones. I'd like to make some cool artwork to display for each of these folders as well as a short description for what's inside each of them for the end user.

If I were to make TXT files for every sub-folder for the program to use, how would you suggest I name these, and how would you suggest I put the data in for your program to scrape? There wouldn't be much to add on these as far as media. Currently I'm only putting one picture from a folder called /Artwork/Console/.

I was thinking that for naming them I could start every folder entry with "FOLDER xxxxxxxxx.TXT"?

No need to give an answer on that right away. Id' like to see what you think about it.

I'm hoping that eventually we can get the entire process automated so that only existing media is added to the XML files and if any media is upgraded/added that you could simply re-run the program and get a completely working gamelist.xml file that contains the upgrades without having to worry about re-testing that everything still works and/or going through and making a lot of changes by hand.

UDb23

@Used2BeRX Just a quick answer from my side as @meleu is the master in this area.
I read a post somewhere that folder images can be used in ES but can't remember how this is done. To provide necessary information about suggested folders you may create a info_folder_xxx.txt file or even just a single folderInfo.txt file that includes all necessary folder info.
Please check how we managed this for overlays here.; it may give you some idea. on how to structure it.

Used2BeRX

@UDb23 I already know how to make the folder images to show up. I was asking how to make the "FOLDERxxxxxx.TXT" files work in your program. I haven't made any txt files to run through your program, but here's an example of what a working <folder></folder> entry looks like in my gamelist.xmls for the Atari systems:

<folder>
		<path>./(1) Licensed</path>
		<image>/home/pi/RetroPie/Media/A7800/Artwork/Console/7800.png</image>
		<name>(1) Licensed Atari 7800 ProSystem Games</name>
		<releasedate>1982</releasedate>
		<developer>Atari Inc.</developer>
		<publisher>Ateri Inc.</publisher>
		<desc>LICENSED ATARI 7800 PROSYSTEM GAMES
==========================================
The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986. It is almost fully backward-compatible with the Atari 2600, the first console to have backward compatibility without the use of additional modules. It was considered affordable at a price of US$140.

The 7800 has significantly improved graphics hardware over the 2600, but uses the same audio chip. It also shipped with a different model of joystick than the 2600-standard CX40.

The 1986 launch is sometimes referred to as a "re-release" or "relaunch" because the Atari 7800 had originally been announced on May 21, 1984, to replace Atari Inc.'s Atari 5200, but a general release was shelved due to the sale of the company. A few units were released to test markets in June 1984 though.
               </desc>
</folder>

My question to you was how would you like these individual sub-folder synopsis.txt files created?

Used2BeRX

@UDb23 It took a couple of days work, but I now have 100% confirmed filename/case matches for every rom, synopsis.txt and low def Pi-Zero artwork. Everything for the Pi-Zero is now completely prepared for testing when the program is ready.

To Do List:
100% match for the highest quality artwork I have for use with a greater Pi.
100% match for videos, manual scans, vgmap zips, gamefaq zips for use with a greater Pi. *

(* No current use for manuals, vgmaps or gamefaqs, but hopefully somebody will find a way to use them in EmulationStation like we can use them on the other platform)

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

here's an example of what a working <folder></folder> entry looks like

nice, thanks.

how would you like these individual sub-folder synopsis.txt files

Let's wait for @meleu 's considerations on this; meanwhile I finalize the program.

Used2BeRX

@UDb23 I got pretty much everything re-cased and compared now for both the PiZero and Pi3 setups. I'm still waiting on changing the highest quality artwork to JPG to see if you can make the program recognize what type of file is in the media folders since the original PNG files would look better.

I'm ready to start making those synopsis.txt files for all of the folders whenever you guys have directions for me. Turns out there is going to be quite a bit of rain the next 3-4 days so the gig I had is going to be put off for a few more days at least.

I also had a very long list I put together of about 500-600 romlist names that I wanted to change in those synopsis files a few months back that I finally got around to doing today. I'm not going to say that the names are perfect across nearly 10,000 games, but they're pretty damn close. I tried to make them exactly how they are displayed on the boxart for the games. I've been through that list about 5 or 6 times over the years and I'm really happy with how it looks now.

Looking forward to being able to test more out and see if having all this artwork is going to break my Pi Zero or not. :)

meleu

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).
the art files.
where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).
please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

mediamogul

@meleu

I'm ready to start.
Here's what I need...

alt text

phulshof

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

lilbud

@mediamogul I love GOTG

meleu

@phulshof said in Making premade gamelist.xml with xtra media collections:

What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?

I kinda have the same curiosity...

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).

the art files.

where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).

please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).

The gamelists I made before by hand won't work with the artwork now because I made everything upper case. For test purposes you could download one of the 7800 packs I put up, but be aware that everything that will be happening in the future is designed with all uppercase file names and extensions in mind. I could upload a quick pack with the uppercase, but it would not work with the gamelist.xml.

The art files would be in ~/RetroPie/Media/{SYSTEM_NAME}/Artwork/
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/3D Boxart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Action
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Box Front
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Cart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Console
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Titles

(/Artwork/Console/ is a folder that will have the artwork meant to be displayed when resting on a folder or sub-folder. I still need to know how I should make the synopsis.txt files for folders)

You won't need me to rename the systems for a while, since you're just looking at the Atari 7800 pack right now. I thought I had everything prepared for this until I read that. That's going to be a big ass pain for me to do since they were all set up that way for the other system. Quite a few "systems" we had are mixed with other systems because the emulators played several different systems. For example Master System/Game Gear/SG-1000 are all grouped together, Genesis/32x/Sega CD are all grouped together, etc. And I'll get to do it twice because there is a low def set and a high def set. Fun times.

Anyway. Here's the link: https://mega.nz/#!QKxmlCYL!z5ck_O7L7pNzL19Z4HdMZ64FikmYGC7J1mpfXOGvEsk

EDIT:

Renaming the folders themselves to make them match the RetroPie wont' be a big deal. That being said, I thought about it and I will not be separating any of the media. That introduces far too many possibilities for there being problems in a setup that I know is 100% correct at the moment.

So all media for
Gameboy, Gameboy Advance and Gameboy Color will be in /Media/gb/
Genesis, 32x, Sega CD will be in /Media/megadrive/
Master System, Game Gear, SG-1000 will be in /Media/mastersystem/
Neo Geo Pocket, Neo Geo Pocket color will be in /Media/ngp/
Turbografx, TurboCD will be in /Media/pcengine/

If I absolutely need to make these name changes for the media folders, that's the best I can do.

Used2BeRX

@phulshof said in Making premade gamelist.xml with xtra media collections:

Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?

It will be downloadable somewhere. I can't give the link out here though.
None of this will have anything to do with MAME. I am unaware what a MAME software list is.
I dunno. I like it.

meleu

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.
Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".
Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).
Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

Used2BeRX

@meleu said in Making premade gamelist.xml with xtra media collections:

@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?

Here is what I understood so far:

You have a huge set of very carefully collected game's metadata.

Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".

Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).

Our fellow @UDb23 is helping you with this conversion.

Could you confirm if all the sentences above are true?

Now, could you fill the gap in the sentence below?:

"meleu, can you help us to ____________?"

It's data. We never used xml though. Our system also relied on an individual games crc values and the file name for all extras and data so we didn't have any reason to load every single piece of information for every single game when the system booted. The pi seems rather inefficient that way, but I'm sure since it's still getting upgrades that eventually that won't be a problem.
Yes, although a factor of 10 may have slightly overstated it. I was not happy with the in-menu scraper or the other not so intuitive method compared to what we have. As stated above I'm not a fan of the XML necessity at all, but I'm trying my best here to integrate the work we've already done. It's like having a bunch of kids and moving to a new country and making them all learn to speak the language.
Yes. UDb23 already made that happen on a 1-to-1 basis, as well as adding in pointers to media (In the folders I already have, not the ones you're requesting I switch everything to).
Yes.
"meleu, can you help us to______________?"

I don't know the answer to that. I don't know what UDb23 is asking you to do either.