Making premade gamelist.xml with xtra media collections
-
@phulshof said in Making premade gamelist.xml with xtra media collections:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
I kinda have the same curiosity...
-
@meleu said in Making premade gamelist.xml with xtra media collections:
I'm ready to start.
Here's what I need to actually start coding:
- a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).
- the art files.
- where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at
~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys). - please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).
The gamelists I made before by hand won't work with the artwork now because I made everything upper case. For test purposes you could download one of the 7800 packs I put up, but be aware that everything that will be happening in the future is designed with all uppercase file names and extensions in mind. I could upload a quick pack with the uppercase, but it would not work with the gamelist.xml.
The art files would be in ~/RetroPie/Media/{SYSTEM_NAME}/Artwork/
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/3D Boxart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Action
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Box Front
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Cart
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Console
~/RetroPie/Media/{SYSTEM_NAME}/Artwork/Titles(/Artwork/Console/ is a folder that will have the artwork meant to be displayed when resting on a folder or sub-folder. I still need to know how I should make the synopsis.txt files for folders)
You won't need me to rename the systems for a while, since you're just looking at the Atari 7800 pack right now. I thought I had everything prepared for this until I read that. That's going to be a big ass pain for me to do since they were all set up that way for the other system. Quite a few "systems" we had are mixed with other systems because the emulators played several different systems. For example Master System/Game Gear/SG-1000 are all grouped together, Genesis/32x/Sega CD are all grouped together, etc. And I'll get to do it twice because there is a low def set and a high def set. Fun times.
Anyway. Here's the link: https://mega.nz/#!QKxmlCYL!z5ck_O7L7pNzL19Z4HdMZ64FikmYGC7J1mpfXOGvEsk
EDIT:
Renaming the folders themselves to make them match the RetroPie wont' be a big deal. That being said, I thought about it and I will not be separating any of the media. That introduces far too many possibilities for there being problems in a setup that I know is 100% correct at the moment.
So all media for
Gameboy, Gameboy Advance and Gameboy Color will be in /Media/gb/
Genesis, 32x, Sega CD will be in /Media/megadrive/
Master System, Game Gear, SG-1000 will be in /Media/mastersystem/
Neo Geo Pocket, Neo Geo Pocket color will be in /Media/ngp/
Turbografx, TurboCD will be in /Media/pcengine/If I absolutely need to make these name changes for the media folders, that's the best I can do.
-
@phulshof said in Making premade gamelist.xml with xtra media collections:
Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?-
It will be downloadable somewhere. I can't give the link out here though.
-
None of this will have anything to do with MAME. I am unaware what a MAME software list is.
-
I dunno. I like it.
-
-
@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?
Here is what I understood so far:
- You have a huge set of very carefully collected game's metadata.
- Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".
- Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).
- Our fellow @UDb23 is helping you with this conversion.
Could you confirm if all the sentences above are true?
Now, could you fill the gap in the sentence below?:
"meleu, can you help us to ____________?"
-
@meleu said in Making premade gamelist.xml with xtra media collections:
@Used2BeRX Hey man, really sorry, I'm a little lost. Can we start from the beginning?
Here is what I understood so far:
- You have a huge set of very carefully collected game's metadata.
- Your metadata seems to be very very superior than those we can usually get via traditional scraping methods "by a factor of 10".
- Your metadata files aren't in the ES-friendly gamelist.xml format and you want to automate this conversion (from txt to gamelist.xml).
- Our fellow @UDb23 is helping you with this conversion.
Could you confirm if all the sentences above are true?
Now, could you fill the gap in the sentence below?:
"meleu, can you help us to ____________?"
-
It's data. We never used xml though. Our system also relied on an individual games crc values and the file name for all extras and data so we didn't have any reason to load every single piece of information for every single game when the system booted. The pi seems rather inefficient that way, but I'm sure since it's still getting upgrades that eventually that won't be a problem.
-
Yes, although a factor of 10 may have slightly overstated it. I was not happy with the in-menu scraper or the other not so intuitive method compared to what we have. As stated above I'm not a fan of the XML necessity at all, but I'm trying my best here to integrate the work we've already done. It's like having a bunch of kids and moving to a new country and making them all learn to speak the language.
-
Yes. UDb23 already made that happen on a 1-to-1 basis, as well as adding in pointers to media (In the folders I already have, not the ones you're requesting I switch everything to).
-
Yes.
-
"meleu, can you help us to______________?"
I don't know the answer to that. I don't know what UDb23 is asking you to do either.
-
@Used2BeRX said in Making premade gamelist.xml with xtra media collections:
@phulshof said in Making premade gamelist.xml with xtra media collections:
Out of curiosity:
What's the eventual plan with this code/xml/art? Will it be downloadable from somewhere? Integrated into GamesDB? ....?
How do these lists compare to the MAME software lists?
How do they compare to the HyperSpin XML files (with corresponding art sets)?-
It will be downloadable somewhere. I can't give the link out here though.
-
None of this will have anything to do with MAME. I am unaware what a MAME software list is.
-
I dunno. I like it.
I guess my reason for asking is that the general problem people encounter is to find the games with matching database information and matching art work. Apparently you have a set, which, if made available, people would have to match with their games.
As an alternative, HyperSpin has created XML files with metadata for many systems, with corresponding artwork from their own site and EmuMovies, where matching games can easily be found. These XML files have been filtered for duplicate games and non-playable games (to us non-Japanese speaking gamers).
MAME also supports consoles/home computers/handhelds via their software lists, which contains meta information, and for which matching games can easily be found. Artwork is handled via MAME EXTRAs, though is severely lacking (I spent a lot of time creating thousands of screenshots/titleshots for MSX emulation for instance).
Can you give an indication of the number of games per system you have meta information on? It would give an indication of how it compares to the HyperSpin and MAME alternatives. I personally use the HyperSpin lists at the moment, since finding matching artwork is a huge pain for the alternatives.
-
-
@phulshof Here's the count of the systems currently on my RetroPie. This does not include any arcade games or CD systems (segaCD, TurboCD, PSX).
NES = 1,756
SNES = 1, 144
Game Boy = 668
Game Boy Advance = 1,118
Game Boy Color = 596
Sega Genesis = 959
Sega 32x = 39
Sega Master System = 319
Sega Game Gear = 308
SG-1000 = 66
Atari 2600 = 715
Atari 5200/800 = 263
Atari 7800 = 86
Atari LYNX = 86
TurboGrafx 16 = 220
Neo Geo Pocket/Color = 49There are a few other systems like Wonderswan and Supervision and Virtual Boy I haven't bothered setting up yet.
Every one of these systems is completely clone free. Priority of games was US, then EU, then Japan (other). Any Japanese games that are included that were not translations are games that were in english or had very little text and are completely playable.
There are a TON of Translations, Hacks, Prototypes, Unlicensed and other games outside of the official releases. Every game has boxarts. For games that did not have boxart, our team had a talented graphic artist that made system specific "generic boxart" that looked just like an official box for the system. All games that have a title screen have a title shot (almost 100% with all of the newer systems had one... not so much with old systems like Atari 2600). Every game has an action shot. There are a lot of carts/CDs/Disks, but that's hit or miss. Every boxart also has a PNG image "3D Boxart". They look nice, but they're not "real". Somebody took all of our Boxarts and tilted them at an angle on a template that makes them look like a DVD case.
Also with our system comes GameFAQs ripped from the site and renamed exactly as the roms (and all of the files in these zips were renamed by hand so they weren't 250 characters long and you can actually see what they are). Not usable currently in RetroPie, but works awesome on our setup when you want to pull up a gamefaq in the middle of a game.
Tons of manuals as well as rips of VGMaps, all matched to the rom names.
(I'm not sure if I can post any gamefaqs or vgmaps rips here. That is probably against forum rules. Can't make any use of them currently with RetroPie anyhow).
If we can get this program to the point where it can make a perfect gamelist.xml file with our file structure without having to do any manual edits after it's done, then any new games, improved videos, artwork, manuals could be put in place of the old stuff and a new gamelist.xml could be created and FTPd to the Pi and it would reflect any changes. At least, that's what I hope to accomplish here.
-
Well, since I'm relatively new here I'm not sure what is or is not allowed on these forums, but feel free to drop me an email so I can get you in touch with some other people working on similar projects. It would be nice to combine all this work, and provide this material as downloads to people, and I can certainly be of assistance there. :) I can also handle conversion to other front-ends that do support some of these missing features, and help build a cross-front-end set of material this way (with matching XML and/or dat files).
-
@phulshof I'm interested. I don't want to give out my email here in the threads though, so how would we best get in touch?
-
My email is in my profile. 😀
-
@Used2BeRX said in Making premade gamelist.xml with xtra media collections:
@meleu, can you help us to______________?"
I don't know the answer to that. I don't know what UDb23 is asking you to do either.
OK. Let's try to summarize @Used2BeRX needs, check who can best support him to do what and if what we can do is worth the effort vs using or integrating with other existing metadata.
About the needs (@Used2BeRX please confirm):
A) convert the txt files to XML with specific parsing and specific items/tags generation in the xml (A1- one to one conversion txt to xml A2-generate global gamelist.xml based on all files in a folder)I wrote a windows program (MetaX) to do this; currently finalizing it to include all tags and fix minor conversion issues.
B) As @Used2BeRX wrote
When the program reads "1ST ULTIMATE SUPER MARIO BROS.TXT", I'd like to see it be able to recognize what sub folder the rom "1ST ULTIMATE SUPER MARIO BROS.ZIP" is in and add that path to the <path></path> tag, instead of just putting "./" before the file name.
For this example, using the table I had posted above, this line should read as:
<path>/home/pi/RetroPie/roms/nes/(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP</path>
(I think ./(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP would be the same thing???)
Right now this is how it's reading:
<path>./Frank's First Ultimate Super Mario Bros..zip</path>This is something I think @meleu can do with a script, that would modify the single xml files or even the global gamelist.xml (both generated by MetaX).
C) An additional nice idea would be to generate system specific gamelist.xml based on A) and B) but including only the ROMs that are actually used/installed on the Pi (avoiding to load a huge xml with tons of info related to roms that are not installed; and probably slowing down ES). @Meleu: can this be done ?
-
@UDb23 I think I got it. Will try to sketch something this evening.
-
@UDb23 @meleu Yes UDb23. That seems to be exactly what I was looking for.
Assuming the script works that will locate the rom (with an exact match for the name), I was also hoping that we could take that a step further so that instead of just creating static Media locations as it does now, it would actually find the media with the exact match for the file name, and it would A) only add media tags for existing media and B) put the exact extension of the file match.
Right now, all of the media I have will be exactly as we discussed for those static directories MetaX creates for each entry, but I'm hoping if we get this set up right that anybody at any time could upgrade media and not be concerned with what the extension is as long as the file name is an exact match. This would be important, for instance, when the videos get HD upgrades. Our old system was only SD and the videos currently are in WMV8 format. I have some people right now who claim they are beginning to create MP4 versions of much higher quality, and I was hoping this could recognize that change and create the gamelist.xml accordingly.
Also, I want to create synopsis entries for every one of my folders that point to a single picture, or possibly even a video that would be picked up by MetaX and added to the gamelist.xml (I posted that Atari 7800 example a day or two ago). But we can discuss that part later, and what I would need to do to prepare those.
-
@phulshof I know it shows up when you look at your profile, but emails are blocked to other users. I don't remember where I read that, but I'm not seeing yours.
-
@Used2BeRX said in Making premade gamelist.xml with xtra media collections:
@phulshof I know it shows up when you look at your profile, but emails are blocked to other users. I don't remember where I read that, but I'm not seeing yours.
No worries; I can always use a bit more spam. 😀
-
I've made my first step.
A) convert the txt files to XML with specific parsing and specific items/tags generation in the xml (A1- one to one conversion txt to xml A2-generate global gamelist.xml based on all files in a folder)
As far as I understood @UDb23 is taking care of item A (txt to xml). I did nothing about it.
B) As @Used2BeRX wrote
When the program reads "1ST ULTIMATE SUPER MARIO BROS.TXT", I'd like to see it be able to recognize what sub folder the rom "1ST ULTIMATE SUPER MARIO BROS.ZIP" is in and add that path to the <path></path> tag, instead of just putting "./" before the file name.
I think I completed the item B, but not with that conversion from TXT filename and looking for the ZIP. The script I've made needs a ready gamelist.xml. As far I understood this is enough for you guys. Test the script I'm talking about below and see if it works for your needs.
C) An additional nice idea would be to generate system specific gamelist.xml based on A) and B) but including only the ROMs that are actually used/installed on the Pi
I will try to take care of item C later.
the script
fullpath-gamelist.shCurrently it only puts the full path to the ROM in the
<path>. If the ROM is not found, the script don't change anything.You can see it here: https://github.com/meleu/share/blob/master/fullpath-gamelist.sh
And download it with this command:
wget https://raw.githubusercontent.com/meleu/share/master/fullpath-gamelist.shExecute it with
--helpto see the usage:$ ./fullpath-gamelist.sh --help Usage: ./fullpath-gamelist.sh [OPTIONS] gamelist.xml The OPTIONS are: -h|--help print this message and exit. -u|--update update the script and exit. -d|--directory DIRECTORY look for files in DIRECTORY (default: "/home/pi/RetroPie").Examples
"fix"
<path>for NES gamelist.xml:./fullpath-gamelist.sh ~/RetroPie/roms/nes/gamelist.xml"fix"
<path>for Mega Drive gamelist.xml searching for ROMs in a specific directory:./fullpath-gamelist.sh -d /alternative/roms/megadrive ~/RetroPie/roms/megadrive/gamelist.xmlupdate the script (as I'm going to implement new features, it avoids the need to download the script again using that big-and-hard-to-remember URL)
./fullpath-gamelist.sh --update -
An example of how the gamelist.xml looks like before and after executing the script (pay atention at the
<path>):$ cat gamelist.xml <gameList> <game> <path>./Mega Man (USA).zip</path> <name>Mega Man</name> <desc>The very first Mega Man game for NES.</desc> <publisher>Capcom</publisher> <developer>Capcom</developer> <image>./boxart/Mega Man (USA).png</image> <marquee>./wheel/Mega Man (USA).png</marquee> <video>./snap/Mega Man (USA).mp4</video> <genre>Action</genre> <players>1</players> </game> </gameList> $ ./fullpath-gamelist.sh gamelist.xml Updated entry for "Mega Man (USA).zip". $ cat gamelist.xml <?xml version="1.0"?> <gameList> <game> <path>/home/pi/RetroPie/roms/nes/Mega Man (USA).zip</path> <name>Mega Man</name> <desc>The very first Mega Man game for NES.</desc> <publisher>Capcom</publisher> <developer>Capcom</developer> <image>./boxart/Mega Man (USA).png</image> <marquee>./wheel/Mega Man (USA).png</marquee> <video>./snap/Mega Man (USA).mp4</video> <genre>Action</genre> <players>1</players> </game> </gameList> -
@meleu said in Making premade gamelist.xml with xtra media collections:
As far as I understood @UDb23 is taking care of item A (txt to xml).
yes
An example of how the gamelist.xml looks like before and after executing the script
NIce ! thanks a lot.
-
@meleu @UDb23 Thanks for working on this guys.
I've read meleu's post a few times and I just got to admit that I don't have the coding IQ to really give any opinion on how the progress is going until I can see an updated version in action.
I'd love to be able to give more input at this stage for you guys, but I'm going to have to be able to run the program and see how it outputs things to be able to weigh in on the subject. I hope you understand.
Really looking forward to test driving the next update. :)
In the mean time, if you guys could come up with a way you want me to write up the {folder}.txt files for a system, that would be great. I have a TON of sub-folders over all of the systems and that's going to take me quite a bit of time to write them up by hand. Would love to have them ready to go to work along with the program when it is capable of handling them.
Just so you don't have to scroll up to look for it again, I'll take the liberty of posting what a working entry in the gamelist.xml for a folder looks like:
<folder> <path>./(1) Licensed</path> <image>/home/pi/RetroPie/Media/A7800/Artwork/Console/7800.png</image> <name>(1) Licensed Atari 7800 ProSystem Games</name> <releasedate>1982</releasedate> <developer>Atari Inc.</developer> <publisher>Ateri Inc.</publisher> <desc>LICENSED ATARI 7800 PROSYSTEM GAMES ========================================== The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986. It is almost fully backward-compatible with the Atari 2600, the first console to have backward compatibility without the use of additional modules. It was considered affordable at a price of US$140. The 7800 has significantly improved graphics hardware over the 2600, but uses the same audio chip. It also shipped with a different model of joystick than the 2600-standard CX40. The 1986 launch is sometimes referred to as a "re-release" or "relaunch" because the Atari 7800 had originally been announced on May 21, 1984, to replace Atari Inc.'s Atari 5200, but a general release was shelved due to the sale of the company. A few units were released to test markets in June 1984 though. </desc> </folder>I want to know how you want me to write these as individual text files, and if you would like an identifier at the beginning of the file name like "FOLDER_XXXXXXXX.TXT"
-
@Used2BeRX said in Making premade gamelist.xml with xtra media collections:
I don't have the coding IQ to really give any opinion on how the progress is going until I can see an updated version in action.
Look the comparison I posted here. Pay atention at the
<path>xml element before and after I run the script. I need to know if it's what you want.Try executing the script yourself and see it in action. ;-)
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.