Making premade gamelist.xml with xtra media collections

meleu

@UDb23 said in Making premade gamelist.xml with xtra media collections:

Any suggestions ?

Try to guess... Yes, shell script. :-)

It seems to be easy to implement. The most challenging part will be the ROM filename matching (same with our rpie-art).

I can help with it. I'll try to take a look into it on Wednesday.

UDb23

@meleu nice, thanks :-)
Meanwhile I'll fix a few bugs in the conversion Program.

meleu

@UDb23 The conversion to xml is really simple and straightforward to do via shell script, a few lines of sed tricks and, voilá, xml ready! If you guys really need to do it on windows, maybe it's time to know the cygwin. ;-)

About the scraper, did he ( @Used2BeRX ) put the txt/xml files in some repo?

A quick reading through the posts in this thread makes me believe that the files are at some of those web servers that don't allow downloads directly from command line.

@Used2BeRX putting your files in github really increases the chances to other people contribute with your hardwork and use it. If you don't know how it works, consider to learn at least the basics.

I see three pros:

The txt files seems to be human readable, then anyone would be able to contribute.
Users could download files directly from the command line (pi users love this feature).
Your work would be easily accessible (a friendly URL, not those random chars of those dropbox-like web servers you're using).

P.S.: I reread my post and maybe I'm sounding like a smartass. Sorry, this is not my intention. :-)

Cheers.

UDb23

@meleu you're not sounding like that; don't worry. ;-)
About the program I already made it in Visual Studio 2017 (vb); you won't believe it but Microsoft gives the community edition for free!
These text files are not all structured in the same way so there still are a few fixes to be done to parse them correctly. No Problem on this side; it's fun to me to re-start some coding in visual basic after so many years. Working on a real project is far better then just studying; and vb has changed a lot over the years.

Used2BeRX

@UDb23 said in Making premade gamelist.xml with xtra media collections:

Now he'd like to automate the install of metadata and media only for the roms a user actually has installed. Basically like scraping roms using local metadata and media instead of online. Any suggestions ?

Actually, although that might not be a bad idea for future usability for other people without my setup, that's not exactly what I was asking. (And I noticed another issue while putting together this example):

When the program reads "1ST ULTIMATE SUPER MARIO BROS.TXT", I'd like to see it be able to recognize what sub folder the rom "1ST ULTIMATE SUPER MARIO BROS.ZIP" is in and add that path to the <path></path> tag, instead of just putting "./" before the file name.

For this example, using the table I had posted above, this line should read as:

<path>/home/pi/RetroPie/roms/nes/(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP</path>

(I think ./(3) Unlicensed - Homebrew/(3_10) Super Mario Bros Updates/1ST ULTIMATE SUPER MARIO BROS.ZIP would be the same thing???)

Right now this is how it's reading:

<path>./Frank's First Ultimate Super Mario Bros..zip</path>

No path, but the other problem I noticed is that it's taking the altered game name instead of using the rom file name. In this example that means the file name is incorrect and the Case is wrong because the case of the TXT file and ZIP file and all Xtras will be all Caps "XXXXXX.XXX"

Hope that clears up the confusion. I also hope it can be done. That's the 10,000 or so lines that I would be forced to do by hand that I mentioned before if we can't do this.

Used2BeRX

@meleu At this point I'm really only concerned with getting everything up and running right with good gamelist.xml files that aren't going to take me a year or two to write by hand.

I plan on sharing the work at that point, but I don't have any interest in putting all of that work into re-hosting stuff here. It's easily over 100,000 files, and once videos are thrown in the mix it's a giant collection that I doubt would fit anywhere here.

I think I mentioned to Herb Fergus that I'm alright with somebody else putting it all up as long as credit is given to the team behind the work. That would all be in a readme when a release comes out.

meleu

@Used2BeRX OK man. As the Overlay Master requested, I'll write the scraper. I just need to know where the xml files are supposed to be placed.

UDb23

@meleu if I'm the overlay master then you're the Script Grand Master ;-).
@Used2BeRX ok, misunderstood original request. Now clear.

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

No path, but the other problem I noticed is that it's taking the altered game name instead of using the rom file name. In this example that means the file name is incorrect and the Case is wrong because the case of the TXT file and ZIP file and all Xtras will be all Caps "XXXXXX.XXX"

I modified the program: now the exact TXT filename (with .zip extension) will be used for the <path> tag.
So if TXT filename corresponds to ROM filename (case included) the xml will inlcude the correct name for the rom file.

Actual path were the ROM resides will then be found by @meleu 's new script, that will also automagically modify the path tag in the .xml accordingly.

UDb23

@Used2BeRX New version (06) uploaded.

No more errors (as far as I could check) with txt files import.
Error was due to max number of lines limit I used (was not expecting that long descriptions). Now basically unlimited.
ROM name exactly as txt filename (as mentioned previously)
"convert all" checkbox allows to convert all txt files in folder (tested on 7800 TXTs, converted 90 files without any issue).

Next to be implemented:

implement all items/tags you requested (not yet finalized)
custom media path (can be set by user)
create global gamelist.xml (converting all txt in folder)
fix append issue (currently appends to existing xml with same name, if file already exists)

Used2BeRX

@UDb23 It works without crashing now. I did the Atari 7800 (90) for myself and I also did the NES (1,756 files!). I'm pretty sure that with a group that large not spitting out any errors that we're good on that now.

Two issues I noticed so far. (I'll study the output files and see if I find any more).

Zip file romname is showing the proper case now, but the extension is a lowercase ".zip". Can your program recognize that and case the extension accordingly? For my purposes, every file involved now will be all uppercase to avoid any confusion, so forcing it is also acceptable, but that will limit the use of the program for others not using my collection in the future.
All other media is still showing the game title instead of the romname, so the lines are wrong and the case is incorrect. Everything in the <artwork-type><movies><manual><vgmap><marquee>, etc. should be the exact same as the file name in the <path> tag (the extension will be different though of course).

Actual path were the ROM resides will then be found by @meleu 's new script, that will also automagically modify the path tag in the .xml accordingly.

I'd like to test this out. Was this implemented in this change, or you're going to put it in a future update? If it's in there now, how do I have to have the file structures set up since there is never a question about where the roms are found?

Future ideas:

If this script works, perhaps we can do even more upgrades to the program that would be similar to this one. For instance, a person might have artwork that is mixed between .JPG and .PNG. Using a similar script the exact file names could be put on each line of the tags (and also you could have the ability to completely skip tags if you didn't have the particular artwork).

I may or may not have somebody upgrading a lot of the videos we have. The old videos were .WMV and the new videos would be MP4. That would be awesome to have the program determine what extension to put in there.

UDb23

Glad to hear it works and was able to manage that huge number of files :-)

1.Can your program recognize that and case the extension accordingly?

Should be able to detect actual case of extension of original file and apply to .zip accordingly.

2 All other media is still showing the game title .....should be the exact same as the file name in the <path> tag (the extension

I've overseen this; sure I can do. Very easy fix

Actual path were the ROM resides...

Current idea is that my program creates the xml files (eventually with some user customizable path for roms and media; to be set by user).

@meleu very kindly agreed to write a new specific script that would "scrape" (check) where actual roms are and automatically modify the required xml files. I'll wait for his feedback and proposal on this, including where xml file should be available to be accessed thru the web from his script (e.g. github). This would avoid user the need to download ALL xml files locally just to use/install only the ones he has roms for.

UDb23

btw I have complete metadata for VIc20 roms I got thru a combination of an own written scraper (mobygames based) and other sources found on the web. Also have most cover art. Will try to find a way to generate the related xml for Retropie like I did for your txt files.
Likely it will be an additional convert command/option of MetaX.

Used2BeRX

@UDb23 Sounds good man.

I know you got a lot on your plate with this now and I don't want to bog you down with requests on it, but I was thinking about how to handle the sub-folders. Currently, I had written entries for the folders on the Atari gamelists that have information about the console itself as well as pointers to a .PNG file that shows a nice pic of the system.

If you look at the NES sub-folders, there are a TON of different ones. I'd like to make some cool artwork to display for each of these folders as well as a short description for what's inside each of them for the end user.

If I were to make TXT files for every sub-folder for the program to use, how would you suggest I name these, and how would you suggest I put the data in for your program to scrape? There wouldn't be much to add on these as far as media. Currently I'm only putting one picture from a folder called /Artwork/Console/.

I was thinking that for naming them I could start every folder entry with "FOLDER xxxxxxxxx.TXT"?

No need to give an answer on that right away. Id' like to see what you think about it.

I'm hoping that eventually we can get the entire process automated so that only existing media is added to the XML files and if any media is upgraded/added that you could simply re-run the program and get a completely working gamelist.xml file that contains the upgrades without having to worry about re-testing that everything still works and/or going through and making a lot of changes by hand.

UDb23

@Used2BeRX Just a quick answer from my side as @meleu is the master in this area.
I read a post somewhere that folder images can be used in ES but can't remember how this is done. To provide necessary information about suggested folders you may create a info_folder_xxx.txt file or even just a single folderInfo.txt file that includes all necessary folder info.
Please check how we managed this for overlays here.; it may give you some idea. on how to structure it.

Used2BeRX

@UDb23 I already know how to make the folder images to show up. I was asking how to make the "FOLDERxxxxxx.TXT" files work in your program. I haven't made any txt files to run through your program, but here's an example of what a working <folder></folder> entry looks like in my gamelist.xmls for the Atari systems:

<folder>
		<path>./(1) Licensed</path>
		<image>/home/pi/RetroPie/Media/A7800/Artwork/Console/7800.png</image>
		<name>(1) Licensed Atari 7800 ProSystem Games</name>
		<releasedate>1982</releasedate>
		<developer>Atari Inc.</developer>
		<publisher>Ateri Inc.</publisher>
		<desc>LICENSED ATARI 7800 PROSYSTEM GAMES
==========================================
The Atari 7800 ProSystem, or simply the Atari 7800, is a home video game console officially released by Atari Corporation in 1986. It is almost fully backward-compatible with the Atari 2600, the first console to have backward compatibility without the use of additional modules. It was considered affordable at a price of US$140.

The 7800 has significantly improved graphics hardware over the 2600, but uses the same audio chip. It also shipped with a different model of joystick than the 2600-standard CX40.

The 1986 launch is sometimes referred to as a "re-release" or "relaunch" because the Atari 7800 had originally been announced on May 21, 1984, to replace Atari Inc.'s Atari 5200, but a general release was shelved due to the sale of the company. A few units were released to test markets in June 1984 though.
               </desc>
</folder>

My question to you was how would you like these individual sub-folder synopsis.txt files created?

Used2BeRX

@UDb23 It took a couple of days work, but I now have 100% confirmed filename/case matches for every rom, synopsis.txt and low def Pi-Zero artwork. Everything for the Pi-Zero is now completely prepared for testing when the program is ready.

To Do List:
100% match for the highest quality artwork I have for use with a greater Pi.
100% match for videos, manual scans, vgmap zips, gamefaq zips for use with a greater Pi. *

(* No current use for manuals, vgmaps or gamefaqs, but hopefully somebody will find a way to use them in EmulationStation like we can use them on the other platform)

UDb23

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

here's an example of what a working <folder></folder> entry looks like

nice, thanks.

how would you like these individual sub-folder synopsis.txt files

Let's wait for @meleu 's considerations on this; meanwhile I finalize the program.

Used2BeRX

@UDb23 I got pretty much everything re-cased and compared now for both the PiZero and Pi3 setups. I'm still waiting on changing the highest quality artwork to JPG to see if you can make the program recognize what type of file is in the media folders since the original PNG files would look better.

I'm ready to start making those synopsis.txt files for all of the folders whenever you guys have directions for me. Turns out there is going to be quite a bit of rain the next 3-4 days so the gig I had is going to be put off for a few more days at least.

I also had a very long list I put together of about 500-600 romlist names that I wanted to change in those synopsis files a few months back that I finally got around to doing today. I'm not going to say that the names are perfect across nearly 10,000 games, but they're pretty damn close. I tried to make them exactly how they are displayed on the boxart for the games. I've been through that list about 5 or 6 times over the years and I'm really happy with how it looks now.

Looking forward to being able to test more out and see if having all this artwork is going to break my Pi Zero or not. :)

meleu

I'm ready to start.

Here's what I need to actually start coding:

a gamelist.xml for the tests (preference for the classcis: atari2600, nes, megadrive, snes, gameboys... only one is enough).
the art files.
where the art files are supposed to be placed (a quick reading on the posts above made me believe the files will be at ~/RetroPie/Media/{SYSTEM_NAME}, but I need a confirmation from you guys).
please, name the systems the same way as RetroPie does (e.g.: atari2600 and not A2600).