Making premade gamelist.xml with xtra media collections

cafarellidigital

Hey everyone, I started following this in the beginning, but I've been away and busy the last few weeks, and lost track of where this is at now. It seems it's been successful, but not sure what that means.

Are there now premade gamelist.xml files along with images and videos? Are these available for download? If not now, will they be eventually?

Used2BeRX

@cafarellidigital Right now, it's more of a "proof of concept" that has worked.

The only thing that I have tested that works for sure is the option of the script that just includes the rom path and the path to the Box Front image. Still untested are all other types of artwork, videos and other media that is not currently even usable in EmulationStation and would need to have functions coded in to make use of.

What I have right now is pretty much perfect for the Pi Zero. Low-res boxart and minimal gamelist.xml files that do not have any of the game descriptions in them to keep their size as small as possible.

The great news though is that I have now proven that you can have gamelist entries for nearly 8,000 games over nearly 20 console systems and display small boxart without making RetroPie hang, and without breaking any of the emulator functions.

When I was using artwork that hadn't been resized (to 180x240, 240x180 or 210x210), and I had full XMV files with the game description and all of the media being pointed to for each game, the EmulationStation was seriously lagging on startup, I would have major slow downs while scrolling through games, and the Atari 5200 and SNES emulators even refused to launch any games.

We're still quite a way from having anything uploaded and ready to go. I'm open to doing that, but I'd need to know a place here where literally 100,000 to 200,000 or more pieces could be hosted.

Given that I cannot share roms here, there would also have to be some sort of way to get people to have the roms they provide in the correct folders, since there are easily over 100-150 sub folders throughout the different console systems included. To my knowledge, there is no way to create datfiles that have sub-folders for the roms. (If anybody knows a way to do this, please let me know).

Without these sub-folders for the games, you would have a very messy romlist that included the original official games, games from other countries, translated games, prototypes, pirates, game hacks and others all in a single folder. For the NES that would be a single folder of nearly 1,800 games to scroll through of all different varieties.

Right now I am working with a guy to "upgrade" our sets to the no-intro standard as much as possible. Many of our roms are not in no-intro like game hacks and translations and such, but the goal is to at least have a no-intro rom for all of the official releases by region. This isn't for me, since I know the roms I have work and are personally tested by me on another platform. This is to make it easier for people using a datfile to match their rom names to ours so they would work with the sets.

There's still a ton of work to be done, and a ton more getting these things available someplace online where they won't be taken down and setting it up in such a way where it's not still a huge chore for the end user.

I'm open to any suggestions you may have about any of that.

mediamogul

@Used2BeRX

getting these things available someplace online where they won't be taken down

Why would anything be taken down?

Used2BeRX

@mediamogul I can't think of any specific reason since there wouldn't be roms. Torrent sites over the years had a lot of it, but they're all shut down now. We used to upload to places like Megaupload back in the day too and all of the sudden links would disappear. It takes a lot of effort to make stuff like this publicly available only to see it disappear after the fact. I'd need to be relatively confident that they weren't going anywhere for a long while before I put all that effort into something again.

EDIT: Believe it or not, providing this stuff without roms is much more difficult for me than with them, so there is a lot of extra work that goes into providing this stuff without them.

I could just dump it all in zips somewhere, but what good is that if none of it is easily usable. Without spending tons of hours making the datfiles that people could actually use their roms with our stuff, they would have to find and rename nearly 10,000 roms themselves by hand.

That's a big reason why we're working on at least getting the offical releases upgraded to no-intro since many of them were not part of no-intro, tosec or goodsets. That right there is proving to be a HUGE effort that is of no benefit to me personally since I know the roms we have are fine.

Unless somebody knows of a way to make datfiles that will read roms in sub folders and place them in sub-folders for the end user, I would also have to create a dat-file for every single sub-folder, and that is not a quick or easy process in and of itself.

EDIT2: I'm not bitching or anything. Just explaining that this is going to be a major project, and letting anybody interested know not to expect anything to happen soon. We're a long way from there. It's still in a conceptual state. I'm not even sure what the best way to go about it would be.

Of course the easiest way would be to just make an image of my setup when it's completed that somebody could just burn to an SD card and be done with it, but even if that were possible here I'm still a long way off from having even just that.

cafarellidigital

@Used2BeRX ahh ok, thanks for that great explanation!

Regarding the usability of this "database", I understand what you mean when you say that it's easier to provide it with the roms than without, but isn't this mostly due to file and folder naming conventions? And wouldn't it be easier to rename my rom files to accommodate your conventions?

For instance, I only have about 50 NES roms. That's not that many to rename. If you have a gamelist.xml with perfect consistency and images/videos/marquees of consistent size and quality, I think it would be infinitely easier to just rename my roms to match, no?

Used2BeRX

@cafarellidigital It would definately be easier for you to rename your roms in any case, but with only 50 games it wouldn't be hard at all.... In theory.

The problem comes in the file locations for your roms and media. If you had the media from any future packs and put them in the right spot, the media would not be a problem, but you would likely want to go in after the fact and delete a ton of it to not waste space on your Pi. The artwork really doesn't take up all that much space, but without checking I would say that the videos for the NES collection alone would probably be somewhere in the area of 10-20GB for all 1,700+ games, and since you'd only be using 50 of them that's a ton of wasted space.

The other problem comes in the file structure for your roms. Because of the nature of the gamelist.xml files, if your names aren't a perfect case match in the EXACT folders that mine were in, they will just show the generic rom name in the list without benefiting from any of the media.

For example, if you just had 50 roms in the /home/pi/RetroPie/roms/nes/ directory, you would not see any of the work. If they were all US roms, they would need to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/. It gets even more complicated if some of the games you had were from Europe, or were translations. Europe games would have to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_4) Europe Licensed/, and translations would need to be in /home/pi/RetroPie/roms/nes/(2) Translated/`.

I hope that paints a better picture of how difficult this will be to try to organize and present in a way that is worth the time for the end user when it needs to be done without roms included. :)

Used2BeRX

@meleu Hey meleu. Did you say that the script is set to look for CUE files when doing CD systems like Sega CD, Turbografx CD and PSX?

I just discovered that the Pi Zero amazingly seems to play Sega CD games perfectly (at least the few that I tried). There is no way I could fit them all on the small SD card I had, but I'm going to copy over a bunch of txt files that are renamed to the exact CD/synopsis names I have with a .CUE extension and run your script to get the gamelist.xml entries and the path and box front info in them.

Then I was going to put a few games on there at a time and test out that they're all working right. :)

Used2BeRX

@Used2BeRX Wow.... I don't believe it, but every single game for both the SegaCD and the TurboGrafx CD that I've tried works just fine on a Pi Zero! I've had this thing for months and didn't even bother testing either of those systems. Mainly because I didn't have the proper images (mine were ISO/MP3/CUE).

Also, since the Pi Zero struggles with NES and SNES emulation unless you're using quicknes and pisnes, I didn't think there was any way that a 500MB+ CD based game was going to have a chance on this system.

So... Once I get everything properly renamed, here's what I'm going to do....

I'm going to copy all of the CUE sheets to a mirrored folder layout.
I'm going to copy these mirrored layouts to the proper place that the folder synopsis TXT files are assuming the games will be.
If the script is set up right and it will look for .CUE files on SegaCD and Turbo Grafx CD, the gamelist.xml should add all of these games even if the BIN or ISO is not there.
Then I'll be able to do groups of games to test to make sure they're all working.

As far as I know, there is no compatibility spreadsheet for the Pi Zero and Sega CD/TurboGrafx CD.... correct guys?

Let me know if you're interested. I'll make sure the spreadsheet doesn't have anything linking to any roms or anything like that on it. I copy and pasted this stuff from an old spreadsheet of ours from about a decade ago, so there may be some identifying things in there that I will make sure to remove before giving this to you.

I'd say if a game is marked as working on the Pi Zero that there shouldn't be any problems with it on any Pi that's more powerful. :)

meleu

@Used2BeRX please, consider learning the basics of github and then create a repository for your synopsis text files there. I think it would be useful for the community (and would make me recover the will to work on that script :) ).

Used2BeRX

@meleu How would they be useful to the community by themselves?

I'm not arguing the point, I'm just curious where you're going with that, and why that would make you recover the will to work on the script :)

EDIT: Knowing what you have in mind would also help me plan a way to present the information. I don't have any idea what you think could be done with these files alone, so if you let me know what you're thinking I could plan it better than just throwing everything up haphazardly.

Used2BeRX

@meleu Hey meleu.

I did what I said I was going to do and after naming everything correctly to match the synopsis and artwork, I uploaded all of the .CUE files to the Pi Zero for both the TurboGrafx CD and the Sega CD.

PC Engine/Turbo CD works perfectly! 100% match on the first try without having to edit any of the synopsis files. :)

But the Sega CD didn't find any of the .CUE files. (It did find the artwork though).

I was wondering if you forgot to add .CUE file search to segacd or possibly you added that code to the genesis/megadrive by mistake??

meleu

@Used2BeRX I'm away from my main computer (where I have the synopsis.txt), but the searching for .cue happens if the Media: CD in the text files, no matter what is in the Platform: field.

Can you check if your segacd synopsis files have Media: CD?

By the way, this is the line of code responsible for that:
https://github.com/meleu/used2betxt/blob/master/Used2BeTXT.sh#L183

Used2BeRX

@meleu Ahhh. I see. Is there a way I can temporarily alter that real quick?

I just checked both the Sega CD and the PSX synopsis files and, of course, they have Media: Compact Disc.

I was wondering if I could do a quick alteration to include both CD or Compact Disc until you get a chance to put it in your official script so when I make an update in the future it's still in there.

BTW... you didn't answer my question about what you were thinking with the synopsis files on github. I still have some alterations I want to make before I would put them up there, but I wanted to know what your plans would be with them.

Now that I know your script doesn't make case sensitivity an issue anymore like MetaX did, I am getting tired of looking at everything on my end IN ALL CAPS!!!!! :)

I'm going to take a day or so to Bulk Rename everything to CaMeL CaSe, re-upload everything to the pi and re-run the script for all of the systems. So I wouldn't even think about uploading the synopsis files until I do that first.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu How would they be useful to the community by themselves?

I thought that those files (that Xtras stuff) from the XBox community were popular, then I thought it would be useful for those guys who are willing to migrate to RetroPie. But TBH I have no idea how popular it is.

why that would make you recover the will to work on the script :)

Because it would benefit the community. :)

EDIT: Knowing what you have in mind would also help me plan a way to present the information.

Well, to be honest I don't have a clear idea of what to do next. I have some ideas, but not a clear one.

Anyway, having those files easily available on github (and not those obscure mega/sendspace links) would increase the chance to involve people to do something with that info.

What I found most valuable in your files was the synopses for hacks and homebrews, maybe those descriptions should be submitted to some of the popular databases.

Used2BeRX

@meleu I literally replied to you about 3 seconds before you replied. I did talk about what you just talked about, as well as had a question regarding the script. :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I was wondering if I could do a quick alteration to include both CD or Compact Disc until you get a chance to put it in your official script so when I make an update in the future it's still in there.

I like to follow a principle that says:
The software should adapt to humans, not vice versa.

That's why I am constantly asking to not change your files. :)

I'll make the script recognize Compact Disc as CD, so no need to change your files.

EDIT: no rush to post your files to github, but consider posting when you think it's ready! :-)

Used2BeRX

@meleu Of course :)

I was asking if there was a way I could temporarily alter your script so I could re-run it on my end right now since you were away from it yourself.

Just to be sure though, keep the CD in there as well as Compact Disc. The CD works perfect for TurboGrafx CD.

BTW... if you want to share any of your ideas that you mentioned you might have for the synopsis and what not, feel free to let me know what they are.

I would like to share this with people, but as you can imagine it's a daunting task. If sharing roms wasn't a problem I could just make some huge SD card images and be done with it. Since that's not an option there's about 100 times more work on my end to try to figure out a way to make this work for other people.

meleu

@Used2BeRX

I was asking if there was a way I could temporarily alter your script so I could re-run it on my end right now

Not exactly in the script, but you can replace every Compact Disc by CD in the synopsis with sed. Do the following:

change the directory to where the synopsis are placed:

cd /path/to/segacd/synopsis

sed can do the magic:

sed -i 's/Compact Disc/CD/' *.txt

(that .txt is case sensitive, so it can be .TXT on your end)

meleu

@Used2BeRX don't mind about the sed stuff above. I already changed the script. Please --update and test it.

Used2BeRX

@meleu Wow, that's a cool trick. I literally changed over 100 entries for the `Famicom Disk System~ by hand the other night. It took me about 20 minutes.

I'm just a big dumb donkey. I like to surround myself with smart people. :)

I'm not going to change anything. I've got some other stuff to do and I'm just going to turn in early after I can get some file transfers cued up to my thumb drives so I can start testing the Sega CD and Turbo CD games tomorrow hopefully. I'll just wait until you can change the script.