Making premade gamelist.xml with xtra media collections

mediamogul

getting these things available someplace online where they won't be taken down

Why would anything be taken down?

Used2BeRX

@mediamogul I can't think of any specific reason since there wouldn't be roms. Torrent sites over the years had a lot of it, but they're all shut down now. We used to upload to places like Megaupload back in the day too and all of the sudden links would disappear. It takes a lot of effort to make stuff like this publicly available only to see it disappear after the fact. I'd need to be relatively confident that they weren't going anywhere for a long while before I put all that effort into something again.

EDIT: Believe it or not, providing this stuff without roms is much more difficult for me than with them, so there is a lot of extra work that goes into providing this stuff without them.

I could just dump it all in zips somewhere, but what good is that if none of it is easily usable. Without spending tons of hours making the datfiles that people could actually use their roms with our stuff, they would have to find and rename nearly 10,000 roms themselves by hand.

That's a big reason why we're working on at least getting the offical releases upgraded to no-intro since many of them were not part of no-intro, tosec or goodsets. That right there is proving to be a HUGE effort that is of no benefit to me personally since I know the roms we have are fine.

Unless somebody knows of a way to make datfiles that will read roms in sub folders and place them in sub-folders for the end user, I would also have to create a dat-file for every single sub-folder, and that is not a quick or easy process in and of itself.

EDIT2: I'm not bitching or anything. Just explaining that this is going to be a major project, and letting anybody interested know not to expect anything to happen soon. We're a long way from there. It's still in a conceptual state. I'm not even sure what the best way to go about it would be.

Of course the easiest way would be to just make an image of my setup when it's completed that somebody could just burn to an SD card and be done with it, but even if that were possible here I'm still a long way off from having even just that.

cafarellidigital

@Used2BeRX ahh ok, thanks for that great explanation!

Regarding the usability of this "database", I understand what you mean when you say that it's easier to provide it with the roms than without, but isn't this mostly due to file and folder naming conventions? And wouldn't it be easier to rename my rom files to accommodate your conventions?

For instance, I only have about 50 NES roms. That's not that many to rename. If you have a gamelist.xml with perfect consistency and images/videos/marquees of consistent size and quality, I think it would be infinitely easier to just rename my roms to match, no?

Used2BeRX

@cafarellidigital It would definately be easier for you to rename your roms in any case, but with only 50 games it wouldn't be hard at all.... In theory.

The problem comes in the file locations for your roms and media. If you had the media from any future packs and put them in the right spot, the media would not be a problem, but you would likely want to go in after the fact and delete a ton of it to not waste space on your Pi. The artwork really doesn't take up all that much space, but without checking I would say that the videos for the NES collection alone would probably be somewhere in the area of 10-20GB for all 1,700+ games, and since you'd only be using 50 of them that's a ton of wasted space.

The other problem comes in the file structure for your roms. Because of the nature of the gamelist.xml files, if your names aren't a perfect case match in the EXACT folders that mine were in, they will just show the generic rom name in the list without benefiting from any of the media.

For example, if you just had 50 roms in the /home/pi/RetroPie/roms/nes/ directory, you would not see any of the work. If they were all US roms, they would need to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_1) US Licensed/. It gets even more complicated if some of the games you had were from Europe, or were translations. Europe games would have to be in /home/pi/RetroPie/roms/nes/(1) Licensed/(1_4) Europe Licensed/, and translations would need to be in /home/pi/RetroPie/roms/nes/(2) Translated/`.

I hope that paints a better picture of how difficult this will be to try to organize and present in a way that is worth the time for the end user when it needs to be done without roms included. :)

Used2BeRX

@meleu Hey meleu. Did you say that the script is set to look for CUE files when doing CD systems like Sega CD, Turbografx CD and PSX?

I just discovered that the Pi Zero amazingly seems to play Sega CD games perfectly (at least the few that I tried). There is no way I could fit them all on the small SD card I had, but I'm going to copy over a bunch of txt files that are renamed to the exact CD/synopsis names I have with a .CUE extension and run your script to get the gamelist.xml entries and the path and box front info in them.

Then I was going to put a few games on there at a time and test out that they're all working right. :)

Used2BeRX

@Used2BeRX Wow.... I don't believe it, but every single game for both the SegaCD and the TurboGrafx CD that I've tried works just fine on a Pi Zero! I've had this thing for months and didn't even bother testing either of those systems. Mainly because I didn't have the proper images (mine were ISO/MP3/CUE).

Also, since the Pi Zero struggles with NES and SNES emulation unless you're using quicknes and pisnes, I didn't think there was any way that a 500MB+ CD based game was going to have a chance on this system.

So... Once I get everything properly renamed, here's what I'm going to do....

I'm going to copy all of the CUE sheets to a mirrored folder layout.
I'm going to copy these mirrored layouts to the proper place that the folder synopsis TXT files are assuming the games will be.
If the script is set up right and it will look for .CUE files on SegaCD and Turbo Grafx CD, the gamelist.xml should add all of these games even if the BIN or ISO is not there.
Then I'll be able to do groups of games to test to make sure they're all working.

As far as I know, there is no compatibility spreadsheet for the Pi Zero and Sega CD/TurboGrafx CD.... correct guys?

Let me know if you're interested. I'll make sure the spreadsheet doesn't have anything linking to any roms or anything like that on it. I copy and pasted this stuff from an old spreadsheet of ours from about a decade ago, so there may be some identifying things in there that I will make sure to remove before giving this to you.

I'd say if a game is marked as working on the Pi Zero that there shouldn't be any problems with it on any Pi that's more powerful. :)

meleu

@Used2BeRX please, consider learning the basics of github and then create a repository for your synopsis text files there. I think it would be useful for the community (and would make me recover the will to work on that script :) ).

Used2BeRX

@meleu How would they be useful to the community by themselves?

I'm not arguing the point, I'm just curious where you're going with that, and why that would make you recover the will to work on the script :)

EDIT: Knowing what you have in mind would also help me plan a way to present the information. I don't have any idea what you think could be done with these files alone, so if you let me know what you're thinking I could plan it better than just throwing everything up haphazardly.

Used2BeRX

@meleu Hey meleu.

I did what I said I was going to do and after naming everything correctly to match the synopsis and artwork, I uploaded all of the .CUE files to the Pi Zero for both the TurboGrafx CD and the Sega CD.

PC Engine/Turbo CD works perfectly! 100% match on the first try without having to edit any of the synopsis files. :)

But the Sega CD didn't find any of the .CUE files. (It did find the artwork though).

I was wondering if you forgot to add .CUE file search to segacd or possibly you added that code to the genesis/megadrive by mistake??

meleu

@Used2BeRX I'm away from my main computer (where I have the synopsis.txt), but the searching for .cue happens if the Media: CD in the text files, no matter what is in the Platform: field.

Can you check if your segacd synopsis files have Media: CD?

By the way, this is the line of code responsible for that:
https://github.com/meleu/used2betxt/blob/master/Used2BeTXT.sh#L183

Used2BeRX

@meleu Ahhh. I see. Is there a way I can temporarily alter that real quick?

I just checked both the Sega CD and the PSX synopsis files and, of course, they have Media: Compact Disc.

I was wondering if I could do a quick alteration to include both CD or Compact Disc until you get a chance to put it in your official script so when I make an update in the future it's still in there.

BTW... you didn't answer my question about what you were thinking with the synopsis files on github. I still have some alterations I want to make before I would put them up there, but I wanted to know what your plans would be with them.

Now that I know your script doesn't make case sensitivity an issue anymore like MetaX did, I am getting tired of looking at everything on my end IN ALL CAPS!!!!! :)

I'm going to take a day or so to Bulk Rename everything to CaMeL CaSe, re-upload everything to the pi and re-run the script for all of the systems. So I wouldn't even think about uploading the synopsis files until I do that first.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

@meleu How would they be useful to the community by themselves?

I thought that those files (that Xtras stuff) from the XBox community were popular, then I thought it would be useful for those guys who are willing to migrate to RetroPie. But TBH I have no idea how popular it is.

why that would make you recover the will to work on the script :)

Because it would benefit the community. :)

EDIT: Knowing what you have in mind would also help me plan a way to present the information.

Well, to be honest I don't have a clear idea of what to do next. I have some ideas, but not a clear one.

Anyway, having those files easily available on github (and not those obscure mega/sendspace links) would increase the chance to involve people to do something with that info.

What I found most valuable in your files was the synopses for hacks and homebrews, maybe those descriptions should be submitted to some of the popular databases.

Used2BeRX

@meleu I literally replied to you about 3 seconds before you replied. I did talk about what you just talked about, as well as had a question regarding the script. :)

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I was wondering if I could do a quick alteration to include both CD or Compact Disc until you get a chance to put it in your official script so when I make an update in the future it's still in there.

I like to follow a principle that says:
The software should adapt to humans, not vice versa.

That's why I am constantly asking to not change your files. :)

I'll make the script recognize Compact Disc as CD, so no need to change your files.

EDIT: no rush to post your files to github, but consider posting when you think it's ready! :-)

Used2BeRX

@meleu Of course :)

I was asking if there was a way I could temporarily alter your script so I could re-run it on my end right now since you were away from it yourself.

Just to be sure though, keep the CD in there as well as Compact Disc. The CD works perfect for TurboGrafx CD.

BTW... if you want to share any of your ideas that you mentioned you might have for the synopsis and what not, feel free to let me know what they are.

I would like to share this with people, but as you can imagine it's a daunting task. If sharing roms wasn't a problem I could just make some huge SD card images and be done with it. Since that's not an option there's about 100 times more work on my end to try to figure out a way to make this work for other people.

meleu

@Used2BeRX

I was asking if there was a way I could temporarily alter your script so I could re-run it on my end right now

Not exactly in the script, but you can replace every Compact Disc by CD in the synopsis with sed. Do the following:

change the directory to where the synopsis are placed:

cd /path/to/segacd/synopsis

sed can do the magic:

sed -i 's/Compact Disc/CD/' *.txt

(that .txt is case sensitive, so it can be .TXT on your end)

meleu

@Used2BeRX don't mind about the sed stuff above. I already changed the script. Please --update and test it.

Used2BeRX

@meleu Wow, that's a cool trick. I literally changed over 100 entries for the `Famicom Disk System~ by hand the other night. It took me about 20 minutes.

I'm just a big dumb donkey. I like to surround myself with smart people. :)

I'm not going to change anything. I've got some other stuff to do and I'm just going to turn in early after I can get some file transfers cued up to my thumb drives so I can start testing the Sega CD and Turbo CD games tomorrow hopefully. I'll just wait until you can change the script.

meleu

@Used2BeRX said in Making premade gamelist.xml with xtra media collections:

I'll just wait until you can change the script.

I already changed the script. We are posting almost at the same time! :-)

Used2BeRX

@meleu Heh.. yeah, I saw that, but I had already moved my keyboard to the Pi to update and run it.

100% success rate with boxart on Sega CD cue files. :)

Now I have to figure out how to use a USB stick on the Pi for CD testing. I had all the images for the Sega and Turbo CD in ISO/MP3 format and it just just a little over 50 GB. Now that they're all in BIN/CUE and ISO/CUE they're over 150GB!!!!

I'll be doing that and in between re-casing everything like I said I would. AFter that I should be able to look into github.

Thanks again man.