NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?)
-
Hey man. I would be thrilled if github allows for the massive amount of storage we're talking about here and somebody would work with me to get it up there. It was suggested by @meleu quite a while back and I signed up for it but couldn't figure out at all what to do with it and he's too busy to hold my hand. Consider me interested if you or somebody else would be willing to aid in this. I have way too much work on my end here to be spending tons of time trying to figure github out on my own. :)
Since things aren't at a ready state just yet, most of it would not really benefit from it, but something small like the synopsis would be easy to re-up when it gets the final edits. Once I start making the full artwork size mass conversions they would start being ready as well. I doubt though that I'd ever be able to store videos there, would I?
Anyhow.... the note above is actually just a small part of the overall project. It's more just what I'm working on now. I believe a more full picture of the scope of things is in the first 10 or so posts of this thread. I might just create a new thread in the near future when I'm ready to put the spreadsheet up that really illustrates what is already done and what still needs to be done.
Things are backed up right now, but it's always better to have an extra one. :)
-
@used2berx what operating system are you using on the system where you work on this project? Do you have any idea how large the collection might be when it's complete?
I'll try to summarize a github workflow based on your OS/environment.
The second place I would suggest keeping in mind is the Internet Archive at archive.org. I am confident that they would host this collection. Once you have a user account there you can upload through their website, plus there are other methods available to add to the archive if the content is too large to upload through the web.
I'm glad to keep talking with you about this. After all the work to standardize the images themselves, hopefully it won't be too hard to find one or two solid, long-term hosts for your results.
-
@markwkidd I use Windows 10.
I couldn't even begin to imagine how big it will be at this point. All I can say is it will be a lot smaller before the videos are made.
The synopsis files are actually very small. Right now, between the games and the folders there are right around 2,050 of them, and they're all 1kb or less.
Images will be all over the place since I'll have 3 sets of them with different sizes for the Boxes and Carts. The Title and Action shots will be the same for all 3 sets and will be small though. Right now, because there are 4 image types for each set at the moment, this will be about 8,080 images per set.
There actually is a lot of other aspects to the project as well, but currently RetroPie is unable to take advantage of them like the XBox does. (such as manuals in either PDF format or converted to JPG and zipped up).
My "plan" right now was to upload them all together somewhere, where they would need to be then downloaded and transferred over to the correct place on your Pi/XBox. That is the way it must be done regardless on the XBox, but maybe with github people could just download the thing straight to the proper location on their Pi if it were set up correctly?
EDIT: When I say "right now" about the amount of images, it's because I do plan on adding more games before this is all done. I'm sure before I'm done there will be more translations since they seem to pop up every once and a while, and I still haven't really gone in and decided which hacks will make the cut. There were already a lot of them in there from ten years ago, and I only removed a few that were in some way broken. A lot of them probably wouldn't have made the cut now, but since all the work has been done for them at this point except for the videos I'm going to keep the rest. I'm sure that there are still some really great ones out there that I haven't discovered though.
-
OK that makes sense. If you're willing to spend a little time setting up git and github (maybe 30 minutes to an hour if things go smoothly?) I think it would fit into your way of working.
With git, we would designate a root folder on you PC where all of the collection lies within as subfolders. This would sync onto github as the root of your project's github repository, where among other things, it would be easy for people to script it into automated build projects like you were describing.
As you continue to work on your project, you can periodically synchronize to github on whatever schedule you want. The number of files you're proposing (8,080 in this phase) is within the specs of a standard github repository.
Next time you have a chance, make sure you can access your github.com account, and then see if you can create a new "repository" for your project. Once you have access to github.com I can try to help you figure out the best way to install the git client on your Windows 10 PC and designate that root folder so that it doesn't alter your organizations scheme or workflow if at all possible.
-
Here's the libretro thumbnails repository, in fact. This would be a good reference point: https://github.com/libretro/libretro-thumbnails
-
@markwkidd said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):
Here's the libretro thumbnails repository, in fact. This would be a good reference point: https://github.com/libretro/libretro-thumbnails
That's pretty much exactly what I would be looking to do, assuming you could then download them directly to where they needed to go on the Pi. Is that what somebody could do now by running a command from the Pi console?
Actually, once meleu's script is complete, I could also set it up along with the proper shortcuts so that any time I did update things in the future, people could just re-run the script and any changes that were made would be reflected the next time they rebooted. :)
I should have some time this week to look into this. I'll make sure I can still get into my github and we'll take it from there. Thanks a lot for this. It will give a great option for ensuring that anybody trying to use my set(s) will be working with the exact setup I have without worrying they did it wrong.
BTW... how does it handle folders? I would have empty folders for all of the sub-categories for the roms. Would I have to have an empty text file in these folders for them to actually copy over?
I'm hoping to find somebody who can make a script for me at some point that would automatically move all of the roms they found to the right folders after they run them against my datfile and they're renamed. There are still a lot of more obscure games that people would have to hunt down and/or patch on their own, but once you see my spreadsheet you'll see that I've heavily detailed all of this and that a large portion of them can be found through full GoodNES/No-Intro/TOSEC sets.
Maybe I can make datfiles with sub-folders in them? I know I can make one for each sub-folder, but beyond that I kind of get lost.
-
@used2berx said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):
That's pretty much exactly what I would be looking to do, assuming you could then download them directly to where they needed to go on the Pi. Is that what somebody could do now by running a command from the Pi console?
Yes. Right now you could, for example, type this single git command into a console on a pi or PC with git installed and it will download the whole libretro thumbnail collection:
git clone http://github.com/libretro/libretro-thumbnails.gitSlightly more sophisticated commands could sync only parts of the repository.
BTW... how does it handle folders? I would have empty folders for all of the sub-categories for the roms. Would I have to have an empty text file in these folders for them to actually copy over?
I am not positive, but I do think you might have to put a text file in order to get an empty folder to sync to a github repository. Alternately if this is going to be installed by a script, you might consider letting the script create the folders on the local machine when the script is executed.
As I said, glad to give a hand if my experience could be useful to you. I'll keep an eye out for your next updates.
-
@markwkidd Awesome man. I will definitely try to make this a part of my work. This seems like the ideal solution, at least for everything besides the videos. This way if I have some minor updates here or there I don't have to re-up the whole thing somewhere else.
BTW... is there a way for people to just be able to update anything after they last downloaded? For instance, if I add a few games (not roms, but all the xtra media), or if I update some artwork or text files/gamelist.xml stuff would they be able to just grab the changes since the last update?
You're right about the script. No sense in putting empty folders here. I'm hoping that I can build a datfile with sub-directories and the script won't even be necessary. I'm a while from that point though.
Thanks so much for the help. I have been pretty sick recently, so the limited amount of time I've felt like working on the project while I wasn't at work I dedicated to the images, so I haven't had any time to look into this yet. I hope you're not going anywhere in a hurry though and I can get some assistance if I need it when I start putting stuff on github.
I should have some updates soon, but this cart project really is taking a lot longer than I thought it would. It looks really sweet though. I don't want to rush it and do a sloppy job.
Later
-
@used2berx said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):
BTW... is there a way for people to just be able to update anything after they last downloaded? For instance, if I add a few games (not roms, but all the xtra media), or if I update some artwork or text files/gamelist.xml stuff would they be able to just grab the changes since the last update?
Yep, github makes incremental uploads and incremental downloads a breeze.
Thanks so much for the help. I have been pretty sick recently, so the limited amount of time I've felt like working on the project while I wasn't at work I dedicated to the images, so I haven't had any time to look into this yet. I hope you're not going anywhere in a hurry though and I can get some assistance if I need it when I start putting stuff on github.
Take your time, this is an impressive undertaking! I hope your health improves.
-
@markwkidd Thanks man. I'll let you know when I start messing around with github. I think I'm going to finish the carts first, and then I wanted to go through the synopsis files (that make the gamelist.xml with meleu's script) and do some editing with the genre tags. I think right now there's something insane like 200 different types of genre's and most of that is just due to the order of the game types not being standard. Once I get that done, I'll probably start with the synopsis files on github since they're small and it will be real quick to upload and change things.
Quick update on the cart status.
Finished categories: US Licensed, Europe Licensed, VS Arcade games, and all game hacks.
Almost finished: Japan Licensed (untranslated), Adult Games, Zapper Games, Unsupported Controller games.
To Do: All FDS disks, Translated Games, Unlicensed Games, Prototypes, Pirates and Original Homebrew.
It's hard to count them right now since I've hidden some stuff on the spreadsheet to make this work easier so I don't have specific counts by type, but if I had to guess I've probably individually edited around 1,200 to 1,300 of the 2,019 current titles. Some of the more obscure titles are going to be tricky because the cart image sizes for them vary pretty wildly, so it's going to be a lot less mindless work than most of what I've already done.
In the mean time, we have 3 new translated games that came out recently that will be added as well :)
-
@markwkidd and anybody else following this thread :)
There has been a few developments on the side that are exciting for this project.
- The first is an AWESOME
gamelist.xmlto Spreadsheet script that was written and shared to us by @mitu that was just pointed out to me this morning. I'm super stoked on this and I really hope that he sees my request and can find the time to reverse this and get it so we can then convert the Spreadsheet back togamelist.xmlThis will be excellent news for any future synopsis editing and save loads of time. Working on the synopsis files is one of the most tedious aspects of the process, and being able to do the work for an entire system on a spreadsheet would be wonderful.
Thread for @mitu 's script is here: https://retropie.org.uk/forum/topic/16319/gamelist-excel-export-script
- @meleu had a little bit of time to do a little more work on our
synopsis.txttogamelist.xmlscript that does WAY TOO MANY things to mention right here. This is a KEY element in my work, and meleu's awesome script is so powerful at this point and what it can do in about 12 hours right now by running it on my Pi Zero would literally take me thousands of hours to write by hand!
Latest post on the script is here, but you'd probably have to do a ton of work scrolling through to find everything that is done. (I should get a list together of what is implemented and the instructions on how to use everything and update everything on the first post..... It's on my to do list): https://retropie.org.uk/forum/topic/10406/making-premade-gamelist-xml-with-xtra-media-collections/409
He says that it shouldn't be any problem to reverse the
synopsis.txttogamelist.xmlthe other way around, so if he and mitu can both reverse their scripting work than that will open up a lot of new and awesome ways of editing and creating all of the information that can be pulled from thegamelist.xmlfiles and displayed on your rom lists screen.The
synopsis.txtfiles are individual text files with all of the game information without any tags that are used with great effect on the XBox and was an invention of a few great members of the old xbox-scene community and myself brainstorming ideas over a decade ago. If we can reverse the scripting processes that meleu and mitu have made, we can go fromsynopsis.txttogamelist.xmlto Spreadsheet, and then when I'm done doing all of my synopsis editing and adding more games and their information, back from Spreadsheet togamelist.xmltosynopsis.txtso we'll have perfect updated files that will work with both the Pi and the XBox.This is all very exciting and I can't wait to see what happens. :)
- The first is an AWESOME
-
Slowly trudging my way through the Japan carts now. Man... this is so boring..... lol. I can't wait till this part is done. Most of the work I do is extremely repetitive, but in many cases I don't find the work to be boring and it's easy for me to get into a groove and work for hours without even noticing the time going by. For whatever reason this really seems like doing chores to me and I can't get into it. I hope to knock a few hundred of them down in the next 24 hours. The closer I get to the finish line here the more likely the work will stop being so prohibitive for me.
Hope to have some good news soon. :)
-
@used2berx Finished all of the Japan and Translated carts except for some special carts like the Jaleco and Namcot carts which are a different size and/or have more elaborate carts that will require more work to edit properly. I think I'm going to go to the FDS carts next since I made them all from labels placed over a template. I'll I will have to do is make one good template to delete what needs to be removed and then that will be a quick 100 or so more images down.
Slowly but surely getting there. Probably just about 400 more images to go after FDS is done.
-
@used2berx Got some more done today and just did a recount of what still needs to be done. I've got 340 more carts, not including the FDS disks that should be really quick after I make the template since the disks are all from the same image.
I've been teaching myself a few tricks with layers to make some of this a lot smoother. When I'm done I will look at the finished images on the TV and any that I feel could be redone with these new skills I will improve.
Most of the images I have left are random images that weren't photographed a standard way like a lot of the Official releases were, so they will take a while to go through. I was actually individually editing most of the original images at first because I was still learning things. Had I known then what I know now, i would have saved dozens of hours of editing. At least I know it whenever I'm doing this part for the next system I'm going to work on.
Updates when I've got them... :)
-
I just completed all of the FDS updates this morning, as well as all of the "Adult Games", so the total carts that need to be done is down to about 320. :)
Note that these images were reduced in size by 50% because they were too large to upload here at around 1.8MB each!
The only categories I have left to work on now are Unlicensed, Prototypes, Pirates, Original Homebrew and a few stragglers from Translations and US Unlicensed games.
-
Nice... I really hit my stride with these and since the number is getting so low and it's not so overwhelming I'm back to where I can dive into it for a few hours at a time without driving myself crazy. :)
Only 242 more carts to go. Man I can't wait to be able to play around with the mass size convert and see how this stuff looks.
-
Here's some Original Homebrew carts that I had to make from scratch. They're reduced in size by 50% as well to upload here.
-
Ugggggg.....
I just discovered that after my latest reboot that the online aspect ratio calculator for some reason checked off "round to the nearest whole number, so I'm not sure how long I've been making the images with the wrong dimensions. I think when I'm all finished I'm going to have to go back and re-check all of them now in case this didn't happen only in the last few days. Bummer :(
Well... that took the wind out of my sails tonight. Going to be taking a break from this for a while. I've got about 200 more images left besides all of this re-checking I have to do now.
-
Went back and fixed all of them by date after figuring out where it went wrong. I'm still going to check all of them when they're done after I make some tables for ratios so I don't have to rely on calculators anymore for future systems and when I work on the boxarts for NES/FDS.
Just did a recount of carts I still have to do and I'm down to 203 left. 90% done after almost 2 months of my free time working on this. Can't wait. :)
-
181 cart images left to do. All that's left is Unlicensed and Pirate carts.
Almost there. :)
Man... I hope that Boxart will be so much more easy to do for the ratio work like I think it will.
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.