NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?)

lilbud

@used2berx If you look on the frontpage of www.thecoverproject.net, someone redid all the nes games box art.

Used2BeRX

@lilbud I'll take a look at them, but they seem smaller than the ones that I have. Most of my official ones are anywhere from 1,200 to 1,500 pixels tall, and there are over 2,000 of them overall, not just the 750 or so official US releases.

I didn't have to fix any of these large images, since the place I got them from already had them fully restored at this size. I did restore hundreds of Japanese and China pirate covers though, and I will be replacing many of them with covers that @darknior made since he's an actual graphic artist and I'm just a weekend warrior when it comes to that stuff.

There may be a few in there that could replace some that I've got though. Thanks.

chipsnblip

great work on this project so far, been keeping an eye on it for a while. your dedication is inpiring @Used2BeRX!

@lilbud I'll take a look at them, but they seem smaller than the ones that I have. Most of my official ones are anywhere from 1,200 to 1,500 pixels tall, and there are over 2,000 of them overall, not just the 750 or so official US releases.

the full images are much larger, for example Mach Rider is 3431x2100 weighing in @ 5.2 MB. i had to enable javascript to get to the mediafire download though (ugh)

Used2BeRX

@chipsnblip said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):

great work on this project so far, been keeping an eye on it for a while. your dedication is inpiring @Used2BeRX!

Thanks man. :)

I've been at work on the Pi since around May of last year when my brother gave me one of the Pi Zero's he bought to make for two of his in-laws. It was supposed to be a quick job just getting the games to actually work so he could put them in NES carts and give them as gifts before the end of summer for the kids to play. Then it was supposed to be Christmas presents. I've promised him that they'll at least get it by this summer time. lol.... I really hope I can deliver on that.

By around August, I started working on NES exclusively. This is upgrading a set that I worked on with many, many other people with for the XBox mod community around 10-12 years ago, but I felt we had spread ourselves far too thin tackling too many systems at once, and in the end it was hard to keep everyone on the same page, so quality varied in nearly every aspect. It was just such a monstrous challenge. I felt we did a really great job at the end, and everyone I had given an XBox to afterward was amazed by it, but I always felt it could have been better.

So now I'm just taking it as slow as it need to be and doing it on my own to ensure everything is done in a uniform fashion. As I've gone along I've even decided to do things that have never occurred to me in the first place. Making these transparent cartridge images is an example of that. Had I not needed to make them PNG images to make the background transparent in order to preserve their aspect ratio for the inevitable resizing, I wouldn't have decided to go the extra mile and really make them shine by making them have completely transparent backgrounds. They really look amazing when you're scrolling through hundreds of them. I'm hoping I can convince the guys running retro-pie to add more image tags such as <cart>, <boxfront>, <title> and <action> for anybody who wanted to really take advantage of my work when it's complete.

@lilbud I'll take a look at them, but they seem smaller than the ones that I have. Most of my official ones are anywhere from 1,200 to 1,500 pixels tall, and there are over 2,000 of them overall, not just the 750 or so official US releases.

the full images are much larger, for example Mach Rider is 3431x2100 weighing in @ 5.2 MB. i had to enable javascript to get to the mediafire download though (ugh)

Yes, I see. I misread when I glanced at the page the first time and saw the 600dpi request and for some reason thought the images were 600 pixels. (Forgive me for that guys, as well as my lack of replying until now... I've been deathly ill for about a week. I even missed a few days of work and had my first 3 days off in about 2 months and spent them sleeping... :( )

The funny thing is, the place that I got my cover images must have gotten them from The Cover Project, and then just haphazardly cut the box front out of them. All of them were 2,100 pixels tall, but they ranged from 1,524 to 1,548 for width. I always found that pretty odd that the height was the same for 99% of them, but the width always varied. Now I can get the original images and re-crop them myself for a uniform size rather than further cutting down the larger ones to match the ones that were cropped too much.

Now I also know who to talk to for their blessing on using these images. I knew the site I had gotten them from didn't actually get them directly from the reproduction (guy, girl, team?) that made them all.

Although I will be looking into crowd funding for future work (and hence, not offering any roms at any time in these collections), I'm hoping that anybody who has provided any of the building blocks for the final creations I make will give their blessing in return for the much larger scope of what they will be getting back in the other aspects of the project. This will be the definitive NES collection when it is done, and nothing that exists will come close to it. I will not be holding any of the work for ransom, and I do hope that once a set is released that the big sites out there all end up using what's in it.

I haven't decided exactly how I'm going to go about this all just yet, but what I'm considering is getting any patreon/kickstarter stuff in place, making any low production crap video I might need to make since I have ancient tech and no film school knowledge, and then just releasing my entire NES work which will at that point have probably taken over a year of my free time to complete as a proof of concept of a much larger goal and letting it speak for itself at that point.

I'm thinking that legally speaking I should be shielded since I won't be providing any actual roms... and especially since I will jump right out of the gates giving the whole damn thing away for free in the first place. It's all stuff that many of our beloved gaming community sites already have on display and have had, sometimes for almost decades now... albiet in much higher and consistent quality. I look at it as I'm not selling a product, or really anything that somebody couldn't do themselves if they were insane enough to do it all on their own. I'm really just asking for people who have better stuff to do with their lives to show their appreciation for the insane amount of time and attention to detail that I give these projects for everyone else to enjoy.

We'll see what the future holds. I really do love working on this. I don't know how I can possibly keep working on it like I do though given my financial situation. It's already slowed down to a snail's pace since I had to finally start working again, so my free time all of the sudden has some real value to me. (Which really burns me that I've had to spend the last 4 or 5 days in bed now!!!!!)

Thanks for your support. :)

incunabula

Maybe i've missed it somewhere in this thread, but are you making this collection available for download somewhere?

Used2BeRX

@incunabula

Short answer: Yes, but I don't know where yet.

Long Answer follows....

Eventually I will be, when it is completed. Everything will be included except for the roms, of course.

I have yet to decide if I will be releasing this before or after I re-make all of the videos in HD. If you've missed my thoughts on the videos before is that although any sets out there are adaquite, there is a lot of differences in quality, length, volume and style of the video sets that are out there because too many hands were in the cookie jar. My plan is to re-make every video as well as make the first known videos for many hacks and translations, pirates, unlicensed, etc. They would all be in a standard format, and my plan would be to do them in an arcade-style "attract mode" where you get around 15-20 seconds of gameplay followed by the title-screen animation.

Obviously, this last part will be a massive undertaking on its own, and at this point in time I'm not even sure I have any tech in my house even capable of performing such a feat. Chances are more likely the videos would be a part II release for the NES at a later date.

My plan a this moment is to make 3 separate releases. One for the XBox, one for the Pi Zero and one for the Pi 3. The XBox release is pretty straightforward, but due to the much more complex nature of the RetroPie system and the necessity for the gamelist.xml to use any of the extra media, it will rely on all of the media to be in the exact location that I set on a per-system basis or else it will not work with the gamelist.xml files that I provide, that will be made using the script that the most awesome Meleu from this site has created for us.

For this reason, although I cannot and will not provide any roms at any point in time, I will be providing datfiles and a way to easily ensure that the roms you find on your own are all in the correct folders in as simple a process as possible. This will all take a lot of planning. Xbox makes this all easy. RetroPie makes this extremely tricky.

The Pi Zero set will not ever include videos since it can't handle them anyhow. It will have the lowest quality artwork. It will be the highest quality I can get away with without negatively impacting system performance which has yet to be determined.

The XBox will include videos, but they will need to be converted by the end user to XMV. I cannot and will not provide the XMV files because they use a proprietary M$ software. The videos will not play in any other format other than XMV which is unfortunate because not only does this at an extra step of converting, but it also requires lowering the video quality more than necessary for the specs of the machine. This will have larger artwork than the Pi Zero. I believe the specs that are considered optimal have your larger dimension somewhere between 550 to 600 pixels.

The Pi 3 will have the highest quality videos. Most likely in MP4, but I will be asking a lot of questions from people who know more about these things than I do before settling on any format. Artwork will be the highest quality of all 3 sets, although it still will not come close to the actual dimensions of much of the raw artwork. I've been told by people who know much more about these things that the ideal large dimension for artwork on the Pi 3 is only around 700 pixels, which is anywhere from 1/2 to only 1/3 of the size of many of the images I currently have. I've been told that they look amazing even at that size on a large 1080 screen though, so I wouldn't worry too much about it. I'm keeping the larger images in case down the road we all have 150" 4k TVs and the Pi 5 can handle them without downsizing them.

In the mean time, I'm not too worried about it myself. I'm doing all my work on a Pi Zero, with a rear-projection 720p TV and a 10 year old computer with 2gigs of ram that sometimes makes me think that it isn't really even up to the task of running Windows 10, what with the frequent Firefox restarts I constantly need to do to free up my memory.

incunabula

Sounds good, i just hope you have a backup solution in your workflow.

markwkidd

Please consider synchronizing your collection to a github repository or something like that which would eventually also make it easy to distribute & incorporate into setup scripts. Even if you keep it private while it's a work in progress it might pay off in the long term.

If that's something you'd consider, I'm sure there are several of us here who would help you create a githup repository. I have done a large MAME thumbnail for RetroArch and @herb_fargus has created a similarly large github thumbnail repo for RetroPie.

Your project and your focus are impressive! Whatever you use for backup, thanks for sharing!

Used2BeRX

@markwkidd

Hey man. I would be thrilled if github allows for the massive amount of storage we're talking about here and somebody would work with me to get it up there. It was suggested by @meleu quite a while back and I signed up for it but couldn't figure out at all what to do with it and he's too busy to hold my hand. Consider me interested if you or somebody else would be willing to aid in this. I have way too much work on my end here to be spending tons of time trying to figure github out on my own. :)

Since things aren't at a ready state just yet, most of it would not really benefit from it, but something small like the synopsis would be easy to re-up when it gets the final edits. Once I start making the full artwork size mass conversions they would start being ready as well. I doubt though that I'd ever be able to store videos there, would I?

Anyhow.... the note above is actually just a small part of the overall project. It's more just what I'm working on now. I believe a more full picture of the scope of things is in the first 10 or so posts of this thread. I might just create a new thread in the near future when I'm ready to put the spreadsheet up that really illustrates what is already done and what still needs to be done.

Things are backed up right now, but it's always better to have an extra one. :)

markwkidd

@used2berx what operating system are you using on the system where you work on this project? Do you have any idea how large the collection might be when it's complete?

I'll try to summarize a github workflow based on your OS/environment.

The second place I would suggest keeping in mind is the Internet Archive at archive.org. I am confident that they would host this collection. Once you have a user account there you can upload through their website, plus there are other methods available to add to the archive if the content is too large to upload through the web.

I'm glad to keep talking with you about this. After all the work to standardize the images themselves, hopefully it won't be too hard to find one or two solid, long-term hosts for your results.

Used2BeRX

@markwkidd I use Windows 10.

I couldn't even begin to imagine how big it will be at this point. All I can say is it will be a lot smaller before the videos are made.

The synopsis files are actually very small. Right now, between the games and the folders there are right around 2,050 of them, and they're all 1kb or less.

Images will be all over the place since I'll have 3 sets of them with different sizes for the Boxes and Carts. The Title and Action shots will be the same for all 3 sets and will be small though. Right now, because there are 4 image types for each set at the moment, this will be about 8,080 images per set.

There actually is a lot of other aspects to the project as well, but currently RetroPie is unable to take advantage of them like the XBox does. (such as manuals in either PDF format or converted to JPG and zipped up).

My "plan" right now was to upload them all together somewhere, where they would need to be then downloaded and transferred over to the correct place on your Pi/XBox. That is the way it must be done regardless on the XBox, but maybe with github people could just download the thing straight to the proper location on their Pi if it were set up correctly?

EDIT: When I say "right now" about the amount of images, it's because I do plan on adding more games before this is all done. I'm sure before I'm done there will be more translations since they seem to pop up every once and a while, and I still haven't really gone in and decided which hacks will make the cut. There were already a lot of them in there from ten years ago, and I only removed a few that were in some way broken. A lot of them probably wouldn't have made the cut now, but since all the work has been done for them at this point except for the videos I'm going to keep the rest. I'm sure that there are still some really great ones out there that I haven't discovered though.

markwkidd

OK that makes sense. If you're willing to spend a little time setting up git and github (maybe 30 minutes to an hour if things go smoothly?) I think it would fit into your way of working.

With git, we would designate a root folder on you PC where all of the collection lies within as subfolders. This would sync onto github as the root of your project's github repository, where among other things, it would be easy for people to script it into automated build projects like you were describing.

As you continue to work on your project, you can periodically synchronize to github on whatever schedule you want. The number of files you're proposing (8,080 in this phase) is within the specs of a standard github repository.

Next time you have a chance, make sure you can access your github.com account, and then see if you can create a new "repository" for your project. Once you have access to github.com I can try to help you figure out the best way to install the git client on your Windows 10 PC and designate that root folder so that it doesn't alter your organizations scheme or workflow if at all possible.

markwkidd

Here's the libretro thumbnails repository, in fact. This would be a good reference point: https://github.com/libretro/libretro-thumbnails

Used2BeRX

@markwkidd said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):

Here's the libretro thumbnails repository, in fact. This would be a good reference point: https://github.com/libretro/libretro-thumbnails

That's pretty much exactly what I would be looking to do, assuming you could then download them directly to where they needed to go on the Pi. Is that what somebody could do now by running a command from the Pi console?

Actually, once meleu's script is complete, I could also set it up along with the proper shortcuts so that any time I did update things in the future, people could just re-run the script and any changes that were made would be reflected the next time they rebooted. :)

I should have some time this week to look into this. I'll make sure I can still get into my github and we'll take it from there. Thanks a lot for this. It will give a great option for ensuring that anybody trying to use my set(s) will be working with the exact setup I have without worrying they did it wrong.

BTW... how does it handle folders? I would have empty folders for all of the sub-categories for the roms. Would I have to have an empty text file in these folders for them to actually copy over?

I'm hoping to find somebody who can make a script for me at some point that would automatically move all of the roms they found to the right folders after they run them against my datfile and they're renamed. There are still a lot of more obscure games that people would have to hunt down and/or patch on their own, but once you see my spreadsheet you'll see that I've heavily detailed all of this and that a large portion of them can be found through full GoodNES/No-Intro/TOSEC sets.

Maybe I can make datfiles with sub-folders in them? I know I can make one for each sub-folder, but beyond that I kind of get lost.

markwkidd

@used2berx said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):

That's pretty much exactly what I would be looking to do, assuming you could then download them directly to where they needed to go on the Pi. Is that what somebody could do now by running a command from the Pi console?

Yes. Right now you could, for example, type this single git command into a console on a pi or PC with git installed and it will download the whole libretro thumbnail collection: git clone http://github.com/libretro/libretro-thumbnails.git

Slightly more sophisticated commands could sync only parts of the repository.

BTW... how does it handle folders? I would have empty folders for all of the sub-categories for the roms. Would I have to have an empty text file in these folders for them to actually copy over?

I am not positive, but I do think you might have to put a text file in order to get an empty folder to sync to a github repository. Alternately if this is going to be installed by a script, you might consider letting the script create the folders on the local machine when the script is executed.

As I said, glad to give a hand if my experience could be useful to you. I'll keep an eye out for your next updates.

Used2BeRX

@markwkidd Awesome man. I will definitely try to make this a part of my work. This seems like the ideal solution, at least for everything besides the videos. This way if I have some minor updates here or there I don't have to re-up the whole thing somewhere else.

BTW... is there a way for people to just be able to update anything after they last downloaded? For instance, if I add a few games (not roms, but all the xtra media), or if I update some artwork or text files/gamelist.xml stuff would they be able to just grab the changes since the last update?

You're right about the script. No sense in putting empty folders here. I'm hoping that I can build a datfile with sub-directories and the script won't even be necessary. I'm a while from that point though.

Thanks so much for the help. I have been pretty sick recently, so the limited amount of time I've felt like working on the project while I wasn't at work I dedicated to the images, so I haven't had any time to look into this yet. I hope you're not going anywhere in a hurry though and I can get some assistance if I need it when I start putting stuff on github.

I should have some updates soon, but this cart project really is taking a lot longer than I thought it would. It looks really sweet though. I don't want to rush it and do a sloppy job.

Later

markwkidd

@used2berx said in NES RessurctionXtras 2.0 Project (thread previously titled: Does anybody work on Emulators to fix broken games here?):

BTW... is there a way for people to just be able to update anything after they last downloaded? For instance, if I add a few games (not roms, but all the xtra media), or if I update some artwork or text files/gamelist.xml stuff would they be able to just grab the changes since the last update?

Yep, github makes incremental uploads and incremental downloads a breeze.

Thanks so much for the help. I have been pretty sick recently, so the limited amount of time I've felt like working on the project while I wasn't at work I dedicated to the images, so I haven't had any time to look into this yet. I hope you're not going anywhere in a hurry though and I can get some assistance if I need it when I start putting stuff on github.

Take your time, this is an impressive undertaking! I hope your health improves.

Used2BeRX

@markwkidd Thanks man. I'll let you know when I start messing around with github. I think I'm going to finish the carts first, and then I wanted to go through the synopsis files (that make the gamelist.xml with meleu's script) and do some editing with the genre tags. I think right now there's something insane like 200 different types of genre's and most of that is just due to the order of the game types not being standard. Once I get that done, I'll probably start with the synopsis files on github since they're small and it will be real quick to upload and change things.

Quick update on the cart status.

Finished categories: US Licensed, Europe Licensed, VS Arcade games, and all game hacks.

Almost finished: Japan Licensed (untranslated), Adult Games, Zapper Games, Unsupported Controller games.

To Do: All FDS disks, Translated Games, Unlicensed Games, Prototypes, Pirates and Original Homebrew.

It's hard to count them right now since I've hidden some stuff on the spreadsheet to make this work easier so I don't have specific counts by type, but if I had to guess I've probably individually edited around 1,200 to 1,300 of the 2,019 current titles. Some of the more obscure titles are going to be tricky because the cart image sizes for them vary pretty wildly, so it's going to be a lot less mindless work than most of what I've already done.

In the mean time, we have 3 new translated games that came out recently that will be added as well :)

Used2BeRX

@markwkidd and anybody else following this thread :)

There has been a few developments on the side that are exciting for this project.

The first is an AWESOME gamelist.xmlto Spreadsheet script that was written and shared to us by @mitu that was just pointed out to me this morning. I'm super stoked on this and I really hope that he sees my request and can find the time to reverse this and get it so we can then convert the Spreadsheet back to gamelist.xml This will be excellent news for any future synopsis editing and save loads of time. Working on the synopsis files is one of the most tedious aspects of the process, and being able to do the work for an entire system on a spreadsheet would be wonderful.

Thread for @mitu 's script is here: https://retropie.org.uk/forum/topic/16319/gamelist-excel-export-script

@meleu had a little bit of time to do a little more work on our synopsis.txt to gamelist.xml script that does WAY TOO MANY things to mention right here. This is a KEY element in my work, and meleu's awesome script is so powerful at this point and what it can do in about 12 hours right now by running it on my Pi Zero would literally take me thousands of hours to write by hand!

Latest post on the script is here, but you'd probably have to do a ton of work scrolling through to find everything that is done. (I should get a list together of what is implemented and the instructions on how to use everything and update everything on the first post..... It's on my to do list): https://retropie.org.uk/forum/topic/10406/making-premade-gamelist-xml-with-xtra-media-collections/409

He says that it shouldn't be any problem to reverse the synopsis.txt to gamelist.xml the other way around, so if he and mitu can both reverse their scripting work than that will open up a lot of new and awesome ways of editing and creating all of the information that can be pulled from the gamelist.xml files and displayed on your rom lists screen.

The synopsis.txt files are individual text files with all of the game information without any tags that are used with great effect on the XBox and was an invention of a few great members of the old xbox-scene community and myself brainstorming ideas over a decade ago. If we can reverse the scripting processes that meleu and mitu have made, we can go from synopsis.txt to gamelist.xml to Spreadsheet, and then when I'm done doing all of my synopsis editing and adding more games and their information, back from Spreadsheet to gamelist.xml to synopsis.txt so we'll have perfect updated files that will work with both the Pi and the XBox.

This is all very exciting and I can't wait to see what happens. :)

Used2BeRX

Slowly trudging my way through the Japan carts now. Man... this is so boring..... lol. I can't wait till this part is done. Most of the work I do is extremely repetitive, but in many cases I don't find the work to be boring and it's easy for me to get into a groove and work for hours without even noticing the time going by. For whatever reason this really seems like doing chores to me and I can't get into it. I hope to knock a few hundred of them down in the next 24 hours. The closer I get to the finish line here the more likely the work will stop being so prohibitive for me.

Hope to have some good news soon. :)