Backup SD Card by copying out files?
-
@pjft Thanks for the response. I like the plan of copying off the roms and then deleting them and transferring the data onto a smaller SD card with the rpi-clone tool. Does the clone tool make the smaller sd card bootable as well? I did find a version on github, not sure if it is the same tool you are speaking of. If you have time, any directions or link would be appreciated. If not, I will be resourceful and dig into some reading! I really appreciate all the help and responses, i think this will really improve my setups.
I am going to run to the local computer store and buy a few SD and USB sticks. Do you think an 8gb sd is the right size? All of the games will be on the SD card, i suppose scraped art that is in home/pi/.emulationstation will remain on the SD Card? Edit: looks like a 16gb class 10 card is only $1.00 more for me, so i might as well just get those.
As for #2 - after reading the manual instructions, the manual instructions say not to use the automatic transfer, so i assume that once the transfer is complete I will want to turn that service off so it doesn't try to transfer every time.
-
I must have missed your last post.
(1) Yes, all of that would work.
(2) Yes, you would leave it on. I believe it triggers an action that would set everything back when deactivated.
(3) Yes again. At that point you could just plug the external drive into your computer to transfer ROMs.
Edit:
so i assume that once the transfer is complete I will want to turn that service off so it doesn't try to transfer every time.
It doesn't transfer every time.
-
@mediamogul Thanks! Off to pick up some parts and give this try!
-
Good luck. It sounds like either method will get you where you want to be.
-
@mediamogul @pjft this could take awhile! I have about 90 gigs to transfer, it has been running for 10 minutes and we are up to 1gb transferred! I am doing this for 2 builds, do you know if i can simply copy the files from one usb stick to another over my PC? or does the script do something to format the usb stick when it is first inserted? I noted that when looking at the retropie manager that the mount is named /media/usb0 - that is making me wonder if I have to do this process twice instead of a simply copy?
Also, I noticed that the link for formatting USB drives for Windows does not provide proper information for formatting USB drives larger than 32GB. Windows will not allow you to format drives larger than 32GB as fat32. Luckily the drives I purchased where already formatted fat32, but the docs might want to get updated? Thanks again
-
do you know if i can simply copy the files from one usb stick to another over my PC? or does the script do something to format the usb stick when it is first inserted?
That should work. Just make sure you rename the copied 'RetroPie' folder to 'retropie-mount' in the root of the external drive and it should see that it's already been copied and act accordingly.
-
@mediamogul @pfjt - Still running! I am at 76 gb transferred. Once the roms/folders are transferred to the USB stick and i want to delete them from the SD card, what is the best process to do so? Can I just go into WinSCP and select the rom folder and delete all the folders within? or does that file structure need to remain on the SD card and I then go into each system folder and delete the contents from within each one? Also, can retropie run without the USB plugged in now that the entire folder has been moved, or does the USB always need to be plugged in when running? I want to make sure i don't delete any ROM files since it has taken so long to transfer them!
-
@TMNTturtlguy Ok, so a few things:
- My recommendation is to leave part of the folder structure in the SD card, but with no ROMs there. Perhaps just one of those folders with placeholder shortcuts (like Amiga's "Launch UAE" or something) would work. Or a single ROM in a single folder. The reason is that, if/when you need to boot RetroPie without the USB there, you still want it to be able to launch ES properly, and I don't know what would happen if you had no systems/ROMs there.
- As I was saying, as long as the base folder structure is there, it should boot fine. I imagine you'll be mounting the USB on top of the RetroPie folder, so both BIOS and roms folders would benefit form existing in the SD card, even if empty.
In my case, that's what I do when I want to run rpi-clone. I boot it without the USB mounted, and then run the script. It'll copy whatever's needed to the new SD card.
Depending on the script, though, if you do have the USB plugged in, and if it's mounted in ~/RetroPie, it may start copying over the ROMs to the backup (as it doesn't know that that folder is actually now residing on a USB stick). Just a note, from personal experience :)
Other than that, you should hopefully be good to go.
-
@herb_fargus
Bonjour je. NE sais pas qui contacté je voudrais me désinscrire désabonner du forum le plutot possible je recois plus de 100 mail de retropie par jour. Cordialement -
Click your picture on the top right > Settings >
- Uncheck:
- Email: Send an email if a new chat message arrives and I am not online
- Email: Send an email when replies are made to topics I am subscribed to
- Watch: Watch topics you create
- Watch: Watch topics that you reply to
- Subscribe to Digest: Off
Hope this works.
- Uncheck:
-
I didn't know corrupting SD cards during the backup process was a thing. Yikes, one more thing to worry about.
This is an interesting alternative way to do backups - one thing it might do is provide another workaround for the problem where SD card images grow to the size of the card being cloned, and won't necessarily fit on other cards of the same "size" (particularly different brands).
If this works as advertised, then I should be able to just do a fresh RetroPie install on any new SD card (even a smaller one, theoretically), and then transfer the date over using SFTP. So long as the data itself doesn't exceed the size of the new SD card, it should work.
A few questions though:
(1) What is the rough size of all those files? Is it just the sum of all its parts (ROMs, cover art, config information), or do the files themselves also "grow" to the size of the available storage?
(2) Does this retain ALL the setup information? Will my controllers just automatically work like they did before?
(3) In @mediamogul 's list of folders to copy, he used an external thumb drive for ROMS. If I have my RetroPie folder just on the microSD card (like I assume the majority of us do), can I backup and re-transfer that just the same? IIRC, the "RetroPie" folder contains roms, gamelist, boxart files, and metadata, so shouldn't that all still work in the new transfer?
I kinda want to try this just to see if I can make a functional backup. I have a 32GB off-brand microSD card. I couldn't do straight image copy from my SanDisk 32GB card because the SanDisk had slightly larger capacity. The "fresh RetroPie install followed by copying backed up folders" method might work!
-
@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.
The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?
-
how do you go about transferring your backup files from the script back onto the SD card?
I just do it manually. I thought about writing a script to restore the files, but I decided it wasn't a good idea for the long term, as file locations in RetroPie are known to change from time to time. The whole thing takes me about 20 minutes, so it's not that bad.
-
@mediamogul how do you move the write protected files like the boot.cfg? Do you use putty and use a sudo command or do you simply put the card in your computer and move files?
-
You can copy it over with
sudo cp -f, but I always like to take a look at those two files and compare them first to see if anything has been added or removed since I last did a fresh install. It's probably not necessary, but I always like to err on the side of caution. -
@TMNTturtlguy said in Backup SD Card by copying out files?:
@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.
The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?
That's unfortunate, sorry that happened to you. I've been using that win32diskimager thing to copy the backups to my Windows computer, so hopefully just reading the card to create an image on a computer wouldn't corrupt the source card (or is less likely to). 32GB card, creates an image that is 29.7GB in size - takes about 28 minutes on my computer.
I don't mind doing this process at all, though I haven't gotten either a different 32GB card or a 64GB to actually get it to work.
-
@TMNTturtlguy a tip from experience: if you plug the card into the computer via a SD card adaptor, flip the "read only" physical switch on the adaptor before the backup. I only make it readable if I'm restoring an image, otherwise the adaptor is always in read only mode. This way there are no weird file system operations from Windows/Mac OS.
-
@pjft Yeah, i was suspecting that is what is corrupting my card. I think I linked it to dropbox actually. I work remotely and our company uses dropbox to share files. I had dropbox open in the background and when i inserted the card it asked if i wanted to import the files to dropbox. I think that probably corrupted the card. Question though, if the switch is flipped to read only, how do you make any modifications to the card? I am guessing you are suggesting that the card only be inserted for writing a backup image and it is possible to make an image from a locked card?
-
@TMNTturtlguy well, yes and no. I never do any modifications to the card from anywhere but on the Pi. I don't have much there to change myself anyway - the odd theme or setting, but goes nicely via SSH. Also, Mac OS didn't have native support for ext2 file systems so I can only see the first partition, which is kind of useless anyway.
Otherwise all my ROMs, videos, etc are in a USB drive. In fact, I have a transfer folder on my USB drive that I use if I want to copy something to the Pi (and then on the Pi I copy it to the right place).
But yeah, no changing anything in the card from your computer. First rule of RetroPie Club. :)
-
@pjft That is what i thought! I have never changed anything from my computer, I use WinSCP or Putty to do everything, however I was clarifying as this thread has adde @StormJH1 who is also looking at these alternative ways to move files and wanted to be clear with him on transferring files.
My backup to my USB card is completed, but i am actually doing real work right now! So I won't be able to play with it more until later. I might have to invest several hours into redoing some of my gamelists. About half are in the roms folder, the other half live in .emulationstation. Does .emulationstation move to the USB? As I learn more I have started to do things best practice, just have not had the chance to clean up all of the gamelist and locations because it takes so much time!
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.