Backup SD Card by copying out files?
-
I didn't know corrupting SD cards during the backup process was a thing. Yikes, one more thing to worry about.
This is an interesting alternative way to do backups - one thing it might do is provide another workaround for the problem where SD card images grow to the size of the card being cloned, and won't necessarily fit on other cards of the same "size" (particularly different brands).
If this works as advertised, then I should be able to just do a fresh RetroPie install on any new SD card (even a smaller one, theoretically), and then transfer the date over using SFTP. So long as the data itself doesn't exceed the size of the new SD card, it should work.
A few questions though:
(1) What is the rough size of all those files? Is it just the sum of all its parts (ROMs, cover art, config information), or do the files themselves also "grow" to the size of the available storage?
(2) Does this retain ALL the setup information? Will my controllers just automatically work like they did before?
(3) In @mediamogul 's list of folders to copy, he used an external thumb drive for ROMS. If I have my RetroPie folder just on the microSD card (like I assume the majority of us do), can I backup and re-transfer that just the same? IIRC, the "RetroPie" folder contains roms, gamelist, boxart files, and metadata, so shouldn't that all still work in the new transfer?
I kinda want to try this just to see if I can make a functional backup. I have a 32GB off-brand microSD card. I couldn't do straight image copy from my SanDisk 32GB card because the SanDisk had slightly larger capacity. The "fresh RetroPie install followed by copying backed up folders" method might work!
-
@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.
The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?
-
how do you go about transferring your backup files from the script back onto the SD card?
I just do it manually. I thought about writing a script to restore the files, but I decided it wasn't a good idea for the long term, as file locations in RetroPie are known to change from time to time. The whole thing takes me about 20 minutes, so it's not that bad.
-
@mediamogul how do you move the write protected files like the boot.cfg? Do you use putty and use a sudo command or do you simply put the card in your computer and move files?
-
You can copy it over with
sudo cp -f, but I always like to take a look at those two files and compare them first to see if anything has been added or removed since I last did a fresh install. It's probably not necessary, but I always like to err on the side of caution. -
@TMNTturtlguy said in Backup SD Card by copying out files?:
@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.
The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?
That's unfortunate, sorry that happened to you. I've been using that win32diskimager thing to copy the backups to my Windows computer, so hopefully just reading the card to create an image on a computer wouldn't corrupt the source card (or is less likely to). 32GB card, creates an image that is 29.7GB in size - takes about 28 minutes on my computer.
I don't mind doing this process at all, though I haven't gotten either a different 32GB card or a 64GB to actually get it to work.
-
@TMNTturtlguy a tip from experience: if you plug the card into the computer via a SD card adaptor, flip the "read only" physical switch on the adaptor before the backup. I only make it readable if I'm restoring an image, otherwise the adaptor is always in read only mode. This way there are no weird file system operations from Windows/Mac OS.
-
@pjft Yeah, i was suspecting that is what is corrupting my card. I think I linked it to dropbox actually. I work remotely and our company uses dropbox to share files. I had dropbox open in the background and when i inserted the card it asked if i wanted to import the files to dropbox. I think that probably corrupted the card. Question though, if the switch is flipped to read only, how do you make any modifications to the card? I am guessing you are suggesting that the card only be inserted for writing a backup image and it is possible to make an image from a locked card?
-
@TMNTturtlguy well, yes and no. I never do any modifications to the card from anywhere but on the Pi. I don't have much there to change myself anyway - the odd theme or setting, but goes nicely via SSH. Also, Mac OS didn't have native support for ext2 file systems so I can only see the first partition, which is kind of useless anyway.
Otherwise all my ROMs, videos, etc are in a USB drive. In fact, I have a transfer folder on my USB drive that I use if I want to copy something to the Pi (and then on the Pi I copy it to the right place).
But yeah, no changing anything in the card from your computer. First rule of RetroPie Club. :)
-
@pjft That is what i thought! I have never changed anything from my computer, I use WinSCP or Putty to do everything, however I was clarifying as this thread has adde @StormJH1 who is also looking at these alternative ways to move files and wanted to be clear with him on transferring files.
My backup to my USB card is completed, but i am actually doing real work right now! So I won't be able to play with it more until later. I might have to invest several hours into redoing some of my gamelists. About half are in the roms folder, the other half live in .emulationstation. Does .emulationstation move to the USB? As I learn more I have started to do things best practice, just have not had the chance to clean up all of the gamelist and locations because it takes so much time!
-
First rule of RetroPie Club.
Speaking of the club, we really need to rethink the whole "not talking about RetroPie club" rule, as it seems to be negatively affecting our enrollment.
-
@mediamogul I concede that that was a bit of a dumb move for the RetroPie club, in retrospect, together with the "no shirts, no shoes" thing.
-
@TMNTturtlguy I wouldn't have moved .emulationstation to the USB, but you can certainly move over the gamelists only.
I suspect that, if they're present on the actual roms/<system> folders, they override the .emulationstation ones. Just make sure to replace all paths by the correct ones (i.e. images pointing to .emulationstation rather than to the RetroPie/roms/<something> folder).
-
@pjft yeah, that is what i though, i just have to move the ones from .emulationstation into the roms folder and go through a few hundred roms and change the paths....on all the images and videos...talk about a fun evening!
-
@TMNTturtlguy I mean, other than the number of files, it shouldn't be a lot more complex than:
1 - Open file
2 - Find and replace <reference to /home/pi/.emulationstation folder or equivalent> with/home/pi/RetroPie/romsor equivalent, depending on the structure.
3 - Save.
4 - Move file.That's how I did it when I had to go through the same thing.
-
I might have to invest several hours into redoing some of my gamelists.
What happened to your gamelists?
Edit: Ah, I see it now.
-
@pjft Light Bulb! DUH, yup, do that for the path for the images and videos, save and close! I just did this when I created my "hacks" systems, I am using the same gamelist, just copied pasted, find all and replaced with the new 'systemname'hacks path! Thanks for reminding me how easy things can be when you slow down for a second and think thing through.
-
@pjft I successfully backed up my roms to the USB drive and that works great. Deleted all but one rom and bios form each system from the 128gb SD Card and put the USB back into the pi and it was perfect. Reads the games off of the usb stick.
Then i did the rpi-clone. Ran that and put the 16gb card into the pi, started right up with one game in each system. Turned the pi off, put the USB in and turned on the 16gb card and still only 1 game. The new image on the 16gb card is not reading the USB drive. The original 128gb card is. Any thoughts? I was thinking that the raspi-clone was an identical clone, but it is not operating that way?
Edit: When i go into WinSCP and look at the media folder is says that the usb location does not exist.
-
@TMNTturtlguy It certainly should. Also, for reference, I was successful in using an 8GB card as well, if you want to go as small as possible.
Would I be correct in assuming that you actually installed rpi-clone from
https://github.com/billw2/rpi-clone?
The instructions I have in my own cheatsheet are:
sudo ./rpi-clone sda -f -v where "sda" is the device of the destination disk (i.e. may be sdb, for instance). Also, MAKE SURE TO BOOT THIS WITHOUT THE ROMS USB DRIVE PLUGGED IN, OTHERWISE THIS WILL NOT WORK AS INTENDED.I wrote them for myself. You can find out which logical device the destination card is via
dfand seeing which sd<something> matches the external SD card via USB.What happens right now if you unplug and re-plug the USB drive, though? You should at least be able to see it mounting as an external drive, even if it doesn't mount as the ROM destination folder.
I'm not fully familiar with the automated mount thingie, but shouldn't be rocket science. What I expect to happen would either be:
- Things work as expected;
- USB drive is recognized, but isn't mounting in the right place for some reason.
If none are happening, that should be a different thing. Definitely unplug it and plug it to see if it's properly recognized.
You can list all devices via
dfas mentioned.Let us know how it goes.
-
@pjft Yes, i installed from the location you have listed above. I also used the sudo code above exactly as stated and i did not have the USB Roms drive plugged in. I actually did this process twice with the same results both times.
If i unplug the roms usb and plug it back in, nothing happens, light on the usb blinks a few times, but it is not read. If i use df it does not show the Roms USB and it also does not show up in WinSCP. Here is a shot of the df:
also assuming you would help troubleshoot, i took these shots of the rpi-clone process.Here were my steps. Shutdown pi, take out Roms USB. Start up pi and quit emulationstation. Using putty i ran the rpi-clone. It asked if i wanted to unmount /media/usb0
At this point the only thing plugged into the pi is the usb card reader with the new smaller SD card in it. I chose no and it cancels the clone. I have to choose yes to proceed. Again, the Roms USB was not installed.
Next i get this screen, the first attempt i named the drive, the second attempt shown here, i did not name the drive, thinking that maybe the name of the drive was causing the error.
Ran this through, finished, no luck with the roms usb. Nothing even shows up in my WinSCP media, it is empty. I tried to go in an reenable the USB Automated mount, no luck. Tried to mount a new usb, wont do mount a new usb either. I hope this is enough info to trouble shoot? I have not yet slept and i have to work in 4 hours, so i am going to shut down for the night, but hope to continue this tomorrow. Thanks!
Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.
Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.