Backup SD Card by copying out files?

pjft

Click your picture on the top right > Settings >

Uncheck:
- Email: Send an email if a new chat message arrives and I am not online
- Email: Send an email when replies are made to topics I am subscribed to
- Watch: Watch topics you create
- Watch: Watch topics that you reply to
Subscribe to Digest: Off

Hope this works.

StormJH1

I didn't know corrupting SD cards during the backup process was a thing. Yikes, one more thing to worry about.

This is an interesting alternative way to do backups - one thing it might do is provide another workaround for the problem where SD card images grow to the size of the card being cloned, and won't necessarily fit on other cards of the same "size" (particularly different brands).

If this works as advertised, then I should be able to just do a fresh RetroPie install on any new SD card (even a smaller one, theoretically), and then transfer the date over using SFTP. So long as the data itself doesn't exceed the size of the new SD card, it should work.

A few questions though:

(1) What is the rough size of all those files? Is it just the sum of all its parts (ROMs, cover art, config information), or do the files themselves also "grow" to the size of the available storage?

(2) Does this retain ALL the setup information? Will my controllers just automatically work like they did before?

(3) In @mediamogul 's list of folders to copy, he used an external thumb drive for ROMS. If I have my RetroPie folder just on the microSD card (like I assume the majority of us do), can I backup and re-transfer that just the same? IIRC, the "RetroPie" folder contains roms, gamelist, boxart files, and metadata, so shouldn't that all still work in the new transfer?

I kinda want to try this just to see if I can make a functional backup. I have a 32GB off-brand microSD card. I couldn't do straight image copy from my SanDisk 32GB card because the SanDisk had slightly larger capacity. The "fresh RetroPie install followed by copying backed up folders" method might work!

TMNTturtlguy

@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.

The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?

mediamogul

@TMNTturtlguy

how do you go about transferring your backup files from the script back onto the SD card?

I just do it manually. I thought about writing a script to restore the files, but I decided it wasn't a good idea for the long term, as file locations in RetroPie are known to change from time to time. The whole thing takes me about 20 minutes, so it's not that bad.

TMNTturtlguy

@mediamogul how do you move the write protected files like the boot.cfg? Do you use putty and use a sudo command or do you simply put the card in your computer and move files?

mediamogul

@TMNTturtlguy

You can copy it over with sudo cp -f, but I always like to take a look at those two files and compare them first to see if anything has been added or removed since I last did a fresh install. It's probably not necessary, but I always like to err on the side of caution.

StormJH1

@TMNTturtlguy said in Backup SD Card by copying out files?:

@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.

The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?

That's unfortunate, sorry that happened to you. I've been using that win32diskimager thing to copy the backups to my Windows computer, so hopefully just reading the card to create an image on a computer wouldn't corrupt the source card (or is less likely to). 32GB card, creates an image that is 29.7GB in size - takes about 28 minutes on my computer.

I don't mind doing this process at all, though I haven't gotten either a different 32GB card or a 64GB to actually get it to work.

pjft

@TMNTturtlguy a tip from experience: if you plug the card into the computer via a SD card adaptor, flip the "read only" physical switch on the adaptor before the backup. I only make it readable if I'm restoring an image, otherwise the adaptor is always in read only mode. This way there are no weird file system operations from Windows/Mac OS.

TMNTturtlguy

@pjft Yeah, i was suspecting that is what is corrupting my card. I think I linked it to dropbox actually. I work remotely and our company uses dropbox to share files. I had dropbox open in the background and when i inserted the card it asked if i wanted to import the files to dropbox. I think that probably corrupted the card. Question though, if the switch is flipped to read only, how do you make any modifications to the card? I am guessing you are suggesting that the card only be inserted for writing a backup image and it is possible to make an image from a locked card?

pjft

@TMNTturtlguy well, yes and no. I never do any modifications to the card from anywhere but on the Pi. I don't have much there to change myself anyway - the odd theme or setting, but goes nicely via SSH. Also, Mac OS didn't have native support for ext2 file systems so I can only see the first partition, which is kind of useless anyway.

Otherwise all my ROMs, videos, etc are in a USB drive. In fact, I have a transfer folder on my USB drive that I use if I want to copy something to the Pi (and then on the Pi I copy it to the right place).

But yeah, no changing anything in the card from your computer. First rule of RetroPie Club. :)

TMNTturtlguy

@pjft That is what i thought! I have never changed anything from my computer, I use WinSCP or Putty to do everything, however I was clarifying as this thread has adde @StormJH1 who is also looking at these alternative ways to move files and wanted to be clear with him on transferring files.

My backup to my USB card is completed, but i am actually doing real work right now! So I won't be able to play with it more until later. I might have to invest several hours into redoing some of my gamelists. About half are in the roms folder, the other half live in .emulationstation. Does .emulationstation move to the USB? As I learn more I have started to do things best practice, just have not had the chance to clean up all of the gamelist and locations because it takes so much time!

mediamogul

@pjft

First rule of RetroPie Club.

Speaking of the club, we really need to rethink the whole "not talking about RetroPie club" rule, as it seems to be negatively affecting our enrollment.

pjft

@mediamogul I concede that that was a bit of a dumb move for the RetroPie club, in retrospect, together with the "no shirts, no shoes" thing.

pjft

@TMNTturtlguy I wouldn't have moved .emulationstation to the USB, but you can certainly move over the gamelists only.

I suspect that, if they're present on the actual roms/<system> folders, they override the .emulationstation ones. Just make sure to replace all paths by the correct ones (i.e. images pointing to .emulationstation rather than to the RetroPie/roms/<something> folder).

TMNTturtlguy

@pjft yeah, that is what i though, i just have to move the ones from .emulationstation into the roms folder and go through a few hundred roms and change the paths....on all the images and videos...talk about a fun evening!

pjft

@TMNTturtlguy I mean, other than the number of files, it shouldn't be a lot more complex than:

1 - Open file
2 - Find and replace <reference to /home/pi/.emulationstation folder or equivalent> with /home/pi/RetroPie/roms or equivalent, depending on the structure.
3 - Save.
4 - Move file.

That's how I did it when I had to go through the same thing.

mediamogul

@TMNTturtlguy

I might have to invest several hours into redoing some of my gamelists.

What happened to your gamelists?

Edit: Ah, I see it now.

TMNTturtlguy

@pjft Light Bulb! DUH, yup, do that for the path for the images and videos, save and close! I just did this when I created my "hacks" systems, I am using the same gamelist, just copied pasted, find all and replaced with the new 'systemname'hacks path! Thanks for reminding me how easy things can be when you slow down for a second and think thing through.

TMNTturtlguy

@pjft I successfully backed up my roms to the USB drive and that works great. Deleted all but one rom and bios form each system from the 128gb SD Card and put the USB back into the pi and it was perfect. Reads the games off of the usb stick.

Then i did the rpi-clone. Ran that and put the 16gb card into the pi, started right up with one game in each system. Turned the pi off, put the USB in and turned on the 16gb card and still only 1 game. The new image on the 16gb card is not reading the USB drive. The original 128gb card is. Any thoughts? I was thinking that the raspi-clone was an identical clone, but it is not operating that way?

Edit: When i go into WinSCP and look at the media folder is says that the usb location does not exist.

pjft

@TMNTturtlguy It certainly should. Also, for reference, I was successful in using an 8GB card as well, if you want to go as small as possible.

Would I be correct in assuming that you actually installed rpi-clone from

https://github.com/billw2/rpi-clone

?

The instructions I have in my own cheatsheet are:

sudo ./rpi-clone sda -f -v

where "sda" is the device of the destination disk (i.e. may be sdb, for instance).

Also, MAKE SURE TO BOOT THIS WITHOUT THE ROMS USB DRIVE PLUGGED IN, OTHERWISE THIS WILL NOT WORK AS INTENDED.

I wrote them for myself. You can find out which logical device the destination card is via dfand seeing which sd<something> matches the external SD card via USB.

What happens right now if you unplug and re-plug the USB drive, though? You should at least be able to see it mounting as an external drive, even if it doesn't mount as the ROM destination folder.

I'm not fully familiar with the automated mount thingie, but shouldn't be rocket science. What I expect to happen would either be:

Things work as expected;
USB drive is recognized, but isn't mounting in the right place for some reason.

If none are happening, that should be a different thing. Definitely unplug it and plug it to see if it's properly recognized.

You can list all devices via dfas mentioned.

Let us know how it goes.