Backup SD Card by copying out files?

[StormJH1]

@TMNTturtlguy said in Backup SD Card by copying out files?:

@StormJH1 I am in the process right now, so i will post my outcomes shortly. I have had 2 SD card corruptions over the last 3 months, both times were during backups. I would get a completed .img and then put the card back into the pi and it would no longer startup. Put the card back into the computer and it would have a write protection that could not be removed every method of reformatting including DiskPart would not work.

The main thing you need to consider when transfering files by a program like WinSCP is that files like the boot.cfg file are write protected and cannot be replaced/copied over. I think you either have to do a sudo nano in a program like putty, or put the SD card into your computer and make the swap there. @mediamogul @pjft is this a correct assumption? @mediamogul how do you go about transferring your backup files from the script back onto the SD card? Do you have a script to rm and then cp them back on the card?

That's unfortunate, sorry that happened to you. I've been using that win32diskimager thing to copy the backups to my Windows computer, so hopefully just reading the card to create an image on a computer wouldn't corrupt the source card (or is less likely to). 32GB card, creates an image that is 29.7GB in size - takes about 28 minutes on my computer.

I don't mind doing this process at all, though I haven't gotten either a different 32GB card or a 64GB to actually get it to work.

[pjft]

@TMNTturtlguy a tip from experience: if you plug the card into the computer via a SD card adaptor, flip the "read only" physical switch on the adaptor before the backup. I only make it readable if I'm restoring an image, otherwise the adaptor is always in read only mode. This way there are no weird file system operations from Windows/Mac OS.

[TMNTturtlguy]

@pjft Yeah, i was suspecting that is what is corrupting my card. I think I linked it to dropbox actually. I work remotely and our company uses dropbox to share files. I had dropbox open in the background and when i inserted the card it asked if i wanted to import the files to dropbox. I think that probably corrupted the card. Question though, if the switch is flipped to read only, how do you make any modifications to the card? I am guessing you are suggesting that the card only be inserted for writing a backup image and it is possible to make an image from a locked card?

[pjft]

@TMNTturtlguy well, yes and no. I never do any modifications to the card from anywhere but on the Pi. I don't have much there to change myself anyway - the odd theme or setting, but goes nicely via SSH. Also, Mac OS didn't have native support for ext2 file systems so I can only see the first partition, which is kind of useless anyway.

Otherwise all my ROMs, videos, etc are in a USB drive. In fact, I have a transfer folder on my USB drive that I use if I want to copy something to the Pi (and then on the Pi I copy it to the right place).

But yeah, no changing anything in the card from your computer. First rule of RetroPie Club. :)

[TMNTturtlguy]

@pjft That is what i thought! I have never changed anything from my computer, I use WinSCP or Putty to do everything, however I was clarifying as this thread has adde @StormJH1 who is also looking at these alternative ways to move files and wanted to be clear with him on transferring files.

My backup to my USB card is completed, but i am actually doing real work right now! So I won't be able to play with it more until later. I might have to invest several hours into redoing some of my gamelists. About half are in the roms folder, the other half live in .emulationstation. Does .emulationstation move to the USB? As I learn more I have started to do things best practice, just have not had the chance to clean up all of the gamelist and locations because it takes so much time!

[mediamogul]

@pjft

First rule of RetroPie Club.

Speaking of the club, we really need to rethink the whole "not talking about RetroPie club" rule, as it seems to be negatively affecting our enrollment.

[pjft]

@mediamogul I concede that that was a bit of a dumb move for the RetroPie club, in retrospect, together with the "no shirts, no shoes" thing.

[pjft]

@TMNTturtlguy I wouldn't have moved .emulationstation to the USB, but you can certainly move over the gamelists only.

I suspect that, if they're present on the actual roms/<system> folders, they override the .emulationstation ones. Just make sure to replace all paths by the correct ones (i.e. images pointing to .emulationstation rather than to the RetroPie/roms/<something> folder).

[TMNTturtlguy]

@pjft yeah, that is what i though, i just have to move the ones from .emulationstation into the roms folder and go through a few hundred roms and change the paths....on all the images and videos...talk about a fun evening!

[pjft]

@TMNTturtlguy I mean, other than the number of files, it shouldn't be a lot more complex than:

1 - Open file
2 - Find and replace <reference to /home/pi/.emulationstation folder or equivalent> with /home/pi/RetroPie/roms or equivalent, depending on the structure.
3 - Save.
4 - Move file.

That's how I did it when I had to go through the same thing.

[mediamogul]

@TMNTturtlguy

I might have to invest several hours into redoing some of my gamelists.

What happened to your gamelists?

Edit: Ah, I see it now.

[TMNTturtlguy]

@pjft Light Bulb! DUH, yup, do that for the path for the images and videos, save and close! I just did this when I created my "hacks" systems, I am using the same gamelist, just copied pasted, find all and replaced with the new 'systemname'hacks path! Thanks for reminding me how easy things can be when you slow down for a second and think thing through.

[TMNTturtlguy]

@pjft I successfully backed up my roms to the USB drive and that works great. Deleted all but one rom and bios form each system from the 128gb SD Card and put the USB back into the pi and it was perfect. Reads the games off of the usb stick.

Then i did the rpi-clone. Ran that and put the 16gb card into the pi, started right up with one game in each system. Turned the pi off, put the USB in and turned on the 16gb card and still only 1 game. The new image on the 16gb card is not reading the USB drive. The original 128gb card is. Any thoughts? I was thinking that the raspi-clone was an identical clone, but it is not operating that way?

Edit: When i go into WinSCP and look at the media folder is says that the usb location does not exist.

[pjft]

@TMNTturtlguy It certainly should. Also, for reference, I was successful in using an 8GB card as well, if you want to go as small as possible.

Would I be correct in assuming that you actually installed rpi-clone from

https://github.com/billw2/rpi-clone

?

The instructions I have in my own cheatsheet are:

sudo ./rpi-clone sda -f -v

where "sda" is the device of the destination disk (i.e. may be sdb, for instance).

Also, MAKE SURE TO BOOT THIS WITHOUT THE ROMS USB DRIVE PLUGGED IN, OTHERWISE THIS WILL NOT WORK AS INTENDED.

I wrote them for myself. You can find out which logical device the destination card is via dfand seeing which sd<something> matches the external SD card via USB.

What happens right now if you unplug and re-plug the USB drive, though? You should at least be able to see it mounting as an external drive, even if it doesn't mount as the ROM destination folder.

I'm not fully familiar with the automated mount thingie, but shouldn't be rocket science. What I expect to happen would either be:

• Things work as expected;
• USB drive is recognized, but isn't mounting in the right place for some reason.

If none are happening, that should be a different thing. Definitely unplug it and plug it to see if it's properly recognized.

You can list all devices via dfas mentioned.

Let us know how it goes.

[TMNTturtlguy]

@pjft Yes, i installed from the location you have listed above. I also used the sudo code above exactly as stated and i did not have the USB Roms drive plugged in. I actually did this process twice with the same results both times.

If i unplug the roms usb and plug it back in, nothing happens, light on the usb blinks a few times, but it is not read. If i use df it does not show the Roms USB and it also does not show up in WinSCP. Here is a shot of the df:

also assuming you would help troubleshoot, i took these shots of the rpi-clone process.

Here were my steps. Shutdown pi, take out Roms USB. Start up pi and quit emulationstation. Using putty i ran the rpi-clone. It asked if i wanted to unmount /media/usb0

At this point the only thing plugged into the pi is the usb card reader with the new smaller SD card in it. I chose no and it cancels the clone. I have to choose yes to proceed. Again, the Roms USB was not installed.

Next i get this screen, the first attempt i named the drive, the second attempt shown here, i did not name the drive, thinking that maybe the name of the drive was causing the error.

Ran this through, finished, no luck with the roms usb. Nothing even shows up in my WinSCP media, it is empty. I tried to go in an reenable the USB Automated mount, no luck. Tried to mount a new usb, wont do mount a new usb either. I hope this is enough info to trouble shoot? I have not yet slept and i have to work in 4 hours, so i am going to shut down for the night, but hope to continue this tomorrow. Thanks!

[pjft]

@TMNTturtlguy Definitely go to sleep, we'll sort things out one way or the other :)

I'm confused as to why the USB drive doesn't even show up. You do confirm it shows up using your other card, correct?

What does dfshow on the other scenario? What does lsusbshow when you plug the USB drive in? What about dmesg | tail? And what about cat /etc/fstab, in both your SD cards?

I am not an expert in Raspbian, nor in the auto-mount mechanism from recent RetroPie, but I'm happy to chime in and attempt to help you out. Also, this thread might help if nothing funny is going on at the moment.

[ibiza7]

merci beaucoup !!:D ca va mieux

[TMNTturtlguy]

@pjft Thanks for all the help as usual!

Yes, confirmed I am running off of the USB on the original card. Here is the info for the original card running off of the USB:
Working card with working USB - all commands shown

rpi-clone card non working usb - all commands shown

[pjft]

@TMNTturtlguy Is your USB drive a Kingston drive?

In the "bad" card, you mention the USB drive blinks, then stops. Can you unplug it, run those commands (dmesg | tail, lsusband df), and then replug it and run the commands again?

Also, does cat /var/log/syslogoutput anything for you? If so, what does it state for both cards?

I'm somewhat stumped, but we'll make it work.

Alternatively, if this thread doesn't help (just noticed I forgot to paste the link earlier), I'd love to read up on both boot logs (dmesg) - for the good and bad cards.

But a few things first to try. As I said, I have this working though I am not using the automated mount from RetroPie, but rather the old method via fstab. I am confused as to why you would have such differences between cards, though.

[TMNTturtlguy]

@pjft The usb drive is a computer stores brand - they offer lifetime warranties on them, so if they go bad, they just give you a new one. So i can't be positive, but I assume it is the one that shows up.

No USB inserted

USB inserted

Cat /var/log/syslog for non working Card

cat /var/log/syslog for working card