[SCRIPT] Setup script to sync saves to cloud services

jandalf81

Hey folks,

Have you ever wanted to synchonize your saves and states to another machine? Maybe even use it as an offsite backup only? You did? Well, today you can! I created some scripts which let you do just that. They have been tested with DropBox and Google Drive only but other services should work as well.

How do you install it?

So, how do you get these scripts up and running? Simple, just run these commands in the shell:

wget -N -P ~/scripts https://www.dropbox.com/s/t1ertj6zv8d6992/rclone_script-install.sh
chmod 755 ~/scripts/rclone_script-install.sh
~/scripts/rclone_script-install.sh

EDIT:
Please do not use these commands any longer. I've moved the scripts and their documentation to GitHub:
https://github.com/Jandalf81/rclone_script
END EDIT

What does it do?

These commands download a setup script, make it executable and then run it. That script downloads a bunch of other things, here's a rundown:

RCLONE: this is the magic tool which let's us use a multitude of cloud services from RetroPie
PNGVIEW: used to show notifications within RetroPie, so you know what the scripts are doing
IMAGEMAGICK: used to create images containing the notifications
RCLONE_SCRIPT: this is my script which tells RetroPie to up- and download the save files
UNINSTALL script: in case you want to remove these things again. But why would you want to do that?

The setup script also configures some things:

SAVE and STATE files: Normally, these are within each ROM directory. The setup script changes that so each system has it's own directory for saves and states. All the emulators are configured to use these new directories
RUNCOMMAND: There are scripts which are executed each time a ROM is started and stopped. These scripts will now call RCLONE_SCRIPT to up- or download the saves and states

What do you still have to do manually?

The setup script will do as much as possible for you. All you have to do is start the script and follow the instructions. If you already have some saves you need to move them into the new directories manually afterwards.

Then, it's all automated. As long as your RetroPie has internet, the save files will we downloaded when starting and uploaded when stopping a ROM.

How does it look like?

Anyway, here's ~~Wonderwall~~ an IMGur album with some pictures of the setup process and what it looks like afterwards. I recommend taking a look as the setup process for the RCLONE remote is explained here...
https://imgur.com/a/nOFxP5Y

The rest

Please note that this still might have errors. I'll do my best to support anyone having questions or errors but I can not be held responsible for data loss...

Sources

cyperghost

@jandalf81 Not Bad! I've written once a script that just changes location of SRM and SAVESTATES .... take a look here

But in combination with rclone it seems to be smooth. Just a small view what rclone supports. Imho Dropbox sucks but rclone can handle much more...

Rclone is a command line program to sync files and directories to and from:

Amazon Drive (See note)
Amazon S3
Backblaze B2
Box
+ Ceph
- DigitalOcean Spaces

+ Dreamhost

Dropbox
FTP
Google Cloud Storage
Google Drive
HTTP
Hubic
+ IBM COS S3
+ Memset Memstore
+ Mega
Microsoft Azure Blob Storage
Microsoft OneDrive
Minio
Nextcloud
+ OVH
+ OpenDrive
Openstack Swift
+ Oracle Cloud Storage
ownCloud
pCloud
put.io
QingStor
Rackspace Cloud Files
+ SFTP
Wasabi
WebDAV
Yandex Disk
The local filesystem

Sano

Just a question : why don't you install rclone through apt ?
It's available on raspbian and would be more secure/compliant, and you already use raspbian package for imagemagick.

jandalf81

@cyperghost Thanks! Yeah, rclone seemed really capable. I chose that approach to hopefully support a lot of cloud services so nobody can complain that "their" service is not supported... ;-)

@Sano It is? I must admin that I didn't check... I'll update my script accordingly in the coming days!

EDIT:
I just checked installing RCLONE via apt-get... I will not update my script to do this as the version from apt-get is 1,5 years old (1.35, compiled 2017-01-02). There have been a number of important updates since then, e. g. support the DropBox API v2, adding sFTP remote, WebDav remote, Google Drive API v3. Don't know why their version is not updated, though...

jandalf81

UPDATE #1
As another user on Reddit recommended I have implemented a way to start a full sync. Update your local copy of the installation script like above but then start it like this

~/scripts/rclone_script-install.sh "1"

This will download an additional script which can be started via "RetroPie / RCLONE_SCRIPT Full Sync" on your RPi. It will not be fancy. The script will

download all saves newer than on your RPi
upload all saves newer than on your remote

In theory, both sides should have all the latest files afterwards. Please note that this will download all the saves, even if the ROM is not on your RPi! The "normal" script will transfer these files on a per-ROM basis

Newellj79

Very cool!

jandalf81

Just a heads up, I moved the files from DropBox to GitHub:
https://github.com/Jandalf81/rclone_script

I've also written a quite extensive documentation there...

HumanMoose

I want to use this, but every time I install it and reboot my Pi becomes super slow to respond... almost completely unresponsive at boot. I've luckily been able to finally get in the last 2 times and struggle to get it to uninstall with the script and reboot, then things are back to normal. Before I've had it totally bring down my system and had to re-image the card.
I'm using a Pi 3B, not overclocking, and just using the stock 4.4 image that I update just using the retropie-setup update menu... There any log files I can attempt to look at?

jandalf81

@HumanMoose
Yes, the scripts creates log files in it's own directory /home/pi/scripts/rclone_script/rclone_script.log
Unfortunately, I have no idea what could happen here. There's nothing this script installs which starts at boot...?

I'll try to reproduce this. Did you update to 4.4.2 before or after applying this script? Your Pi is a 3B, not 3B+, right?
Can you send the logfiles rclone_script-install.log and rclone_script.log so I can take a look at them?

HumanMoose

@jandalf81
Sorry it's taken me a bit to get back with you, I've been busy...
Yes, my Pi is a 3B not a 3B+, and I updated before running the script.
I just tried reinstalling the script and once I told it to go ahead and reboot, it tanked my system like before. It took a lot longer than before but I was finally able to SSH and SCP into the system, I could only find the rclone_script-install.log in the rclone_script folder, I looked around for the rclone_script.log but couldn't find one anywhere. Looked around the /var/log folder and just got the syslog and messages from there.
What is the best way to send you the log that I do have?

jandalf81

@HumanMoose
Please open a new issue and add the files here: https://github.com/Jandalf81/rclone_script/issues

It seems right that you only found one log file, rclone_script.log is only created when you actually sync the saves upon starting or stopping a ROM. If you did that, there should be a log file in the directory.

I'll try to reproduce it with my Pi now:

fresh install of RetroPie 4.4
update to 4.4.2 via RetroPie Setup
install rclone_script
define a remote for rclone
reboot

Can you please tell me what kind of remote you have configured? Which cloud service do you use? Did you install anything else?

EDIT:
I just played Batman after doing all the steps I mentioned... I don't see any slowdown on my side. Could you check what the htop command gives you? This is kind of like Window's Task manager, it should show you the most expensive processes...

And just to be clear, we are talking about heavy slowdowns right when RetroPie starts, right? We are not talking about starting any game?

HumanMoose

@jandalf81
Well, it looks like I was able to fix it myself somehow....?
Previously when I had been running through the installer, when it gets to the point of asking for the remote base directory on my Dropbox, I was going with retropie. It would create it fine, but I noticed last time that after that and I told it to use the internet as the sync network it seemed to start slowing down my system then.
And the slowdown has been heavy from bootup, it would start loading processes and then when it gets to the splash screens it would throw up error messages at the bottom of the screen that it couldn't load them then after a couple minutes it would be able to start mpeg123 and play my background music, then a few minutes later would give me the Emulation Station load screen and I would just let it sit there and it maybe would finish booting but the menu would still be unresponsive. This is at least 10 minutes into boot. I might be able to SSH into it and if so commands were slow to run, just getting a password prompt takes about 5 seconds instead of being instant.

So this time I uninstalled, rebooted and the system was normal and fine. I started the install process again and this time when it got to asking for the remote base folder on the Dropbox, I left it at RetroArch and it finished up, I didn't notice any slowdown when selecting the internet and rebooted. Since then it has been running fine, I've rebooted several times without issue and was even able to install your other script for the auto save state selector. For whatever reason it seems to not have liked giving the remote folders the retropie name? I'll let you know if I have any other issues, but I seem to be OK now....

jandalf81

@HumanMoose
Thank you very much for this update! It still is strange, though... There is nothing in the script that starts during the installation process or right after boot. I'll try to reproduce that with a fresh install myself.

EDIT:
I just tried the following:

fresh install of RetroPie 4.4.0
updated to RetroPie 4.4.2
installed rclone_script
created a remote to DropBox named "retropie"
reboot
play a game
look at htop

There is no slowdown there. I'm really sorry but I don't see a problem here. Everything on my side looks the way it should.

Carcayu

Sounds pretty interesting. But according to this:

starting a game will trigger RCLONE to download save files for the game from the remote

stopping a game will trigger RCLONE to upload save files for the game to the remote

What happens if I play and save a state out of the network? The next time I start a rom with the rasp connected, It will download and overwrite with the older state?

Also, if I play and save a state out of the network and I don't play it again that rom with the rasp connected, then It will never upload the state? Is there any way to manually update and upload all the states in the raspberry?

jandalf81

Yeah, you got most things right.

Starting or stopping a game will trigger the sync process, but only newer files will be transfered.
If you played a game without internet connection before, your local save is newer than the remote one, so no download should happen.

There's a menu item to trigger a systemwide sync of all save files in both directions. Of course, this also only transfers newer files to the other side. In theory, both sides should contain the newest version of each save afterwards.

Please let me emphasize that I'm a hobby programmer. I make errors. I'll try to help but I cannot guarantee that there'll be no failures ;-)

Carcayu

@jandalf81 thanks a lot! Then I'll download it

robotronman

@jandalf81 Love the script and thank you. My log keeps complaining the gateway cannot be PINGed, even though my pi has access and I can ping my default gateway fine. DNS resolution works fine too.

Looking forward to using this script, as I just lost a major game save on my psx due to my tinkering :)

Thanks agian,
Robo

robotronman

@jandalf81 Just a quick update: Manual full syncs to/from Dropbox work fine, and saw the current network check issues documented on your github page. I added the "return 0" workaround to the script as you mentioned and it now works like a charm! Love the script, very helpful!

Addison

I get the following error in the installation menu and then it kicks me to the command prompt.

1. RCLONE                                                                              
x       1a. Test for RCLONE binary                      [ NOT FOUND   ]                
x       1b. Download RCLONE binary                      [ FAILED      ]

Is this setup script too outdated now?

I finally figured out cloud sharing on my Windows computer and also Android devices today through RetroArch. :)

Would very much love to get that rolling as well in Retropie.

For reference, I did follow the initial instructions found here like suggested:
https://github.com/Jandalf81/rclone_script