Please do not post a support request without first reading and following the advice in https://retropie.org.uk/forum/topic/3/read-this-first

[SCRIPT] Setup script to sync saves to cloud services



  • Hey folks,

    Have you ever wanted to synchonize your saves and states to another machine? Maybe even use it as an offsite backup only? You did? Well, today you can! I created some scripts which let you do just that. They have been tested with DropBox and Google Drive only but other services should work as well.

    How do you install it?

    So, how do you get these scripts up and running? Simple, just run these commands in the shell:

    wget -N -P ~/scripts https://www.dropbox.com/s/t1ertj6zv8d6992/rclone_script-install.sh
    chmod 755 ~/scripts/rclone_script-install.sh
    ~/scripts/rclone_script-install.sh
    

    EDIT:
    Please do not use these commands any longer. I've moved the scripts and their documentation to GitHub:
    https://github.com/Jandalf81/rclone_script
    END EDIT

    What does it do?

    These commands download a setup script, make it executable and then run it. That script downloads a bunch of other things, here's a rundown:

    • RCLONE: this is the magic tool which let's us use a multitude of cloud services from RetroPie
    • PNGVIEW: used to show notifications within RetroPie, so you know what the scripts are doing
    • IMAGEMAGICK: used to create images containing the notifications
    • RCLONE_SCRIPT: this is my script which tells RetroPie to up- and download the save files
    • UNINSTALL script: in case you want to remove these things again. But why would you want to do that?

    The setup script also configures some things:

    • SAVE and STATE files: Normally, these are within each ROM directory. The setup script changes that so each system has it's own directory for saves and states. All the emulators are configured to use these new directories
    • RUNCOMMAND: There are scripts which are executed each time a ROM is started and stopped. These scripts will now call RCLONE_SCRIPT to up- or download the saves and states

    What do you still have to do manually?

    The setup script will do as much as possible for you. All you have to do is start the script and follow the instructions. If you already have some saves you need to move them into the new directories manually afterwards.

    Then, it's all automated. As long as your RetroPie has internet, the save files will we downloaded when starting and uploaded when stopping a ROM.

    How does it look like?

    Anyway, here's Wonderwall an IMGur album with some pictures of the setup process and what it looks like afterwards. I recommend taking a look as the setup process for the RCLONE remote is explained here...
    https://imgur.com/a/nOFxP5Y

    The rest

    Please note that this still might have errors. I'll do my best to support anyone having questions or errors but I can not be held responsible for data loss...

    Sources



  • @jandalf81 Not Bad! I've written once a script that just changes location of SRM and SAVESTATES .... take a look here

    But in combination with rclone it seems to be smooth. Just a small view what rclone supports. Imho Dropbox sucks but rclone can handle much more...

    Rclone is a command line program to sync files and directories to and from:

    • Amazon Drive (See note)
    • Amazon S3
    • Backblaze B2
    • Box
    • Ceph
    • DigitalOcean Spaces
    • Dreamhost
    • Dropbox
    • FTP
    • Google Cloud Storage
    • Google Drive
    • HTTP
    • Hubic
    • IBM COS S3
    • Memset Memstore
    • Mega
    • Microsoft Azure Blob Storage
    • Microsoft OneDrive
    • Minio
    • Nextcloud
    • OVH
    • OpenDrive
    • Openstack Swift
    • Oracle Cloud Storage
    • ownCloud
    • pCloud
    • put.io
    • QingStor
    • Rackspace Cloud Files
    • SFTP
    • Wasabi
    • WebDAV
    • Yandex Disk
    • The local filesystem


  • Just a question : why don't you install rclone through apt ?
    It's available on raspbian and would be more secure/compliant, and you already use raspbian package for imagemagick.



  • @cyperghost Thanks! Yeah, rclone seemed really capable. I chose that approach to hopefully support a lot of cloud services so nobody can complain that "their" service is not supported... ;-)

    @Sano It is? I must admin that I didn't check... I'll update my script accordingly in the coming days!

    EDIT:
    I just checked installing RCLONE via apt-get... I will not update my script to do this as the version from apt-get is 1,5 years old (1.35, compiled 2017-01-02). There have been a number of important updates since then, e. g. support the DropBox API v2, adding sFTP remote, WebDav remote, Google Drive API v3. Don't know why their version is not updated, though...



  • UPDATE #1
    As another user on Reddit recommended I have implemented a way to start a full sync. Update your local copy of the installation script like above but then start it like this

    ~/scripts/rclone_script-install.sh "1"
    

    This will download an additional script which can be started via "RetroPie / RCLONE_SCRIPT Full Sync" on your RPi. It will not be fancy. The script will

    • download all saves newer than on your RPi
    • upload all saves newer than on your remote

    In theory, both sides should have all the latest files afterwards. Please note that this will download all the saves, even if the ROM is not on your RPi! The "normal" script will transfer these files on a per-ROM basis



  • Very cool!



  • Just a heads up, I moved the files from DropBox to GitHub:
    https://github.com/Jandalf81/rclone_script

    I've also written a quite extensive documentation there...



  • I want to use this, but every time I install it and reboot my Pi becomes super slow to respond... almost completely unresponsive at boot. I've luckily been able to finally get in the last 2 times and struggle to get it to uninstall with the script and reboot, then things are back to normal. Before I've had it totally bring down my system and had to re-image the card.
    I'm using a Pi 3B, not overclocking, and just using the stock 4.4 image that I update just using the retropie-setup update menu... There any log files I can attempt to look at?



  • @HumanMoose
    Yes, the scripts creates log files in it's own directory /home/pi/scripts/rclone_script/rclone_script.log
    Unfortunately, I have no idea what could happen here. There's nothing this script installs which starts at boot...?

    I'll try to reproduce this. Did you update to 4.4.2 before or after applying this script? Your Pi is a 3B, not 3B+, right?
    Can you send the logfiles rclone_script-install.log and rclone_script.log so I can take a look at them?



  • @jandalf81
    Sorry it's taken me a bit to get back with you, I've been busy...
    Yes, my Pi is a 3B not a 3B+, and I updated before running the script.
    I just tried reinstalling the script and once I told it to go ahead and reboot, it tanked my system like before. It took a lot longer than before but I was finally able to SSH and SCP into the system, I could only find the rclone_script-install.log in the rclone_script folder, I looked around for the rclone_script.log but couldn't find one anywhere. Looked around the /var/log folder and just got the syslog and messages from there.
    What is the best way to send you the log that I do have?



  • @HumanMoose
    Please open a new issue and add the files here: https://github.com/Jandalf81/rclone_script/issues

    It seems right that you only found one log file, rclone_script.log is only created when you actually sync the saves upon starting or stopping a ROM. If you did that, there should be a log file in the directory.

    I'll try to reproduce it with my Pi now:

    • fresh install of RetroPie 4.4
    • update to 4.4.2 via RetroPie Setup
    • install rclone_script
    • define a remote for rclone
    • reboot

    Can you please tell me what kind of remote you have configured? Which cloud service do you use? Did you install anything else?

    EDIT:
    I just played Batman after doing all the steps I mentioned... I don't see any slowdown on my side. Could you check what the htop command gives you? This is kind of like Window's Task manager, it should show you the most expensive processes...

    And just to be clear, we are talking about heavy slowdowns right when RetroPie starts, right? We are not talking about starting any game?



  • @jandalf81
    Well, it looks like I was able to fix it myself somehow....?
    Previously when I had been running through the installer, when it gets to the point of asking for the remote base directory on my Dropbox, I was going with retropie. It would create it fine, but I noticed last time that after that and I told it to use the internet as the sync network it seemed to start slowing down my system then.
    And the slowdown has been heavy from bootup, it would start loading processes and then when it gets to the splash screens it would throw up error messages at the bottom of the screen that it couldn't load them then after a couple minutes it would be able to start mpeg123 and play my background music, then a few minutes later would give me the Emulation Station load screen and I would just let it sit there and it maybe would finish booting but the menu would still be unresponsive. This is at least 10 minutes into boot. I might be able to SSH into it and if so commands were slow to run, just getting a password prompt takes about 5 seconds instead of being instant.

    So this time I uninstalled, rebooted and the system was normal and fine. I started the install process again and this time when it got to asking for the remote base folder on the Dropbox, I left it at RetroArch and it finished up, I didn't notice any slowdown when selecting the internet and rebooted. Since then it has been running fine, I've rebooted several times without issue and was even able to install your other script for the auto save state selector. For whatever reason it seems to not have liked giving the remote folders the retropie name? I'll let you know if I have any other issues, but I seem to be OK now....



  • @HumanMoose
    Thank you very much for this update! It still is strange, though... There is nothing in the script that starts during the installation process or right after boot. I'll try to reproduce that with a fresh install myself.

    EDIT:
    I just tried the following:

    • fresh install of RetroPie 4.4.0
    • updated to RetroPie 4.4.2
    • installed rclone_script
    • created a remote to DropBox named "retropie"
    • reboot
    • play a game
    • look at htop

    There is no slowdown there. I'm really sorry but I don't see a problem here. Everything on my side looks the way it should.



  • Sounds pretty interesting. But according to this:

    starting a game will trigger RCLONE to download save files for the game from the remote

    stopping a game will trigger RCLONE to upload save files for the game to the remote

    What happens if I play and save a state out of the network? The next time I start a rom with the rasp connected, It will download and overwrite with the older state?

    Also, if I play and save a state out of the network and I don't play it again that rom with the rasp connected, then It will never upload the state? Is there any way to manually update and upload all the states in the raspberry?



  • Yeah, you got most things right.

    Starting or stopping a game will trigger the sync process, but only newer files will be transfered.
    If you played a game without internet connection before, your local save is newer than the remote one, so no download should happen.

    There's a menu item to trigger a systemwide sync of all save files in both directions. Of course, this also only transfers newer files to the other side. In theory, both sides should contain the newest version of each save afterwards.

    Please let me emphasize that I'm a hobby programmer. I make errors. I'll try to help but I cannot guarantee that there'll be no failures ;-)



  • @jandalf81 thanks a lot! Then I'll download it



  • @jandalf81 Love the script and thank you. My log keeps complaining the gateway cannot be PINGed, even though my pi has access and I can ping my default gateway fine. DNS resolution works fine too.

    Looking forward to using this script, as I just lost a major game save on my psx due to my tinkering :)

    Thanks agian,
    Robo



  • @jandalf81 Just a quick update: Manual full syncs to/from Dropbox work fine, and saw the current network check issues documented on your github page. I added the "return 0" workaround to the script as you mentioned and it now works like a charm! Love the script, very helpful!


Log in to reply
 

Contributions to the project are always appreciated, so if you would like to support us with a donation you can do so here.

Hosting provided by Mythic-Beasts. See the Hosting Information page for more information.