User Tools

Site Tools


programming:python:rclone-vers

Rclone Backblaze Version Downloader

I am a big fan of rclone. It's a pretty great tool for running backups to a number of different providers, with a number of great security options like filename and dirname encryption on top of encrypting data. That being said, there are some limitations with the crypt implementation with encrypted files and directories. Basically, you can't get and decrypt the older versions of files (assuming you are saving/have older versions of files) with Backblaze.

Wrapping Rclone

I found this nice little shell script that will actually allow you to get an older version of a file. It's worth noting that rclone-vers.py is a much more optimized version of that shell script, which will scan through every file in your bucket twice to find the desired files. There are no full scans with this script. While it does the job, it's pretty simple and hard-coded. Therefore, I whipped up a Python script that's more flexible, allows you to choose the version to download, or simply download all the older versions.

Setup

This assumes you have already been using rclone with Backblaze with the filename_encryption = standard and directory_name_encryption = true options set. Beyond the obvious, having rclone installed, you will need to modify your rclone.conf file to add some entries on top of what you should have for your crypt setup. Your conf file should be in $HOME/.config/rclone/rclone.conf. Here's an example configuration file (with secret stuff changed):

[remote]
type = b2                                                                       
account = abc1234                                                               
key = qwerty1234                                
endpoint = 
 
[b2_secret]
type = crypt
remote = remote:my-bucket-name-rclone-main          
filename_encryption = standard
directory_name_encryption = true
password = supersecretstuff                                      
password2 = supersecretstuff_2
 
[local]
type = local
nounc =
 
[local-crypt]
type = crypt
remote = local:/tmp
filename_encryption = standard
password = supersecretstuff
password2 = supersecretstuff_2

You should have something similar to this, with the exception of the 2 “local” sections. Those sections are what you will need to add to your config file. You can copy paste the [local] and the [local-crypt] sections above to your config, but do note that the password and password2 fields in [local-crypt] should match what you have in your main b2 config (b2_secret in the example).

It's worth noting that under [local-crypt], the path in the remote key can be changed to whatever you would like. In the example, it's local:/tmp, which means that I'm using /tmp as a temp dir when downloading the files, but you can change /tmp to whatever you would like.

Getting Previous Versions

Now that your config is setup, running the script is pretty self-explanatory given it's –help options (the below example is not the most up to date).

usage: rclone-vers.py [-h] [-c RCLONE_CONF] [-r CRYPT_REMOTE] [-l CRYPT_LOCAL]
                      [-e ENC_FNAME] [-o OUTDIR] [-a] [-V] [-D]
                      [fname]

This will allow you to, given a path to a file, decrypt an older version of
the file

positional arguments:
  fname                 The file to get an older version of (default: None)

options:
  -h, --help            show this help message and exit
  -c RCLONE_CONF, --rclone-conf RCLONE_CONF
                        The path to the rclone config (default:
                        /root/.config/rclone/rclone.conf)
  -r CRYPT_REMOTE, --crypt-remote CRYPT_REMOTE
                        The name of the default encrypted remote from the
                        rclone conf (default: b2_secret)
  -l CRYPT_LOCAL, --crypt-local CRYPT_LOCAL
                        The configuration for local enc. from the rclone.conf
                        (default: local-crypt)
  -o OUTDIR, --outdir OUTDIR
                        The directory to put the unencrypted file in (default:
                        ./)
  -a, --all-versions    Instead of prompting for which version to get, just
                        get all previous versions (note that the *current*
                        version will not be downloaded) (default: False)
  -V, --version         Print the version and exit (default: False)
  -D, --debug           Add debug output (default: False)

The most basic way to run this is simply ./rclone-vers.py /local/path/to/file/with/versions. If there are older versions of the specified file, you will be prompted to choose which one to download (based on the timestamp). You can also choose to run it with the –all-versions option, which will download all the previous versions of the file. Note the it will NOT download the latest version, which you can always just get using rclone copy. The file(s) will, by default, will be downloaded to your current directory.

Depending on your config, you'll likely have to set the –crypt-remote option to match your config.

Feel free to open an issue or submit a pull request on Github if you run into problems, but hopefully this ends up being useful for others too.

programming/python/rclone-vers.txt · Last modified: 2023/10/23 18:47 by jay