Encrypted snapshot-based incremental backups with OpenZFS on FreeBSD 12.2


Encrypted backups everywhere!

October 2020.
The FreeBSD logoImage Image

Goal

Making a backup server which receives incremental updates but can never decrypt the data.

Introduction

FreeBSD is moving to OpenZFS. When it's released, version 13 will have moved to OpenZFS.

Right now it's possible to try OpenZFS alongside FreeBSD's ZFS distribution by building it from the port collection.

There is one feature I'm really interested in: encryption at rest. I always ensure that people who steal my hard drives can never access my data.

Encrypting ZFS pools has been possible for a while now. The most common way of doing it was with GELI. Basically, a new layer is added to the system, which exposes a new block device where ZFS can be used on top. This is working great, with great performance.

However, when moving datasets between systems, ZFS snapshots are still transmitted unencrypted, and the receiving server is responsible for encrypting and storing the data in a proper way. This is good enough if all part of the infrastructure is present and managed in a trusted location and all networks between the two are reliable.

With native ZFS encryption, OpenZFS moves the encryption to the actual file system implementation. This means that we can now use manipulate ZFS snapshots without ever handling unencrypted data, and so we can make a backup server at a location we don't fully trust.

Let's try it!

The source server

Image

We're going to simulate a production server migrating to a newer release and migrating its data to the new ZFS.

We start from a standard 12.1-RELEASE server:

root@jambon-production-server:~ # uname -a
FreeBSD jambon-production-server 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC  amd64

The server has a pool full of sensitive, very important data:

root@jambon-production-server:~ # zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zdata  19.5G  3.55G  15.9G        -         -     3%    18%  1.00x  ONLINE  -

root@jambon-production-server:~ # zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
zdata                 3.50G  15.4G    23K  /zdata
zdata/jambon          3.50G  15.4G    24K  /zdata/jambon
zdata/jambon/bayonne   512M  15.4G   512M  /zdata/jambon/bayonne
zdata/jambon/blanc    1.00G  15.4G  1.00G  /zdata/jambon/blanc
zdata/jambon/parme    2.00G  15.4G  2.00G  /zdata/jambon/parme
zdata/jambon/pays       23K  15.4G    23K  /zdata/jambon/pays

Here is the list of supported features:

root@jambon-production-server:~ # zpool upgrade -v
This system supports ZFS pool feature flags.

The following features are supported:

FEAT DESCRIPTION
-------------------------------------------------------------
async_destroy                         (read-only compatible)
     Destroy filesystems asynchronously.
empty_bpobj                           (read-only compatible)
     Snapshots use less space.
lz4_compress
     LZ4 compression algorithm support.
multi_vdev_crash_dump
     Crash dumps to multiple vdev pools.
spacemap_histogram                    (read-only compatible)
     Spacemaps maintain space histograms.
enabled_txg                           (read-only compatible)
     Record txg at which a feature is enabled
hole_birth
     Retain hole birth txg for more precise zfs send
extensible_dataset
     Enhanced dataset functionality, used by other features.
embedded_data
     Blocks which compress very well use even less space.
bookmarks                             (read-only compatible)
     "zfs bookmark" command
filesystem_limits                     (read-only compatible)
     Filesystem and snapshot limits.
large_blocks
     Support for blocks larger than 128KB.
large_dnode
     Variable on-disk size of dnodes.
sha512
     SHA-512/256 hash algorithm.
skein
     Skein hash algorithm.
device_removal
     Top-level vdevs can be removed, reducing logical pool size.
obsolete_counts                       (read-only compatible)
     Reduce memory used by removed devices when their blocks are freed or remapped.
zpool_checkpoint                      (read-only compatible)
     Pool state can be checkpointed, allowing rewind later.
spacemap_v2                           (read-only compatible)
     Space maps representing large segments are more efficient.

The following legacy versions are also supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
 14  passthrough-x aclinherit
 15  user/group space accounting
 16  stmf property support
 17  Triple-parity RAID-Z
 18  Snapshot user holds
 19  Log device removal
 20  Compression using zle (zero-length encoding)
 21  Deduplication
 22  Received properties
 23  Slim ZIL
 24  System attributes
 25  Improved scrub stats
 26  Improved snapshot deletion performance
 27  Improved snapshot creation performance
 28  Multiple vdev replacements

Yep, this is a good old FreeBSD ZFS pool.

Let's upgrade to 12.2-RELEASE:

root@jambon-production-server:~ # freebsd-update -r 12.2-RELEASE upgrade
root@jambon-production-server:~ # freebsd-update install
root@jambon-production-server:~ # reboot
root@jambon-production-server:~ # freebsd-update install
root@jambon-production-server:~ # uname -a
FreeBSD jambon-production-server 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  amd64

Installing OpenZFS

OpenZFS is present in the port collection:

Building the kernel module requires the source tree to be present. Download it:

root@jambon-production-server:~ # cd /usr/src
root@jambon-production-server:~ # fetch 'http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/src.txz'
root@jambon-production-server:~ # tar --strip-components 2 -x -J -f src.txz

Let's install them:

root@jambon-production-server:~ # portsnap fetch extract
root@jambon-production-server:~ # make -C /usr/ports/sysutils/openzfs-kmod install clean
root@jambon-production-server:~ # make -C /usr/ports/sysutils/openzfs install clean

root@jambon-production-server:~ # pkg info | grep openzfs
openzfs-2020102700             OpenZFS userland for FreeBSD
openzfs-kmod-2020102700        OpenZFS kernel module for FreeBSD

Now, here comes the part where it's hard to not mix up commands.

This means that if you don't prefix your commands you might run the wrong version. OpenZFS pools need to be manipulated with /usr/local/sbin/zpool and /usr/local/sbin/zfs.

We can check that the new ZFS has more features:

root@jambon-production-server:~ # /usr/local/sbin/zpool upgrade -v
FEAT DESCRIPTION
-------------------------------------------------------------
[...]
encryption
     Support for dataset level encryption
[...]

Migrating and encrypting the pool

Image

First, we need to upgrade the pool. This is a one way operation. The pool cannot be downgraded to its previous version.

root@jambon-production-server:~ # /usr/local/sbin/zpool upgrade zdata
This system supports ZFS pool feature flags.

Enabled the following features on 'zdata':
  userobj_accounting
  encryption
  project_quota
  allocation_classes
  resilver_defer
  bookmark_v2
  redaction_bookmarks
  redacted_datasets
  bookmark_written
  log_spacemap
  livelist
  device_rebuild
  zstd_compress

Unfortunately, it's not possible to encrypt an existing dataset. The encryption flag can only be set to datasets when they're created (-o encryption is read only). However, we can copy the data (including the snapshots) to new encrypted datasets.

First, let's create a key:

root@jambon-production-server:~ # dd if=/dev/random of=/etc/jambon-zfs-key bs=1 count=32

Don't forget to copy the key to your backup infrastructure before storing actual data on the pool. If you lose the key, you won't be able to unlock the pool and access the data anymore.

Create an encrypted dataset:

root@jambon-production-server:~ # /usr/local/sbin/zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///etc/jambon-zfs-key zdata/encrypted

Check that the dataset is encrypted:

root@jambon-production-server:~ # /usr/local/sbin/zfs get all zdata/encrypted
NAME             PROPERTY    VALUE        SOURCE
[...]
zdata/encrypted  encryption            aes-256-gcm                 -
zdata/encrypted  keylocation           file:///etc/jambon-zfs-key  local
zdata/encrypted  keyformat             raw                         -

List the snapshots of the source datasets:

root@jambon-production-server:~ # /usr/local/sbin/zfs list -t snapshot -o name zdata/jambon/blanc
NAME
zdata/jambon/blanc@2018
zdata/jambon/blanc@2019
zdata/jambon/blanc@2020
zdata/jambon/blanc@migration

Copy the first snapshot:

root@jambon-production-server:~ # /usr/local/sbin/zfs create zdata/encrypted/jambon
root@jambon-production-server:~ # /usr/local/sbin/zfs send -R zdata/jambon/blanc@2018 \
     | /usr/local/sbin/zfs recv -x encryption zdata/encrypted/jambon/blanc

That's it: the dataset is now encrypted. It got its encryption parameters from the parent dataset we created earlier:

root@jambon-production-server:~ # /usr/local/sbin/zfs get all zdata/encrypted/jambon/blanc
[...]
zdata/encrypted/jambon/blanc  encryption            aes-256-gcm                    -
zdata/encrypted/jambon/blanc  keylocation           none                           default
zdata/encrypted/jambon/blanc  keyformat             raw                            -
zdata/encrypted/jambon/blanc  pbkdf2iters           0                              default
zdata/encrypted/jambon/blanc  encryptionroot        zdata/encrypted                -
zdata/encrypted/jambon/blanc  keystatus             available                      -

Copy the other snapshots:

root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2018 zdata/jambon/blanc@2019 \
     | /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc
root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2019 zdata/jambon/blanc@2020 \
     | /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc
root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2020 zdata/jambon/blanc@migration \
     | /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc

root@jambon-production-server:~ # /usr/local/sbin/zfs list -t snapshot zdata/encrypted/jambon/blanc
NAME                                     USED  AVAIL     REFER  MOUNTPOINT
zdata/encrypted/jambon/blanc@2018         xGB      -     x.00G  -
zdata/encrypted/jambon/blanc@2019         xGB      -     x.00G  -
zdata/encrypted/jambon/blanc@2020         xGB      -     x.00G  -
zdata/encrypted/jambon/blanc@migration    xGB      -     x.00G  -

Now we have the data twice: once unencrypted, and once encrypted:

root@jambon-production-server:~ # /usr/local/sbin/zfs list
NAME                             USED  AVAIL     REFER  MOUNTPOINT
zdata                           7.10G  11.8G       23K  /zdata
zdata/encrypted                 3.55G  11.8G      100K  /zdata/encrypted
zdata/encrypted/jambon          3.55G  11.8G      103K  /zdata/encrypted/jambon
zdata/encrypted/jambon/bayonne   513M  11.8G      513M  /zdata/encrypted/jambon/bayonne
zdata/encrypted/jambon/blanc    1.00G  11.8G     1.00G  /zdata/encrypted/jambon/blanc
zdata/encrypted/jambon/parme    2.00G  11.8G     2.00G  /zdata/encrypted/jambon/parme
zdata/encrypted/jambon/pays     50.2M  11.8G     50.1M  /zdata/encrypted/jambon/pays
zdata/jambon                    3.55G  11.8G       24K  /zdata/jambon
zdata/jambon/bayonne             512M  11.8G      512M  /zdata/jambon/bayonne
zdata/jambon/blanc              1.00G  11.8G     1.00G  /zdata/jambon/blanc
zdata/jambon/parme              2.00G  11.8G     2.00G  /zdata/jambon/parme
zdata/jambon/pays               50.0M  11.8G     50.0M  /zdata/jambon/pays

We can delete the unencrypted version and mount the encrypted version to the previous location:

root@jambon-production-server:~ # /usr/local/sbin/zfs destroy -r zdata/jambon
root@jambon-production-server:~ # /usr/local/sbin/zfs set mountpoint=/zdata/jambon zdata/encrypted/jambon

Check that the data is still there:

root@jambon-production-server:~ # ls /zdata/jambon/parme/
jambons

Congratulations! You migrated your data to native ZFS encryption.

These commands were just examples. If for instance your data was already encrypted using GELI, you'll need to execute the same operation on another disk. Otherwise, you'll encrypt your data twice.

Also keep in mind that ZFS encryption keys are not loaded during the boot process.

For fun, let's close and open the data set again:
root@jambon-production-server:~ # /usr/local/sbin/zfs unmount zdata/encrypted
root@jambon-production-server:~ # /usr/local/sbin/zfs unload-key zdata/encrypted

root@jambon-production-server:~ # /usr/local/sbin/zfs load-key zdata/encrypted
root@jambon-production-server:~ # /usr/local/sbin/zfs mount -a
root@jambon-production-server:~ # ls /zdata/jambon/blanc/
jambons

Sending incremental snapshot to a backup server

Image

Let's say we have a server somewhere that we don't trust completely. It might get stolen, or you don't 100% trust its owner with your life. But on the other side, it has a lot of unused storage that you'd like to use to backup the production server.

That server already has a pool with the latest ZFS:

root@jambon-insecure-server:~ # uname -a
FreeBSD jambon-insecure-server 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  amd64
root@jambon-insecure-server:~ # /usr/local/sbin/zpool create zbackups da1
root@jambon-insecure-server:~ # /usr/local/sbin/zpool list
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zbackups  19.5G   120K  19.5G        -         -     0%     0%  1.00x    ONLINE  -
root@jambon-insecure-server:~ # /usr/local/sbin/zfs list
NAME       USED  AVAIL     REFER  MOUNTPOINT
zbackups   100K  18.9G       24K  /zbackups

Create the destination dataset on the backup server:

root@jambon-insecure-server:~ # /usr/local/sbin/zfs create zbackups/encrypted

Take a snapshot of the source dataset:

root@jambon-production-server:~ # /usr/local/sbin/zfs snapshot -r zdata/encrypted/jambon@to-remote-server

Send everything to the backup server:

root@jambon-production-server:~ # /usr/local/sbin/zfs send -Rw zdata/encrypted/jambon@to-remote-server \
     | ssh root@jambon-insecure-server '/usr/local/sbin/zfs receive zbackups/encrypted/jambon'
From the man page:
-w, --raw For encrypted datasets, send data exactly as it exists on disk. This allows backups to be taken even if encryption keys are not currently loaded. The backup may then be received on an untrusted machine since that machine will not have the encryption keys to read the protected data or alter it without being detected. Upon being received, the dataset will have the same encryption keys as it did on the send side, although the keylocation property will be defaulted to prompt if not otherwise provided. For unencrypted datasets, this flag will be equivalent to -Lec. Note that if you do not use this flag for sending encrypted datasets, data will be sent unencrypted and may be re-encrypted with a different encryption key on the receiving system, which will disable the ability to do a raw send to that system for incrementals.

Check that the data has been copied to the backup server:

root@jambon-insecure-server:~ # /usr/local/sbin/zfs list
NAME                                USED  AVAIL     REFER  MOUNTPOINT
zbackups                           3.55G  15.3G       24K  /zbackups
zbackups/encrypted                 3.55G  15.3G       24K  /zbackups/encrypted
zbackups/encrypted/jambon          3.55G  15.3G      100K  /zdata/jambon
zbackups/encrypted/jambon/bayonne   513M  15.3G      513M  /zdata/jambon/bayonne
zbackups/encrypted/jambon/blanc    1.00G  15.3G     1.00G  /zdata/jambon/blanc
zbackups/encrypted/jambon/parme    2.00G  15.3G     2.00G  /zdata/jambon/parme
zbackups/encrypted/jambon/pays     50.2M  15.3G     50.1M  /zdata/jambon/pays

root@jambon-insecure-server:~ # /usr/local/sbin/zfs list -t snapshot zbackups/encrypted/jambon/blanc
NAME                                               USED  AVAIL     REFER  MOUNTPOINT
zbackups/encrypted/jambon/blanc@2018                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@2019                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@2020                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@migration            8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@to-remote-server     0B      -     1.00G  -

Check that the data is still encrypted on the backup server:

root@jambon-insecure-server:~ # /usr/local/sbin/zfs get all zbackups/encrypted/jambon/blanc
[...]
zbackups/encrypted/jambon/blanc  encryption            aes-256-gcm                -
zbackups/encrypted/jambon/blanc  keylocation           none                       default
zbackups/encrypted/jambon/blanc  keyformat             raw                        -
zbackups/encrypted/jambon/blanc  pbkdf2iters           0                          default
zbackups/encrypted/jambon/blanc  encryptionroot        zbackups/encrypted/jambon  -
zbackups/encrypted/jambon/blanc  keystatus             unavailable                -

That's it, we just backuped an encrypted dataset to a server that doesn't have the key!

Let's try to change some data on the production server and only backup the changes using snapshots:

root@jambon-production-server:~ # dd if=/dev/zero of=/zdata/jambon/blanc/another-file bs=1M count=5
root@jambon-production-server:~ # /usr/local/sbin/zfs snapshot zdata/encrypted/jambon/blanc@work-of-the-day
root@jambon-production-server:~ # /usr/local/sbin/zfs send -w -i zdata/encrypted/jambon/blanc@to-remote-server zdata/encrypted/jambon/blanc@work-of-the-day  \
     | ssh root@jambon-insecure-server '/usr/local/sbin/zfs receive zbackups/encrypted/jambon/blanc'
root@jambon-insecure-server:~ # /usr/local/sbin/zfs list -t snapshot zbackups/encrypted/jambon/blanc
NAME                                               USED  AVAIL     REFER  MOUNTPOINT
zbackups/encrypted/jambon/blanc@2018                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@2019                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@2020                 8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@migration            8K      -     1.00G  -
zbackups/encrypted/jambon/blanc@to-remote-server    50K      -     1.00G  -
zbackups/encrypted/jambon/blanc@work-of-the-day      0B      -     1.01G  -

Done. We just backuped incremental changes between two ZFS snapshots to a server that never has and never will be able to decrypt the data.

In case of emergency, the key can be transferred to the backup server and the dataset can be opened:

root@jambon-production-server:~ # scp /etc/jambon-zfs-key root@jambon-insecure-server:/etc/jambon-zfs-key
root@jambon-insecure-server:~ # /usr/local/sbin/zfs set keylocation=file:///etc/jambon-zfs-key zbackups/encrypted/jambon
root@jambon-insecure-server:~ # /usr/local/sbin/zfs load-key zbackups/encrypted/jambon
root@jambon-insecure-server:~ # ls /zdata/jambon/blanc
another-file    jambons

Conclusion

We have seen that OpenZFS on FreeBSD allowed us to send encrypted snapshot-based incremental snapshots to an insecure backup server that never manipulates the encryption key.