Encrypted snapshot-based incremental backups with OpenZFS on FreeBSD 12.2
Encrypted backups everywhere!
Goal
Making a backup server which receives incremental updates but can never decrypt the data.
Introduction
FreeBSD is moving to OpenZFS. When it's released, version 13 will have moved to OpenZFS.
Right now it's possible to try OpenZFS alongside FreeBSD's ZFS distribution by building it from the port collection.
There is one feature I'm really interested in: encryption at rest. I always ensure that people who steal my hard drives can never access my data.
Encrypting ZFS pools has been possible for a while now. The most common way of doing it was with GELI. Basically, a new layer is added to the system, which exposes a new block device where ZFS can be used on top. This is working great, with great performance.
However, when moving datasets between systems, ZFS snapshots are still transmitted unencrypted, and the receiving server is responsible for encrypting and storing the data in a proper way. This is good enough if all part of the infrastructure is present and managed in a trusted location and all networks between the two are reliable.
With native ZFS encryption, OpenZFS moves the encryption to the actual file system implementation. This means that we can now use manipulate ZFS snapshots without ever handling unencrypted data, and so we can make a backup server at a location we don't fully trust.
Let's try it!
The source server
We're going to simulate a production server migrating to a newer release and migrating its data to the new ZFS.
We start from a standard 12.1-RELEASE server:
root@jambon-production-server:~ # uname -a FreeBSD jambon-production-server 12.1-RELEASE FreeBSD 12.1-RELEASE r354233 GENERIC amd64
The server has a pool full of sensitive, very important data:
root@jambon-production-server:~ # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zdata 19.5G 3.55G 15.9G - - 3% 18% 1.00x ONLINE - root@jambon-production-server:~ # zfs list NAME USED AVAIL REFER MOUNTPOINT zdata 3.50G 15.4G 23K /zdata zdata/jambon 3.50G 15.4G 24K /zdata/jambon zdata/jambon/bayonne 512M 15.4G 512M /zdata/jambon/bayonne zdata/jambon/blanc 1.00G 15.4G 1.00G /zdata/jambon/blanc zdata/jambon/parme 2.00G 15.4G 2.00G /zdata/jambon/parme zdata/jambon/pays 23K 15.4G 23K /zdata/jambon/pays
Here is the list of supported features:
root@jambon-production-server:~ # zpool upgrade -v
This system supports ZFS pool feature flags.
The following features are supported:
FEAT DESCRIPTION
-------------------------------------------------------------
async_destroy (read-only compatible)
Destroy filesystems asynchronously.
empty_bpobj (read-only compatible)
Snapshots use less space.
lz4_compress
LZ4 compression algorithm support.
multi_vdev_crash_dump
Crash dumps to multiple vdev pools.
spacemap_histogram (read-only compatible)
Spacemaps maintain space histograms.
enabled_txg (read-only compatible)
Record txg at which a feature is enabled
hole_birth
Retain hole birth txg for more precise zfs send
extensible_dataset
Enhanced dataset functionality, used by other features.
embedded_data
Blocks which compress very well use even less space.
bookmarks (read-only compatible)
"zfs bookmark" command
filesystem_limits (read-only compatible)
Filesystem and snapshot limits.
large_blocks
Support for blocks larger than 128KB.
large_dnode
Variable on-disk size of dnodes.
sha512
SHA-512/256 hash algorithm.
skein
Skein hash algorithm.
device_removal
Top-level vdevs can be removed, reducing logical pool size.
obsolete_counts (read-only compatible)
Reduce memory used by removed devices when their blocks are freed or remapped.
zpool_checkpoint (read-only compatible)
Pool state can be checkpointed, allowing rewind later.
spacemap_v2 (read-only compatible)
Space maps representing large segments are more efficient.
The following legacy versions are also supported:
VER DESCRIPTION
--- --------------------------------------------------------
1 Initial ZFS version
2 Ditto blocks (replicated metadata)
3 Hot spares and double parity RAID-Z
4 zpool history
5 Compression using the gzip algorithm
6 bootfs pool property
7 Separate intent log devices
8 Delegated administration
9 refquota and refreservation properties
10 Cache devices
11 Improved scrub performance
12 Snapshot properties
13 snapused property
14 passthrough-x aclinherit
15 user/group space accounting
16 stmf property support
17 Triple-parity RAID-Z
18 Snapshot user holds
19 Log device removal
20 Compression using zle (zero-length encoding)
21 Deduplication
22 Received properties
23 Slim ZIL
24 System attributes
25 Improved scrub stats
26 Improved snapshot deletion performance
27 Improved snapshot creation performance
28 Multiple vdev replacements
Yep, this is a good old FreeBSD ZFS pool.
Let's upgrade to 12.2-RELEASE:
root@jambon-production-server:~ # freebsd-update -r 12.2-RELEASE upgrade root@jambon-production-server:~ # freebsd-update install root@jambon-production-server:~ # reboot root@jambon-production-server:~ # freebsd-update install root@jambon-production-server:~ # uname -a FreeBSD jambon-production-server 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC amd64
Installing OpenZFS
OpenZFS is present in the port collection:
- sysutils/openzfs-kmod: The kernel module
- sysutils/openzfs: The command line utilities
Building the kernel module requires the source tree to be present. Download it:
root@jambon-production-server:~ # cd /usr/src root@jambon-production-server:~ # fetch 'http://ftp.fr.freebsd.org/pub/FreeBSD/releases/amd64/12.2-RELEASE/src.txz' root@jambon-production-server:~ # tar --strip-components 2 -x -J -f src.txz
Let's install them:
root@jambon-production-server:~ # portsnap fetch extract root@jambon-production-server:~ # make -C /usr/ports/sysutils/openzfs-kmod install clean root@jambon-production-server:~ # make -C /usr/ports/sysutils/openzfs install clean root@jambon-production-server:~ # pkg info | grep openzfs openzfs-2020102700 OpenZFS userland for FreeBSD openzfs-kmod-2020102700 OpenZFS kernel module for FreeBSD
Now, here comes the part where it's hard to not mix up commands.
- The OS ZFS binaries are stored in
/sbin - OpenZFS binaries are stored in
/usr/local/sbin
This means that if you don't prefix your commands you might run the wrong version. OpenZFS pools need to be manipulated with /usr/local/sbin/zpool and /usr/local/sbin/zfs.
We can check that the new ZFS has more features:
root@jambon-production-server:~ # /usr/local/sbin/zpool upgrade -v
FEAT DESCRIPTION
-------------------------------------------------------------
[...]
encryption
Support for dataset level encryption
[...]
Migrating and encrypting the pool
First, we need to upgrade the pool. This is a one way operation. The pool cannot be downgraded to its previous version.
root@jambon-production-server:~ # /usr/local/sbin/zpool upgrade zdata This system supports ZFS pool feature flags. Enabled the following features on 'zdata': userobj_accounting encryption project_quota allocation_classes resilver_defer bookmark_v2 redaction_bookmarks redacted_datasets bookmark_written log_spacemap livelist device_rebuild zstd_compress
Unfortunately, it's not possible to encrypt an existing dataset. The encryption flag can only be set to datasets when they're created (-o encryption is read only). However, we can copy the data (including the snapshots) to new encrypted datasets.
First, let's create a key:
root@jambon-production-server:~ # dd if=/dev/random of=/etc/jambon-zfs-key bs=1 count=32
Don't forget to copy the key to your backup infrastructure before storing actual data on the pool. If you lose the key, you won't be able to unlock the pool and access the data anymore.
Create an encrypted dataset:
root@jambon-production-server:~ # /usr/local/sbin/zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///etc/jambon-zfs-key zdata/encrypted
Check that the dataset is encrypted:
root@jambon-production-server:~ # /usr/local/sbin/zfs get all zdata/encrypted NAME PROPERTY VALUE SOURCE [...] zdata/encrypted encryption aes-256-gcm - zdata/encrypted keylocation file:///etc/jambon-zfs-key local zdata/encrypted keyformat raw -
List the snapshots of the source datasets:
root@jambon-production-server:~ # /usr/local/sbin/zfs list -t snapshot -o name zdata/jambon/blanc NAME zdata/jambon/blanc@2018 zdata/jambon/blanc@2019 zdata/jambon/blanc@2020 zdata/jambon/blanc@migration
Copy the first snapshot:
root@jambon-production-server:~ # /usr/local/sbin/zfs create zdata/encrypted/jambon
root@jambon-production-server:~ # /usr/local/sbin/zfs send -R zdata/jambon/blanc@2018 \
| /usr/local/sbin/zfs recv -x encryption zdata/encrypted/jambon/blanc
That's it: the dataset is now encrypted. It got its encryption parameters from the parent dataset we created earlier:
root@jambon-production-server:~ # /usr/local/sbin/zfs get all zdata/encrypted/jambon/blanc [...] zdata/encrypted/jambon/blanc encryption aes-256-gcm - zdata/encrypted/jambon/blanc keylocation none default zdata/encrypted/jambon/blanc keyformat raw - zdata/encrypted/jambon/blanc pbkdf2iters 0 default zdata/encrypted/jambon/blanc encryptionroot zdata/encrypted - zdata/encrypted/jambon/blanc keystatus available -
Copy the other snapshots:
root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2018 zdata/jambon/blanc@2019 \
| /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc
root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2019 zdata/jambon/blanc@2020 \
| /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc
root@jambon-production-server:~ # /usr/local/sbin/zfs send -i zdata/jambon/blanc@2020 zdata/jambon/blanc@migration \
| /usr/local/sbin/zfs recv -F zdata/encrypted/jambon/blanc
root@jambon-production-server:~ # /usr/local/sbin/zfs list -t snapshot zdata/encrypted/jambon/blanc
NAME USED AVAIL REFER MOUNTPOINT
zdata/encrypted/jambon/blanc@2018 xGB - x.00G -
zdata/encrypted/jambon/blanc@2019 xGB - x.00G -
zdata/encrypted/jambon/blanc@2020 xGB - x.00G -
zdata/encrypted/jambon/blanc@migration xGB - x.00G -
Now we have the data twice: once unencrypted, and once encrypted:
root@jambon-production-server:~ # /usr/local/sbin/zfs list NAME USED AVAIL REFER MOUNTPOINT zdata 7.10G 11.8G 23K /zdata zdata/encrypted 3.55G 11.8G 100K /zdata/encrypted zdata/encrypted/jambon 3.55G 11.8G 103K /zdata/encrypted/jambon zdata/encrypted/jambon/bayonne 513M 11.8G 513M /zdata/encrypted/jambon/bayonne zdata/encrypted/jambon/blanc 1.00G 11.8G 1.00G /zdata/encrypted/jambon/blanc zdata/encrypted/jambon/parme 2.00G 11.8G 2.00G /zdata/encrypted/jambon/parme zdata/encrypted/jambon/pays 50.2M 11.8G 50.1M /zdata/encrypted/jambon/pays zdata/jambon 3.55G 11.8G 24K /zdata/jambon zdata/jambon/bayonne 512M 11.8G 512M /zdata/jambon/bayonne zdata/jambon/blanc 1.00G 11.8G 1.00G /zdata/jambon/blanc zdata/jambon/parme 2.00G 11.8G 2.00G /zdata/jambon/parme zdata/jambon/pays 50.0M 11.8G 50.0M /zdata/jambon/pays
We can delete the unencrypted version and mount the encrypted version to the previous location:
root@jambon-production-server:~ # /usr/local/sbin/zfs destroy -r zdata/jambon root@jambon-production-server:~ # /usr/local/sbin/zfs set mountpoint=/zdata/jambon zdata/encrypted/jambon
Check that the data is still there:
root@jambon-production-server:~ # ls /zdata/jambon/parme/ jambons
Congratulations! You migrated your data to native ZFS encryption.
These commands were just examples. If for instance your data was already encrypted using GELI, you'll need to execute the same operation on another disk. Otherwise, you'll encrypt your data twice.
Also keep in mind that ZFS encryption keys are not loaded during the boot process.
For fun, let's close and open the data set again:root@jambon-production-server:~ # /usr/local/sbin/zfs unmount zdata/encrypted root@jambon-production-server:~ # /usr/local/sbin/zfs unload-key zdata/encrypted root@jambon-production-server:~ # /usr/local/sbin/zfs load-key zdata/encrypted root@jambon-production-server:~ # /usr/local/sbin/zfs mount -a root@jambon-production-server:~ # ls /zdata/jambon/blanc/ jambons
Sending incremental snapshot to a backup server
Let's say we have a server somewhere that we don't trust completely. It might get stolen, or you don't 100% trust its owner with your life. But on the other side, it has a lot of unused storage that you'd like to use to backup the production server.
That server already has a pool with the latest ZFS:
root@jambon-insecure-server:~ # uname -a FreeBSD jambon-insecure-server 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC amd64
root@jambon-insecure-server:~ # /usr/local/sbin/zpool create zbackups da1 root@jambon-insecure-server:~ # /usr/local/sbin/zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT zbackups 19.5G 120K 19.5G - - 0% 0% 1.00x ONLINE - root@jambon-insecure-server:~ # /usr/local/sbin/zfs list NAME USED AVAIL REFER MOUNTPOINT zbackups 100K 18.9G 24K /zbackups
Create the destination dataset on the backup server:
root@jambon-insecure-server:~ # /usr/local/sbin/zfs create zbackups/encrypted
Take a snapshot of the source dataset:
root@jambon-production-server:~ # /usr/local/sbin/zfs snapshot -r zdata/encrypted/jambon@to-remote-server
Send everything to the backup server:
root@jambon-production-server:~ # /usr/local/sbin/zfs send -Rw zdata/encrypted/jambon@to-remote-server \
| ssh root@jambon-insecure-server '/usr/local/sbin/zfs receive zbackups/encrypted/jambon'
From the man page:
-w, --raw For encrypted datasets, send data exactly as it exists on disk. This allows backups to be taken even if encryption keys are not currently loaded. The backup may then be received on an untrusted machine since that machine will not have the encryption keys to read the protected data or alter it without being detected. Upon being received, the dataset will have the same encryption keys as it did on the send side, although the keylocation property will be defaulted to prompt if not otherwise provided. For unencrypted datasets, this flag will be equivalent to -Lec. Note that if you do not use this flag for sending encrypted datasets, data will be sent unencrypted and may be re-encrypted with a different encryption key on the receiving system, which will disable the ability to do a raw send to that system for incrementals.
Check that the data has been copied to the backup server:
root@jambon-insecure-server:~ # /usr/local/sbin/zfs list NAME USED AVAIL REFER MOUNTPOINT zbackups 3.55G 15.3G 24K /zbackups zbackups/encrypted 3.55G 15.3G 24K /zbackups/encrypted zbackups/encrypted/jambon 3.55G 15.3G 100K /zdata/jambon zbackups/encrypted/jambon/bayonne 513M 15.3G 513M /zdata/jambon/bayonne zbackups/encrypted/jambon/blanc 1.00G 15.3G 1.00G /zdata/jambon/blanc zbackups/encrypted/jambon/parme 2.00G 15.3G 2.00G /zdata/jambon/parme zbackups/encrypted/jambon/pays 50.2M 15.3G 50.1M /zdata/jambon/pays root@jambon-insecure-server:~ # /usr/local/sbin/zfs list -t snapshot zbackups/encrypted/jambon/blanc NAME USED AVAIL REFER MOUNTPOINT zbackups/encrypted/jambon/blanc@2018 8K - 1.00G - zbackups/encrypted/jambon/blanc@2019 8K - 1.00G - zbackups/encrypted/jambon/blanc@2020 8K - 1.00G - zbackups/encrypted/jambon/blanc@migration 8K - 1.00G - zbackups/encrypted/jambon/blanc@to-remote-server 0B - 1.00G -
Check that the data is still encrypted on the backup server:
root@jambon-insecure-server:~ # /usr/local/sbin/zfs get all zbackups/encrypted/jambon/blanc [...] zbackups/encrypted/jambon/blanc encryption aes-256-gcm - zbackups/encrypted/jambon/blanc keylocation none default zbackups/encrypted/jambon/blanc keyformat raw - zbackups/encrypted/jambon/blanc pbkdf2iters 0 default zbackups/encrypted/jambon/blanc encryptionroot zbackups/encrypted/jambon - zbackups/encrypted/jambon/blanc keystatus unavailable -
That's it, we just backuped an encrypted dataset to a server that doesn't have the key!
Let's try to change some data on the production server and only backup the changes using snapshots:
root@jambon-production-server:~ # dd if=/dev/zero of=/zdata/jambon/blanc/another-file bs=1M count=5
root@jambon-production-server:~ # /usr/local/sbin/zfs snapshot zdata/encrypted/jambon/blanc@work-of-the-day
root@jambon-production-server:~ # /usr/local/sbin/zfs send -w -i zdata/encrypted/jambon/blanc@to-remote-server zdata/encrypted/jambon/blanc@work-of-the-day \
| ssh root@jambon-insecure-server '/usr/local/sbin/zfs receive zbackups/encrypted/jambon/blanc'
root@jambon-insecure-server:~ # /usr/local/sbin/zfs list -t snapshot zbackups/encrypted/jambon/blanc
NAME USED AVAIL REFER MOUNTPOINT
zbackups/encrypted/jambon/blanc@2018 8K - 1.00G -
zbackups/encrypted/jambon/blanc@2019 8K - 1.00G -
zbackups/encrypted/jambon/blanc@2020 8K - 1.00G -
zbackups/encrypted/jambon/blanc@migration 8K - 1.00G -
zbackups/encrypted/jambon/blanc@to-remote-server 50K - 1.00G -
zbackups/encrypted/jambon/blanc@work-of-the-day 0B - 1.01G -
Done. We just backuped incremental changes between two ZFS snapshots to a server that never has and never will be able to decrypt the data.
In case of emergency, the key can be transferred to the backup server and the dataset can be opened:
root@jambon-production-server:~ # scp /etc/jambon-zfs-key root@jambon-insecure-server:/etc/jambon-zfs-key root@jambon-insecure-server:~ # /usr/local/sbin/zfs set keylocation=file:///etc/jambon-zfs-key zbackups/encrypted/jambon root@jambon-insecure-server:~ # /usr/local/sbin/zfs load-key zbackups/encrypted/jambon root@jambon-insecure-server:~ # ls /zdata/jambon/blanc another-file jambons