Why the key that decrypts everything is not in the backup

A backup is a claim about the future: that on the day it matters, the bytes come back. The interesting failures are not disk failures — they are dependency cycles you built yourself and did not notice.

The repository password is a secret

This fleet backs up to Cloudflare R2 with restic: file state every 15 minutes with a 90-second jitter, the database dump daily (since 2026-09-29 in a unit of its own, so it is no longer regenerated inside every backup run). The module is modules/fleet-backup.nix, and its header states the intent in one line:

1
2
# Repo passphrase is `restic-passwd` in sops (shared fleet-wide, on purpose:
# any host must be able to restore any other host's repo during recovery).

Two consequences follow immediately. The passphrase is not per-host, so it cannot be per-host-recoverable. And it is a sops secret, so reading it requires an age identity — /var/lib/sops-nix/keys.txt, root-owned 0600, plaintext on disk.

The recovery path is therefore: to read the backup you need the passphrase, and to read the passphrase you need the identity.

Read that as a cycle check

Both ends of that chain are things you could carelessly store inside the backup. The module refuses both:

1
2
3
4
5
# NOTE on /var/lib/sops-nix: this dir holds the host's age identity
# (/var/lib/sops-nix/keys.txt, root:root 0600) which decrypts secrets.yaml.
# It is PLAINTEXT on disk, so it is deliberately NOT backed up raw here:
# the restic passphrase is shared fleet-wide, so a raw copy would place an
# unencrypted private key in R2, readable by any host holding that passphrase.

That is the whole argument in four sentences. Because the passphrase is deliberately shared, a raw copy of the identity would be readable by anything that can read the archive — the backup would contain the key to the backup. The path set is /var/lib/acme, /var/lib/caddy, /etc/nix/netrc and /var/lib/hermes, plus a pg_dumpall on Postgres hosts, and /var/lib/sops-nix is not in it. The raw Postgres data directory is excluded too, because it is version-fragile; the dump is authoritative.

The identity cannot live only on the hosts either. Each host generates its own age key on itself and the private half never leaves the machine; sops.age.sshKeyPaths = [ ] on every host, so the ssh host key is no longer convertible into an age identity. A terminated host therefore cannot supply its own key.

Escrow, and each file’s deliberate omission

The answer is in .sops.yaml, as a second family of creation rules:

1
2
3
4
5
# Disaster-recovery escrow of each host's age identity. Owner + operator
# only: no host can unwrap another host's key.
- path_regex: secrets/age-escrow/[^/]+\.age$
  key_groups:
  - age: [*admin_truva, *hermes_agent]

One rule, no per-host exclusions to get wrong — and the recipient list got shorter rather than longer, deliberately. The first version of this escrow encrypted each host’s identity to every other host, on the reasoning that the host which dies cannot unwrap its own key, so a surviving peer should be able to. That reasoning builds a cross-host escalation path: any compromised host can unwrap every other host’s identity. The escrow is now readable by the owner and the operator only, and the fleet was re-keyed on 2026-09-23 so that the peer-encrypted ciphertext is retired rather than merely unused — the identities it belongs to exist nowhere in the fleet, and no host is a recipient of any escrow blob.

Note where the escrow lives: in the private configuration repository as sops blobs under secrets/age-escrow/<host>.age, so it is recoverable from git alone, without depending on R2 — the system that would be the problem in the scenario the escrow exists for. The one piece that is genuinely human-held is the owner identity, which decrypts the store itself.

The tension is worth saying out loud rather than smoothing over: escrowing a key moves risk from the archive to wherever the escrow lives. A sops blob is safe only because of the recipients listed in it, and a recipient list of two people means the store’s availability rests on those two people. That location needs the highest protection in the system, and it is not the backup server.

A backup you have never restored

The only claim worth making about a backup is a restore you have actually performed. docs/disaster-recovery.md in nix-config-private is written as a runbook — provision the instance, restore the identity, apply the flake, restore state, rejoin the tunnel, repoint DNS, verify — with a recovery point of 15 minutes for file state and a day for the database dump, and a recovery-time target under an hour for a full host rebuild. It also records what is deliberately not backed up, which is the part most runbooks leave out.

It ends with a block titled Verify this before you need it, and the commands are the point:

1
2
sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age >/dev/null && echo escrow-ok
age-keygen -y <(sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age)

The printed public key must equal that host’s anchor in .sops.yaml, and it has to be run where the owner or operator identity is present: hosts are no longer recipients of any escrow blob, including their own. That is a two-line test of the step the runbook labels “WITHOUT THIS NOTHING WORKS”, and it is worth running on a schedule — because a runbook nobody has exercised is documentation of an intention, and a backup system’s documentation is a restore runbook plus the date it was last exercised.

As of 2026-09-27 there is a date to quote: repository integrity checked on all three hosts, and a file-level restore performed on each, with the results recorded in the backup and recovery statement. There is now a second date for the harder half. On 2026-09-28 both database hosts were restored from the physical repository and recovered to a chosen point in time — the restore that matters when a record is wrong rather than when a host is gone — and its measured durations are recorded in the same place. What neither drill shows — a full host rebuild — is stated in the same place, next to them.

What you can check

• The cycle, in the module. nix-config/modules/fleet-backup.nix: the paths = [ ... ] list, and the comment explaining why /var/lib/sops-nix is absent from it.
• The escrow rules. Not publicly checkable: they live in nix-config-private/.sops.yaml, and the blobs under secrets/age-escrow/ are in the same private repository. What the public repository carries is each host’s public identity anchor in .sops.yaml, which is what the verify block compares against.
• That the escrow is testable today. The verify block in docs/disaster-recovery.md, run as the owner or the operator.

For the sibling problem — a status you can inspect versus one you have to trust — the green deploy that deployed nothing makes the same argument about a CI badge. The rest of the host configuration is described in how the fleet is built.