Why the key that decrypts everything is not in the backup

A backup is a claim about the future: that on the day it matters, the bytes come back. The interesting failures are not disk failures — they are dependency cycles you built yourself and did not notice.

The repository password is a secret

This fleet backs up to Cloudflare R2 with restic, hourly, with a ten-minute jitter. The module is modules/fleet-backup.nix, and its header states the intent in one line:

1
2
# Repo passphrase is `restic-passwd` in sops (shared fleet-wide, on purpose:
# any host must be able to restore any other host's repo during recovery).

Two consequences follow immediately. The passphrase is not per-host, so it cannot be per-host-recoverable. And it is a sops secret, so reading it requires an age identity — /var/lib/sops-nix/keys.txt, root-owned 0600, plaintext on disk.

The recovery path is therefore: to read the backup you need the passphrase, and to read the passphrase you need the identity.

Read that as a cycle check

Both ends of that chain are things you could carelessly store inside the backup. The module refuses both:

1
2
3
4
5
# NOTE on /var/lib/sops-nix: this dir holds the host's age identity
# (/var/lib/sops-nix/keys.txt, root:root 0600) which decrypts secrets.yaml.
# It is PLAINTEXT on disk, so it is deliberately NOT backed up raw here:
# the restic passphrase is shared fleet-wide, so a raw copy would place an
# unencrypted private key in R2, readable by any host holding that passphrase.

That is the whole argument in four sentences. Because the passphrase is deliberately shared, a raw copy of the identity would be readable by anything that can read the archive — the backup would contain the key to the backup. The path set is /var/lib/acme, /var/lib/caddy, /etc/nix/netrc and /var/lib/hermes, plus a pg_dumpall on Postgres hosts, and /var/lib/sops-nix is not in it. The raw Postgres data directory is excluded too, because it is version-fragile; the dump is authoritative.

The identity cannot live only on the hosts either. Each host generates its own age key on itself and the private half never leaves the machine; sops.age.sshKeyPaths = [ ] on every host, so the ssh host key is no longer convertible into an age identity. A terminated host therefore cannot supply its own key.

Escrow, and each file’s deliberate omission

The answer is in .sops.yaml, as a second family of creation rules:

1
2
3
- path_regex: secrets/age-escrow/vpn\.age$
  key_groups:
  - age: [*admin_truva, *hermes_agent, *giga, *mail, *public]

Each host’s escrow lists every other recipient and omits the host itself. That is not tidiness, it is purpose-built: the host that dies cannot unwrap its own key, so a surviving peer or the owner must be able to. The comment above those rules names the case as “Oracle terminated the instance”. Recipient sets are spelled out literally, because one anchor list cannot express four different exclusions.

Note where the escrow lives: in the public repository as sops blobs under secrets/age-escrow/<host>.age, so it is recoverable from git alone, without depending on R2 — the system that would be the problem in the scenario the escrow exists for. The one piece that is genuinely human-held is the owner identity, which decrypts secrets.yaml itself.

The tension is worth saying out loud rather than smoothing over: escrowing a key moves risk from the archive to wherever the escrow lives. A public-repo sops blob is safe only because of the recipients listed in it. That location needs the highest protection in the system, and it is not the backup server.

A backup you have never restored

The only claim worth making about a backup is a restore you have actually performed. docs/disaster-recovery.md in nix-config-private is written as a runbook — provision the instance, restore the identity, apply the flake, restore state, rejoin the tunnel, repoint DNS, verify — with an RPO of about an hour and an RTO under an hour for a full host rebuild. It also records what is deliberately not backed up, which is the part most runbooks leave out.

It ends with a block titled Verify this before you need it, and the commands are the point:

1
2
sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age >/dev/null && echo escrow-ok
age-keygen -y <(sops -d --input-type binary --output-type binary secrets/age-escrow/vpn.age)

The printed public key must equal that host’s anchor in .sops.yaml. That is a two-line test of the step the runbook labels “WITHOUT THIS NOTHING WORKS”, and it is worth running on a schedule — because a runbook nobody has exercised is documentation of an intention, and a backup system’s documentation is a restore runbook plus the date it was last exercised.

What you can check

• The cycle, in the module. nix-config/modules/fleet-backup.nix: the paths = [ ... ] list, and the comment explaining why /var/lib/sops-nix is absent from it.
• The escrow rules. nix-config/.sops.yaml: four path_regex: secrets/age-escrow/<host>\.age$ rules, each omitting its own host.
• That the escrow is testable today. The verify block in docs/disaster-recovery.md, run as a recipient on any surviving host.

For the sibling problem — a status you can inspect versus one you have to trust — the green deploy that deployed nothing makes the same argument about a CI badge. The rest of the host configuration is described in how the fleet is built.