A rollback that cannot execute is worse than no rollback

A rollback is the code path that runs when something is already wrong. That is the whole difficulty: it executes in the exact window where you have the least appetite and the fewest options for debugging it, on a host you have just made worse. An untested rollback is therefore not a rollback with a caveat. It is a promise about a command that has never run, attached to a moment when you cannot check.

What the gate does

Each host’s deploy job switches, then runs a health gate. In order: poll the gated services every second for up to about ten seconds (a fixed sleep 40 paid the full cost on every deploy; a fixed short sleep rolls back a service that is merely still activating, because systemctl is-active reports activating as not active), check for failed units, require every declared critical service active, and compare against a pre-switch baseline of long-running enabled services. Any violation rolls back to the previous generation and ends the step with exit 1.

The failure that looks like a result

The runner’s sudo is allow-listed to exactly one command. From the workflow’s own comment on the rollback step:

The runner is allowlisted for exactly ONE privileged command - nixos-rebuild (see security.sudo.extraRules on the host). Everything else (sudo readlink, switch-to-configuration, nix-env) prompts for a password with no TTY, and a REAL gate violation then leaves the host exactly as broken as the gate found it.

A denied sudo is not a crash. It writes its own line to stderr and exits non-zero — and if the step captured its stdout in a command substitution, the substitution is what decides the story. There is no error to propagate: the assignment succeeded, and its value is empty. set -e does not fire, because nothing failed in the sense the shell tracks.

The observed output was this:

1
no previous generation ... cannot roll back

with empty generation variables — because the sudo readlink that produced those numbers had been denied. A permissions failure wearing a “no previous generation” costume. The host stayed on the generation the gate had just rejected, the run went red with a misleading cause, and the log said nothing about a password prompt.

The version in the repository, and its guards

The step reads the profile link without sudo — /nix/var/nix/profiles/system is world-readable, and requiring privilege to read it is what emptied the variables — and then refuses to guess:

1
2
3
4
5
6
CUR_LINK=$(readlink /nix/var/nix/profiles/system)
CURNUM=$(printf '%s' "$CUR_LINK" | sed 's|.*system-\([0-9]*\)-link|\1|')
if ! printf '%s' "$CURNUM" | grep -qE '^[0-9]+$'; then
  echo "!!! cannot read a generation number out of '$CUR_LINK' - refusing to guess a rollback target"
  CURNUM=""
fi

Three guards follow, and each one is a failure that was seen before it was fixed:

• Parse, then validate. The profile link is relative (system-474-link), so a pattern expecting a leading / substitutes nothing, the number stays a non-numeric string, and a numeric filter degrades into a string compare that matches every link — at which point sort -n | tail -1 hands back the generation the gate just rejected. That happened: the rollback re-activated the broken generation, twice, and still recorded ROLLBACK_DONE=1. A deliberate gate drill on giga (run 1649) is what exposed it, and the run’s own log did not distinguish it from a success.
• Refuse a target equal to what failed. If the “previous” closure resolves to the closure that just failed, the step clears the target rather than running a rollback that would change nothing.
• Verify where the profile actually landed. ROLLBACK_DONE=1 is set only when the profile reads back at the exact closure that was aimed at. Otherwise the log says !!! ROLLBACK DID NOT LAND ON THE PREVIOUS GENERATION and HOST NEEDS AN OPERATOR NOW.

Non-interactive, or it is not a check

The one place the step invokes a privileged command that is not on the allowlist is the journal dump for failed units, and it is written to fail explicitly:

1
2
sudo -n journalctl -u "$u" -n 40 --no-pager 2>&1 | mask ||
  echo "(journal not readable without privilege - read it on the host)"

-n is the load-bearing flag. Without it, sudo prints its password lecture, and that lecture is what filled the section on 2026-09-16 while the actual cause stayed hidden — a wall of text that reads like an error and is not one. The pattern generalises: invoke an allow-listed (or possibly-denied) sudo with -n, check its exit status, and say plainly which of “not permitted” and “not found” you are looking at.

Loud, because quiet is the expensive option

The rollback must abort the run and name the host, not absorb the problem. Failures here are not paged — the webhook and its spool-draining crons were removed from this pipeline on 2026-09-16 and the operator reads the runs himself — so the run log is the alert channel, which is one more reason an ambiguous line costs hours.

What you can check

• Read the gate step’s own lines, not the job summary: rollback verified: the profile is on <closure>, or !!! ROLLBACK DID NOT LAND, or no usable previous generation - CANNOT ROLL BACK. A rollback claim that does not name the closure it landed on is not a rollback.
• Check every privileged call in the step against the host’s sudo allowlist. On mail and vpn the allowlist is the nixos-rebuild binary alone.
• Re-test by dispatching a deploy, not by trusting a green run — a green run cannot distinguish a working gate from a disarmed one. The rollback only executes when something is already wrong, which is the worst place for a command that has never been executed.

The mechanics sit in the deploy workflow documented in the infrastructure breakdown, which is also where to look for how the fleet’s other gates avoid the same trap.