Fail closed: when CI cannot compute the diff, evaluate everything

The deploy pipeline decides which hosts a change is about by diffing the push or the pull request. That decision is an inference, and every inference needs a defined answer for the case where it cannot be performed. The answer cannot be “no changes”. A diff that could not be computed and a diff that found nothing are the same empty string, and only one of them means the work is unnecessary.

The inference, and its precondition

The scope job’s computation is one line:

1
git diff --name-only "HEAD~1...HEAD"

That command requires at least two commits of history to be present in the checkout. A fresh Forgejo checkout defaults to a shallow depth of one, so on a merge the parent commit is simply not in the repository, and the command fails with fatal: Invalid symmetric difference expression ...HEAD rather than returning an empty list. Both workflows that diff therefore pin fetch-depth: 0 on their scope job’s checkout: the deploy workflow for pushes, the build-check workflow for pull requests, where the base commit is older still.

That pin is not decoration. The failure has been observed: the eval-giga job’s checkout had no fetch-depth: 0, so on PR events git diff "$BASE...HEAD" died with fatal: Invalid symmetric difference expression ...HEAD and the job reported giga in scope: false. The asymmetry is what made it durable — push events escaped it (depth 1 still contains HEAD~1, so they fell through to the “everything changed” branch), meaning only pull requests silently skipped giga, and a skipped host looks exactly like a host with nothing to do.

Why the error does not surface

The scope step is written as an if/elif chain, and the guard that survives an uncomputable diff is spelled out in the workflow’s own comment:

Fail CLOSED. Inside an elif condition set -e never fires, and the branch below pipes the diff into grep -q .: an uncomputable diff (base object absent, shallow history) therefore fell through to HOSTS="" — a green run that built and evaluated NOTHING. Scope must widen, never silently vanish.

Both halves are the point. set -e is the reflex answer, and it does nothing here: a command in a condition is exempt, so a failing diff selects a branch instead of aborting the run. And the pipeline is deceptive in the other direction — grep -q . exists to say “is there any output”, and against nothing it says no, which is indistinguishable from “there is nothing to see”.

The fallback widens, in both directions

The evaluator’s fix is the shape to copy:

1
2
3
elif ! git diff --name-only "$BASE...HEAD" -- . >/dev/null 2>&1; then
  echo "WARN: cannot diff against '$BASE' - evaluating all deployable hosts"
  HOSTS="mail public vpn"

And the push path carries the equivalent guard, testing the parent commit before the diff is attempted:

1
no previous tip available — deploying all hosts

Unreadable history scopes to every host, not to none. The cost of the fallback is real — a wider nix build and a few minutes of runner time — and that is the correct trade against a silent no-op, because the wide case is visible in the log and the narrow case is not.

Explicit input beats inference

The strongest version of fail-closed in this repository is the one that removes the inference entirely. A manual dispatch takes no checkout and computes no diff: the hosts input is authoritative, and it is required. An empty value aborts with

1
hosts input is EMPTY — refusing to deploy (name hosts explicitly: mail vpn public)

and an unrecognised host name aborts too, rather than being dropped from the scope. giga fails loudly with a pointer to its own workflow, instead of quietly deploying nothing. A required input whose empty case is an error cannot be mistaken for an empty diff — which is why that branch is the pattern worth copying into any script that currently infers scope, as opposed to one that is handed scope.

Two places an empty result and a failure look identical

set -o pipefail is present in these steps, and it is necessary but not sufficient: it makes a pipeline fail when the command on the left fails, and inside a condition that only changes which branch you take. It does not abort the run. pipefail plus an elif that widens is a fix; pipefail alone is decoration.

git diff --quiet is the second trap. It uses exit code 1 for “there are differences” and a different non-zero for “I could not compute this at all”, so a test written as if git diff --quiet; then ... else ... — the form used in flake-update.yml to decide whether flake.lock actually changed — cannot tell “identical” from “cannot diff”. In that direction it errs safely (toward “changed”), but only by accident. Anywhere the same construct gates work by testing for no change, it errs silently toward doing nothing.

What you can check

• Read the declared depth where it is declared. The scope job’s with: block is the authority on fetch-depth; do not infer it from the job list.
• Read the scope line. scoped hosts: '...' is the whole decision. If it prints an empty list when you know you touched a host, the scope step is broken, not the repository empty. Better: assert it non-empty in the step whenever work was expected.
• Confirm the fallback fired. If the log carries no previous tip available — deploying all hosts or cannot diff against '<base>', the run widened on purpose — that is the design working, not a diff that silently missed your change.

The scope line is the same one that makes a green run meaningful, which is where a green deploy that deployed nothing begins. The pipeline itself is documented in the infrastructure breakdown, and the review discipline that keeps these claims checkable is in the engineering standards.