A secret rotation that merged green and reached no host

A credential rotation merges to main, and the run list stays empty. Not a red run — no run. There is nothing to misread, because there is nothing there.

That is a different silent failure from the one where a run goes green and deploys nothing. The fleet deploy has two independent gates between a merge and a host, and they answer two different questions.

One filter decides whether a run exists at all

NixOS Deploy is triggered by a paths: filter (.forgejo/workflows/nix-deploy.yml in the nix-config repository):

1
2
3
4
5
6
7
8
9
  push:
    branches: [main]
    paths:
      - 'flake.nix'
      - 'flake.lock'
      - 'hosts/**'
      - 'modules/**'
      - 'overlays/**'
      - 'secrets/**'

That list is not scoping. It is creation. A merge touching none of those paths produces no workflow run at all — and a missing run is not a pending run. Waiting for it never resolves; there is no job that will eventually appear.

The comment directly above the block exists because of exactly this, and it is worth reading in full:

secrets/** is host-relevant, not docs-like: the sops store is SHARED code (one file, every host’s defaultSopsFile), and the scope step below maps secrets/ to ALL hosts. Without it here, a secrets-only merge is a silent no-op: no run is created, and even a forced run would scope to zero hosts.

Two halves of one sentence, and they are the whole post. No run is created is the filter. Even a forced run would scope to zero hosts is the second gate, and it fails independently — which is why a rotation bug has two chances to be invisible.

The other gate decides which hosts

Once a run exists, its first job computes which hosts the run is about, and the whole output of that computation is one line:

1
scoped hosts: ' public'

The shared-code branch of that computation is an explicit list:

1
git diff --name-only "HEAD~1...HEAD" -- flake.nix flake.lock modules/ overlays/ hosts/common/ secrets/ | grep -q .

A hit there scopes every host, with one measured exception: a flake.lock-only bump that moves just a rev/narHash scopes to vpn alone, and the guard for that path additionally requires that no input was added, removed or renamed. A path under hosts/<host>/ scopes just that host. Anything else scopes nothing, and with an empty scope each host job’s if: contains(needs.scope.outputs.hosts, 'mail') is false, every deploy job is skipped, and the run concludes green. That mechanism — a green run that deployed nothing — is the subject of its own post. It is worth keeping the three apart:

Two lists, and they are not the same list

This is the part that makes the class of bug recur. The paths: filter carries hosts/**; the scope’s shared-code list carries hosts/common/. That difference is deliberate and correct — a change to one host’s directory should reach that host, not the fleet. Each list answers its own question, so copying one into the other is not a fix, and adding a directory to only one of them is a partial fix whose remaining half fails silently.

The rule that follows: when a new top-level directory starts being consumed by hosts, it needs an entry in both places, and the second entry is the one that is easy to forget because the first one makes CI look alive. The same class shows up one level down. The deploy workflow’s filter deliberately names config paths only — no .forgejo/workflows/** — so a workflow-only merge does not auto-deploy, and a workflow change is validated by a manual dispatch instead. The build-check workflow’s filter does include workflows, because it has to be able to validate its own changes. And a script under .forgejo/scripts/** that a workflow step calls needs that path in the calling workflow’s filter, or editing the script runs nothing.

The escape hatch, and why it is not the fix

workflow_dispatch is not subject to paths: filters at all. A manual dispatch always runs, and its hosts input is authoritative — that job takes no checkout and computes no diff, so nothing can scope it down. That makes dispatch a reliable way to deploy a host on purpose, and it is why the forced run case in the comment is still recoverable.

It is not a fix for a missing filter entry, because the failure mode is ignorance: the rotation merged, nothing went red, and nobody knows there is work to do. An escape hatch only helps the person who already suspects.

What you can check

• Reconcile the lists. Read the on: block and the scope job’s shared-code list, then list the directories the hosts actually reference. Every shared one must appear in both.
• Treat a missing run as a result, not a delay. If a push touched a host-relevant file and no run exists a few seconds later, the filter decided that — it is not “still starting”. Match the diff against the filter before waiting for anything.
• If you must deploy now, dispatch. A manual dispatch bypasses the whole question, and names its hosts explicitly.

The same discipline as everywhere else on this site: the deploy pipeline is only as trustworthy as the claim you can independently re-derive from the files it runs from.