A job that waits must not hold the slot it is waiting for

A failed job announces itself: it goes red, names a step, prints a reason. A hung job does none of that. It sits in running holding a runner slot, waiting jobs pile up behind it, and nothing in the pipeline says anything is wrong — from the pipeline’s point of view, work is in progress. What follows is the structural rule that produces it, and the parts of this fleet’s deploy graph that embody it.

One daemon, one job

The capacity of this fleet is not a policy setting; it is a property of the runner software. One forgejo-runner daemon serves one job at a time — that is the tool’s own default, and nothing in the repo raises it. Grepping the fleet configuration tells the story plainly:

1
grep -rn 'capacity\|instances\|runner' nix-config/modules nix-config/hosts | head -40

There is no capacity setting anywhere in nix-config/modules or nix-config/hosts; the only hits are unrelated (an inference-server flag and a Grafana alert comment). What the configuration sets is services.gitea-actions-runner.instances.ymrtech — one instance per host — plus the labels it advertises. The labels differ per host; the count never does. One daemon, one slot.

The workflow comment says it in as many words. deploy-rev’s own header in .forgejo/workflows/nix-deploy.yml explains why that job moved off ci: on the shared ci pool it sat “behind mail’s single capacity-1 slot”, and after a lock-bump merge it measured 3 min 10 s of pure queueing for about two seconds of git work. A label is a matcher, not capacity.

The graph, and the one edge that is a real dependency

.forgejo/workflows/nix-deploy.yml fans out into three per-host deploy jobs:

• deploy-mail — runs-on: [deploy-mail], needs: [scope, deploy-rev], gated if: contains(needs.scope.outputs.hosts, 'mail').
• deploy-vpn — runs-on: [deploy-vpn], needs: [scope, deploy-rev], gated on 'vpn'.
• deploy-public — runs-on: [deploy], needs: [scope, deploy-rev, deploy-vpn], gated on 'public'.

The deploy jobs are per-host and independent by design, so a single-host change does not drag the other hosts’ closures along. There is one ordering edge, and it is not a preference. Each client DNS tier splits its halves across vpn and public — primary on vpn, standby on public — so if both jobs ran concurrently one run could bounce both halves in the same second and take every client’s resolver down. deploy-public therefore waits for deploy-vpn: a real dependency earns an edge.

It also has a price. Each job builds and switches a whole NixOS system, and each demands a runner of its own, so an ordering edge widens the critical path and multiplies the slots the run needs in sequence, not in parallel: the vpn slot must free before the public slot is asked for. That is fine here because the dependency is real. Applied loosely — an edge added for tidiness, or to make logs read better — it buys a longer deploy and a larger pool for nothing.

The rule

A job must never hold the resource it is waiting for.

Concretely, for a bounded pool: if a job occupies a slot while polling for work that also needs a slot, the queue cannot drain. Nothing fails; the waiter waits, the awaited job queues behind it, and the run reports running until the job’s own timeout expires — the v15 API has no run-cancel endpoint at all, so a stuck run holds the capacity-1 runner for its whole timeout-minutes. This fleet has hit that shape and written the rule down: a chain job held a ci slot while polling for the scope job of the build check it had triggered, so it queued behind its own CI. The fix was not a longer timeout — it was to move the waiter to a lane the awaited job can still reach, and to poll the awaited job’s per-job commit status (one small call) instead of a multi-megabyte task listing computed inside the held slot.

Two rules follow from the same accounting: a dependency widens the critical path and multiplies slot demand, so it needs a reason; and because a hung run looks nothing like a failed one, queue depth is worth watching on its own. A run running far longer than normal, with other jobs waiting behind it, is the signature. Do not wait for a red mark that will never come.

What you can check

• The slot count. grep -rn 'capacity' nix-config/modules nix-config/hosts returns no runner capacity setting; each host declares one services.gitea-actions-runner.instances.ymrtech. A daemon registers the sops-rendered /run/secrets/runner-config-<host>, not the Nix labels option — read the two together or you will trust a label list that is not in force.
• A run’s job queue. python3 /var/lib/hermes/workspace/fjlog.py jobs <runIndex> prints its jobs with name, status and duration; fjlog.py status <sha> is the cheaper per-job liveness view. Skipped jobs leave their commit-status context stuck at pending forever, so a green run is usable while a non-terminal aggregate is not.
• Slot occupancy, per host. Every host declares the daemon as systemd.services."gitea-runner-ymrtech", so its own journal on that host — journalctl -u gitea-runner-ymrtech — records what it claimed and when. The capacity lever, if needed, is act_runner’s [runner] capacity.
• The habit. For every needs: edge, ask whether the two jobs could have run at once. If yes, and the edge exists anyway, it costs a slot and a serialisation for nothing. The one edge in this graph has its justification in the file: two halves of a DNS tier must not be bounced in the same second.

For the wider pool — which labels each host advertises, and why the deploy jobs stay pinned while the light gate jobs float — see how the stack is built.