Proportional weights are not caps

A CPUWeight looks like a way to promise a service some CPU. It reads like an allocation: give the important thing a big number, give the backup a small one, and the important thing is protected. That reading is wrong, and the module header in this fleet’s config says so in its own words — the weights “are not caps — on an idle box the background unit still gets everything.”

The file is modules/service-priority.nix. Its header is worth reading before the code.

A weight only decides who wins a tie

cgroup v2 CPUWeight is a relative weight, defaulting to 100, and it is consulted only when two cgroups want the same resource at the same moment. One unit at weight 1000 beside a unit at weight 10 means the first gets a hundred times the second’s share of whatever is contended. It does not mean the first gets a hundred CPUs, or even one. On an idle machine the low-weight unit runs as fast as it likes, because nothing is competing with it. IOWeight is the same idea one layer down, for the block layer.

This fleet sets CPUWeight = 1000 and IOWeight = 1000 on the client-facing units, and 10 and 10 on the build and backup units, and the header is explicit about why that is not a CPUQuota: on an idle box “the background unit still gets everything, so backups/builds do not slow down for nothing.” That is a deliberate trade — fairness, not protection.

What the numbers were, measured

The header records a real measurement from 2026-09-19 on the public host, a 1-vCPU edge: a fleet restic run pushed a portal fetch from 0.25 s time-to-first-byte to 1.60 s, and from 1.41 MB/s to 221 KB/s on the same URL, while a CI build held CPU pressure at roughly 58%. A 6x latency regression with no configuration error anywhere. A neighbour got busy, and the proportional share did exactly what it says on the tin.

The method matters more than the numbers: drive load on one unit, then measure the latency of one known request on the other, on the same host and the same URL. A weight setting is not verified by reading it. It is verified by measuring both sides of the contention.

The load you contained may not be in the cgroup you contained

The second trap is quieter, and it is the one that survives a correct weight table. Builds run in the nix-daemon’s cgroup, not the runner’s: the runner only spawns the nix client, and the daemon forks the builders. So putting a CI runner in its own slice and weighting that slice does not bound the builds it triggers — they are the daemon’s children, weighed wherever the daemon is. The module weights nix-daemon.serviceConfig too, and at 50 rather than 10, because the deploy jobs also go through that daemon and a deploy must not crawl.

That is the general lesson: before you claim a workload is isolated, ask which cgroup it actually lands in.

1
2
systemd-cgls                 # the tree, with units under their cgroups
systemctl status <unit>      # the unit's own slice and cgroup

Rules

• Weights are for fairness. Use them to decide who wins when siblings compete, and expect the loser to take its full share when nothing else wants the resource.
• Quotas and limits are for protection. CPUQuota is a ceiling. MemoryMax is a hard cap. If a latency-sensitive service must not be slowed by a greedy neighbour, that is the tool.
• Verify the cgroup, not the intent. Confirm with systemd-cgls or systemctl status that the work lands where the weight is before you claim isolation.
• Know what cgroup weighting cannot reach. The same header names two limits: IOSchedulingClass is inert on these hosts — the only block device is a rotational volume on mq-deadline, which does not honour io priorities — and on the wireguard host the client dataplane is kernel-side (wg-crypt kworkers, NAT forwarding, softirq) and cannot be cgroup-weighted at all.

What you can check

• modules/service-priority.nix in the nix-config repository: the “not caps” paragraph, the CPUWeight = 1000 / 10 split, and the nix-daemon weight of 50.
• The MemoryHigh = "768M" on the background units and MemoryLow = "128M" on the foreground ones — on these hosts the actual disk-latency protection comes from the memory side, not from the weights.
• On a host: systemd-cgls to see which cgroup a build actually lands in.

For how a deploy is verified on the host rather than assumed from a green run, see how the stack is built.