The config change that never restarted

On a declarative system it is tempting to believe that editing a file is the change. On NixOS it is not. A config file is content, and nothing restarts the process holding the old copy unless the unit says so. The deploy can be green, the file can be correct, and the running system can be unchanged.

The fleet’s own notes have the incident that makes this concrete, and it is worth quoting exactly.

The feature that never appeared

The hermes-agent module renders its settings into config.yaml — but defines no restart trigger. The observed result: config.yaml contained a new platforms.email block while the gateway was still the pre-deploy process (ExecMainStartTimestamp from the previous day, NRestarts=0), and “the feature just never appeared.” The file had the change. The process did not.

That is the whole failure: a config-only deploy that reports success, and a long-lived process still answering from the configuration it loaded a day earlier. EnvironmentFile-style configuration behaves the same way — it is read at start, so an edited file has no effect until the unit restarts.

What NixOS gives you, and what it does not

NixOS offers restartTriggers for exactly this: a unit carrying restartTriggers = [ ... ] restarts when the content of those files changes, which is how a change to a generated config file pulls a restart behind it. Without a trigger, nothing does.

1
2
3
4
5
6
systemd.services.hermes-agent.restartTriggers = [
  (pkgs.writeText "hermes-agent-settings-trigger" (builtins.toJSON {
    settings = config.services.hermes-agent.settings;
    environment = config.services.hermes-agent.environment;
  }))
];

Two details from that wiring matter. The trigger hashes environment as well as settings, because an env-only change would otherwise go stale — and the definition has to be gated on whether the module is enabled, because on a host without it the reference fails evaluation.

The other half is the reload path. A unit with a working ExecReload can pick up a config change without a restart; here unbound is left to reload rather than restart so it keeps its resolver cache, set with systemd.services.unbound.reloadIfChanged = true on the mail and giga hosts. But a unit without a usable reload action will accept a reload request and do nothing useful — a no-op that looks like success. And a graceful reload is not always enough: the notes record that Caddy’s graceful reload does not make it re-read a new user-supplied tls file pair, and only a restart does.

Verify the process, not the file

The check that decides is the running system, not the artifact on disk:

1
2
systemctl show -p ExecMainStartTimestamp <unit>   # when the process actually started
stat -c '%y %n' /path/to/the/config/file          # when the file actually changed

If the process started before the file changed, the process is running the old configuration — whatever the file says. Where a service can report its loaded value, read that instead of the file: a runtime probe that echoes the loaded configuration is a claim about the process, and the file is only a claim about the disk.

The principle

“The file changed” and “the running system changed” are different claims. Only the second is what you meant when you opened the editor, and only the second is worth putting in a change record: a file in the store is an input to a deploy, and the process that started after it is the evidence that the deploy did something.

What you can check

• Pick a service, change something in its configuration, deploy, and see whether ExecMainStartTimestamp moves. If it does not, your change is on disk and not in the process.
• restartTriggers for hermes-agent in hosts/common/users/hermes/default.nix (covering vpn and public) and inline in hosts/giga/default.nix, in the nix-config repository.
• systemd.services.unbound.reloadIfChanged = true in hosts/mail/default.nix and hosts/giga/default.nix — reload instead of restart, to keep the resolver cache.
• The Caddy note about new tls file pairs in hosts/mail/default.nix, which explains why “reload” is not always enough.

For how a deploy is verified on the host after a green run, see how the stack is built.