The DNSSEC trap in a zone you answer yourself

There is a failure mode that looks like an outage and is not one. A name that resolved for months stops resolving everywhere at the same instant — not for one client, but for every machine that asks. The zone is fine, the records are fine, nothing was deleted. What changed is that somewhere in the answer path a resolver tried to prove the answer, could not, and said so.

DNSSEC is why, and the trap is that the failure is not proportional to the mistake.

SERVFAIL is not NXDOMAIN

A validator that cannot build a chain of trust from the root to the answer does not serve the unsigned answer as a courtesy. It returns SERVFAIL. The point of validation is to refuse answers it cannot check.

1
2
3
4
5
6
7
# A validating resolver, asked for a name it cannot validate
$ dig +short ymrtech.com
# (nothing — rcode SERVFAIL)

# The same name, asking the resolver to skip validation
$ dig +short +cd ymrtech.com
11.0.0.7

Those two commands are the whole diagnosis. +cd sets the checking disabled bit, so it shows what the resolver could actually find: if +cd answers and the plain query does not, the data is present and validation is what failed. The records were never the problem.

At the client, SERVFAIL is indistinguishable from a broken zone: “the name does not exist” is what you see, and it is wrong. It resolved a minute ago, it resolves from the server; only the validator differs.

A zone you answer yourself

This fleet answers its own public names internally. The list lives in one place, hosts/common/global/internal-dns-zone.nix:

• primary — AdGuard Home on 11.0.0.1:53, upstream unbound on :5353,
• standby — AdGuard Home on 11.0.0.7:53, upstream unbound on :5335.

Both read the same records, so a client pointed at either address gets the same answers. Without them, a fleet host talking to one of our own services leaves through the provider’s NAT and re-enters with an external source address.

But ymrtech.com is a DNSSEC-signed public zone, and this is where the trap closes: an internal answer carrying a private address has no signature a public validator can accept — there is nothing to check. So the resolver does the only thing it is allowed to do:

It answers SERVFAIL, not the record.

That sentence is the comment on the zone definition, written when the overrides were added. The remedy it prescribes is one option on the validating hosts — in hosts/giga/default.nix:

1
2
3
4
5
# The internal ymr zone is unsigned and answered through AdGuard Home, which
# does not forward DNSSEC records. With the root trust anchor loaded (nixpkgs
# default), the validator answered SERVFAIL for every *.ymr name (verified:
# +cd resolves, plain query fails).
domain-insecure = [ "ymr" "ymrtech.com" ];

hosts/mail/default.nix carries the identical line with the identical reasoning, and both cite the same verification: +cd resolved to 11.0.0.7 while the validating query failed. The public host never hits this at all — it serves ymr from local-zone "ymr." static, so the zone never reaches a validator. Three hosts, three relationships to the same records, and only the two that forward needed the declaration.

The trade, not the mistake

It is tempting to file this under “we broke DNS with DNSSEC”. That reading hides the useful part. DNSSEC buys an answer you can trust and charges a hard failure mode for every break in the chain: a record that is stale, unreachable, or answered from a split-horizon view used to degrade softly — a client fell back, an old address kept working. Under validation there is no soft option. A hairpin answer (internal clients served public addresses, or the reverse) is precisely what a validator rejects, so it does not degrade; it disappears.

The trade is worth paying when the trust anchors are agreed everywhere, and it is a trap when they are not.

What you can check

• Ask for SERVFAIL specifically, not just NXDOMAIN. When a name that used to resolve stops resolving, run both queries above against the resolver the affected client actually uses. +cd answering where the plain query does not is the signature.
• Test from a real client, not from the server. Whether validation succeeds depends on the resolver in the path, so a query from the authoritative side proves nothing about the laptop. Point a client at the resolver it uses in production and repeat.
• Look for the declaration before you look for the bug. On this fleet, domain-insecure in hosts/<host>/default.nix decides whether an internal answer survives validation. If a host validates and answers internal names, those two facts need to agree in writing.
• Prefer unsigned internal zones unless everything agrees: keep a zone you answer yourself unsigned, or make sure every resolver and every client that validates shares the same trust anchors. Half-signed is the state that produces the outage.

The failure is invisible from the middle: nothing in a zone file, a record list or a service page shows an answer being refused for lack of a signature. That is the same argument behind the audit page: a claim you cannot inspect from the client’s position is only a claim about the server’s intentions.

For the wider resolution picture, see how the stack is built.