Cookieless A/B testing without lying to anyone

A/B testing has a reputation it earned. Most of the tooling sold for it answers “which variant won” by first answering who are you, and keeping the answer — a cookie, a session identifier, a device graph — for as long as someone wants to keep asking. None of that is a requirement of measurement; it is a requirement of a business model that got bundled with measurement.

You can assign a visitor to a variant, count the outcome, and hold no identifier for them — by deriving the assignment from something coarse and rotating instead of storing it, and by being clear up front about which questions that makes unanswerable.

The assignment is derived, not stored

The design this site’s stack prescribes, and what its privacy policy describes in §2.1, is: assign server-side — never a client-side splitter, never a blocking third-party script. The edge service decides which variant a request sees; the page only reports what happened. Nothing is set on the client at all.

The assignment key is not an identity. It is an HMAC over a coarse input — a client address reduced to a network-sized prefix, plus the coarse browser family — under a key that rotates on a schedule. Two consequences follow directly:

• Within a rotation window the assignment is stable and reproducible from the request alone, so nothing is remembered between the impression and the conversion.
• Across the boundary the same person hashes to a different key. The two windows cannot be joined back together — by us, or by anyone who later obtains the logs.

That is the mechanism, and it is also the trade. Hash the coarse UA family alongside the address, because one NAT or office IP would otherwise pin everyone behind it to a single arm. Decide the day-bucket timezone once and document it — UTC avoids the DST cliff where a bucket boundary silently moves.

Attribution does not depend on the key. The variant rides in the links: through the CTA, into the signup entry, validated against the declared variant set, into the payment provider’s session metadata, and out as one structured conversion line from the payment webhook. The authoritative conversion is that event — never a click, never a platform-reported number.

What the rotation costs

The cost is precise, and it belongs in the design document before the test rather than in the readout afterwards: you cannot follow a returning visitor across the rotation boundary. Retention-style metrics — did the person who saw arm B come back ten days later — are out of scope by construction. A visitor-level split on conversions also needs volume to resolve anything: below roughly fifty visits a day it cannot detect a sane effect size, and the move there is to test at content or campaign level and aim the instrumentation at clicks, scroll and dwell.

The second cost is caching. Per-visitor content served under the site’s default cache header means one visitor’s variant is served to everyone, so the response must carry Cache-Control: no-store, private plus Vary on the header the proxy itself sets. Bots are served the control variant and marked uncounted, so they cannot skew one arm.

A system that cannot state the question

Here is the argument underneath this, and it is not a compliance argument. A measurement system that requires identifying everyone to answer a question is usually a system that has not defined the question. This design forces the question into the open — what is the key, how long is it stable, what can we not measure — because the answers have to be written down before the first impression. The questions it refuses are not sacrifices; they are the ones nobody had specified.

The same discipline shows up elsewhere here. The fair-share shaper that caps the VPN tiers keeps no per-peer traffic record, because the no-log posture forbids it, so fairness comes out of the data path instead: a guaranteed share plus a burst ceiling, read from nothing but the live counters the kernel holds in memory. The constraint did not weaken the mechanism. It chose it.

What you can check

• The recorded and not-recorded lists. The privacy policy §2.1 states exactly what a page measurement contains — that a page was viewed, the A/B variant when a test is running, the browser’s own timing, uncaught JavaScript errors — and what it does not: no cookie, nothing in local or session storage, no IP stored with a measurement, no cross-site identifier.
• The retention numbers. Log records are kept 14 days, and the page-timing measurements travel inside those same records — there is no separate metrics store for them, so they expire with the logs. The aggregate byte counters are kept at most 90 days. Every period is enforced by the servers and visible in the public configuration.
• The build gate. The measurement library is pinned by hash in the public repository, and every build re-checks that the served file matches the pin, that the page sets no cookie and leaves nothing in browser storage, and that it contacts nothing but this domain’s own collector.
• The checklist for any proposed test: what is the assignment key, how long is it stable, what can we not measure, and what gets published.

None of this claims cookieless measurement is strictly better. It claims the questions a test cannot answer should be decided before the test, not discovered in a dashboard afterwards — the same reason the audit page is built from signed artifacts rather than a status badge. A claim you cannot inspect is only a claim about intentions.