Running a 27B model on a 3090: a budget, not a flag

Fitting a 27B-parameter model on a single consumer card is an arithmetic exercise, not a configuration one. There are two unknowns: weight memory at the quantisation you choose, and KV cache at the context length you actually want. Everything else — offload counts, thread counts, model-loading flags — is downstream of those two.

The failure mode is rarely “it does not fit”. It is “it fits at 4k context and dies at 32k”, or “it fits, and it generates at two tokens per second”. Both are budget failures, and both are visible before you serve a single request if you measure instead of guess.

This is a retired experiment. The GPU box’s local model stack has been switched off — as of 2026-09-16 the serving endpoint had no listener at all (TCP refused) and the host’s AI services were disabled. What remains useful is the method and the harness, not the deployment.

The two numbers that matter

Throughput claims about local inference reduce to two measurements: tokens per second (TPS) and time to first token (TTFT). Both must be taken at the context length you intend to use. A headline TPS on a 512-token prompt is not evidence about a 32k-token session; it is evidence about a different workload.

The retired baseline illustrates this because it is one point, not a verdict: roughly 24 tokens/second decode on a short prompt with a TTFT of 0.36 s, for a 27B model on a 24 GB RTX 3090 with the length ceiling configured at 90000 tokens. That 24 is a point on a curve. Decode TPS falls as context grows, because the KV cache competes for the memory the weights occupy — and that curve, not the number, is the interesting output.

One trap worth keeping: reasoning models stream their thinking in delta.reasoning_content, not delta.content. A TPS counter that reads only content reports zero tokens while the model works perfectly — it looks like a broken server and is a broken counter. Ask the server for exact usage with "stream_options": {"include_usage": true} so the final chunk carries the true prompt-token count and the prefix-cache hits.

Weight memory is a budget line, not a setting

Quantisation trades memory for quality and speed. It is a budget line, not a preference — and it cannot be chosen independently of the context you need, because both draw on the same pool. On the 24 GB card this ran on, the 27B model at the quantisation in use stayed resident at roughly 21 GB of VRAM. That leaves a few gigabytes for the KV cache and compute buffers; a larger context comes out of that remainder.

There is a version of this that looks like a flag. The arch pin is real: RTX 3090 is compute capability 8.6, and pinning CUDA architectures to 8.6 rather than building for every architecture is the single biggest build-time lever. It also changes the derivation hash of every CUDA package, so the official caches stop matching and packages like nccl, onnxruntime and triton recompile from source — hours on a small core count. That is a cost you budget, not a switch you flip.

The term people forget

KV cache grows with context length and with concurrent sessions. It is the term missing from most “will it fit” arithmetic, because it is invisible until the second request arrives. A configured ceiling of 90000 tokens does not mean a 90k-token session is affordable once the resident weights and the cache share one card; it means the server will accept it and then decide.

The budget is a line, not a table:

1
VRAM >= weights(quantisation) + KV(context x concurrent sessions) + overhead

Every term on the right is knowable before launch: the 21 GB resident figure is the first, and the context you need decides the second.

Measure the curve, keep the harness

A single result decays — a driver update, a requantisation, a firmware change moves it. A benchmark you keep and re-run is worth more, because the output you care about is the curve across context lengths. This fleet’s standard harness grows a simulated agent session across roughly twelve turns to about 88k input tokens — near the 90000 ceiling — then runs needle-recall probes, then replaces the middle of the history with a lossy summary and re-probes to measure recall fidelity after compression. Content is deterministic (seeded), so runs are comparable. A --target flag and a quick smoke mode cover shorter runs.

The conclusion is narrow. On a single desktop card, the useful question is which context lengths your workload actually needs — because that answer, not the parameter count, determines what you can run at all. The first full run of the harness has not been completed and published; what exists today is the method and the file, which is what makes the next run comparable to it.

What you can check

• Run the same harness at two context lengths and look at the TPS/TTFT curve for yourself. One number is an anecdote; two points on the same curve are a measurement.
• Record the quantisation alongside every number. A TPS figure without the quantisation that produced it is not reproducible, because the quantisation sets the first budget term.

The infrastructure these services ran on — hosts, GPU modules, the deploy health gate and its rollback — is described at how the stack is built. The benchmark above is the same discipline as the audit page’s signed artifacts: measure, keep the artifact, publish the method, and let the curve, not the headline, be the claim.