I Measured One GPU Three Ways and Got Three Answers

A write-up went around this week: PrismML's Bonsai-27B, 1-bit quantised, running under MLX on a 16 GB M4 Mac mini at 21 tokens/second in about 4.2 GB of RAM. Fully offline. The same model at standard precision wants roughly 48 GB.

We were confident our RTX 5090 served the same model at around 200 tok/s, and set out to publish the receipt. What follows is four rounds of measurement that produced three different answers, two retractions, and one finding worth keeping.

The finding is not a throughput number. It is that we could not honestly produce one, and why.

What we run

Same model, different quantisation and runtime: the Q1_0 GGUF at roughly 1.1 bits per weight, on a llama.cpp fork whose custom kernels read that format.

/work/build/bin/llama-server -m /work/bonsai.gguf -ngl 99 \
  --ctx-size 4096 --reasoning-budget 512 --alias bonsai-27b

3,803,452,480 bytes on disk — 3.54 GiB. Twenty-seven billion parameters, smaller than a lot of container images. One card, n_parallel = 4, unified KV cache.

Round 1: 26 tok/s. Wrong.

First sample against the live endpoint came back at 26.1 tok/s — statistically indistinguishable from a Mac mini. It was wrong because the fleet was mid-generation on the same server. Anything measured against a production endpoint without stating the load is a random variable with a decimal point.

Round 2: ~98–152 tok/s. This one holds up.

Instead of synthetic requests, we read the server's own timers across 264 real fleet requests — decode only, excluding prompt processing:

Large N over real traffic averages the contention out. This is the number we stand behind. It also matches what was already in our configs — the serving recipe has claimed 116 tok/s on RTX5090 for months, and a distributed benchmark recorded 133 single-node.

But it is single-stream, and single-stream on a four-slot server measures latency, not capacity. So it does not settle the 200 question either way.

Round 3: 221 and 228 tok/s. Retracted.

Firing requests concurrently and summing the per-slot decode rates produced 221.2 tok/s at four streams and 228.5 at eight. That looked like vindication, and we published it.

It was an upper bound we had not verified.

Summing per-slot rates is only valid if the slots are actually decoding at the same time. If slot A decodes from 0–2s and slot B from 5–7s, adding their rates describes a throughput the server never produced at any instant. We had asserted the concurrency rather than measured it — the same error as the original 200 claim, one layer deeper.

Round 4: measuring the assumption itself

The test is cheap. If the slots genuinely overlap, the summed decode time must exceed the batch wall-clock. Overlap factor = Σ(decode time) ÷ wall. At four slots, 1.0 means fully serial and 4.0 means fully parallel:

The slots do overlap — 2.4–2.7x is real parallelism, not an artifact. But they never approach 4.0, and in one run they barely overlapped at all. Meanwhile sum-of-rates in these instrumented runs came in at 73–91 tok/s, not 221.

So the 221 and 228 figures were real observations at a favourable moment, not sustained capacity. We cannot presently measure aggregate throughput on this box, because the fleet is using it and the variance (73 to 228) swamps the quantity.

The finding that survived all four rounds

Every round pointed at the same thing, and it is not the GPU.

Decode is healthy. Everything around decode is not. One logged request:

prompt eval time =  9261.93 ms /  21 tokens ( 441.04 ms per token,   2.27 tok/s)
       eval time =   508.58 ms /  64 tokens (   7.95 ms per token, 125.84 tok/s)
      total time =  9770.50 ms /  85 tokens

Sixty-four tokens generated in half a second, then nine and a quarter seconds spent processing twenty-one tokens of prompt. Worst prefill samples run to 1,042 ms per input token.

This also explains round 4. A slot stuck in prefill is not decoding, so it cannot overlap with its neighbours — the prefill pathology is precisely why the overlap factor is 2.4 instead of 4. The two findings are the same finding.

The causes are named in the log:

Context checkpoints cost 149.6 MiB each, up to 32 per slot, on a model whose hybrid attention then discards them:

slot create_check: created context checkpoint 1 of 32 ... size = 149.626 MiB
srv  get_availabl: prompt cache update took 676.91 ms
slot update_slots: forcing full prompt re-processing due to lack of cache data

KV exhaustion collapses the batch by halves until it quits — and a batch of one is exactly where 1,042 ms/token lives:

n_batch = 1024 ... 512 ... 256 ... 64 ... 16 ... 4 ... 1
Context size has been exceeded. i = 652, n_batch = 1, ret = 1

Four slots share a 4,096-token context on a model that trains to 262,144, so live callers sending 4,178-token prompts are rejected outright having done zero work.

We then applied the first fix — -ctxcp 2 instead of 32, and a 2 GiB cache ceiling instead of 8 GiB — and re-measured. On 1,211-token prompts prefill now runs at 2,212–2,590 tok/s steady-state, and the worst per-token prefill fell from 1,042 ms to 78 ms. All three log pathologies went to zero:

Honest caveat, because this post is about exactly this failure mode: the post-fix window is about five minutes of uptime against twenty-nine hours of baseline. Zero counts have had far fewer chances to trip. This is a strong early signal, not a settled result, and it needs a full day before anyone should quote it.

Round 5: the answer, from production instead of a benchmark

We kept trying to create concurrency and measuring the result. The server had been recording the real thing for 29 hours.

Every decode event in the log carries an end timestamp, a duration and a token count — enough to reconstruct exactly when tokens were being delivered. Spreading each request's tokens across its own decode window and binning per second, over 324 decode events spanning 29.2 hours of genuine fleet traffic:

In 29 hours of production this server has never delivered anything close to 200 tok/s. The best ten-second stretch it ever managed was 146.4 — below the 152.3 single-stream ceiling. Aggregate concurrency has essentially never paid off in real traffic.

And the reason is not the card. 2,635 busy seconds out of roughly 105,000 — a 2.5% duty cycle. The overwhelming majority of the time, exactly one request is in flight or none is. You cannot collect a concurrency dividend that the workload never asks for.

That is the whole answer, and it needed no benchmark at all: it was sitting in the log the entire time we were generating synthetic load next to it.

Against the Mac mini, the defensible comparison is the single-stream one: ~98–152 tok/s versus 21, roughly five to seven times, on a card with fifteen times the memory bandwidth. Sublinear, because once the weight file is 3.8 GB, streaming weights stops being the expensive part.

What this actually cost us

Three published numbers, two retracted. 26 was contention. 221 was an unverified assumption. Only the 264-request distribution survived, and it survived because it had a large N over real traffic rather than a clever method.

Both retractions were the same mistake. The original "200" asserted a throughput nobody had measured. Our replacement asserted a concurrency nobody had measured. Being burned by an unverified assumption did not stop us reaching for another one four hours later — it just made the second one better dressed.

The instinct was sound and the evidence was not. There really is throughput being left on this card. It is being lost in prefill, not in silicon, and no amount of confidence about the headline number would have found that. Measuring the thing we were sure of is what turned up the thing we were not looking for.

Honest limits

• The synthetic benchmarks were taken on a live endpoint under concurrent fleet traffic, which is the direct cause of their unusable variance. The round-5 figures do not have that problem — they are the production traffic, measured over 29 hours rather than competed with.
• Round 5 counts only requests that logged a decode timing line (324 of them). Streaming or cancelled requests that never printed one are invisible to it, so the duty cycle is a floor.
• The overlap factor is derived from summed decode durations against wall-clock. It proves parallelism occurred; it does not precisely quantify instantaneous throughput.
• Some concurrency-2 samples came back degenerate, stopping after ~2 tokens. We re-probed that case and it did not reproduce — both streams ran the full 192 tokens to stop_type: limit — so it was a transient, not a defect. Those samples stay excluded, but the exclusion is now explained rather than hand-waved. That re-probe also landed at 26–28 tok/s per stream, against 51–58 in an earlier run of the identical request: further evidence that this endpoint cannot be benchmarked while the fleet is on it.
• Cross-runtime comparison is directional only: the Mac mini figure is 1-bit under MLX, ours is Q1_0 under a llama.cpp fork. We did not run MLX; the 21 tok/s is the author's, cited, not reproduced.
• The A6000 (29 tok/s) and DGX (25–33 tok/s) figures are prior benchmarks, not re-run. The prefill fixes are staged and unapplied, so nothing here measures them.

The number we are actually confident in

3.54 GiB, holding 27 billion parameters, running the same way on a $599 Mac mini at 21 tok/s, on an OptiPlex with no GPU at all at ~4 tok/s, and on one 5090 at ~100–150 — wired into the same scheduler, answering the same API.

That one we measured.