The KV Cache Is the Wall — Two Measurements That Move It

Bonsai runs in your browser. Not a hosted model behind a fetch — the weights land in your tab, the WGSL kernels compile against your GPU, and the tokens come out of hardware you own. The smallest size is 236 MB for 1.7B parameters, which works out to roughly one bit per weight.

Getting the weights small was the interesting problem for about a month. Then it stopped being the interesting problem, because of something that does not show up in any model card.

The wall nobody quotes

Model weights are a fixed cost. You pay 236 MB once, it sits in VRAM, and it does not grow.

The KV cache is not like that. It grows with every token in the conversation, and the arithmetic is unforgiving. Head dimension is 128 across every Bonsai size; the KV-head count comes from the model's own metadata. Take the shape used in the measurements below — 28 layers, 128 head-dim, 4 KV heads, f32 cache — and one position costs:

4 kv-heads × 128 dims × 2 (K and V) × 4 bytes  =  4,096 bytes per layer
                                     × 28 layers  ≈  112 KB per token

112 KB per token. A 2,787-token prompt is over 300 MB of KV cache — more than the entire 236 MB model it is serving. Double the KV heads and you double that.

That is the wall. On a desktop with a real GPU you can shrug at it. In a browser tab, on a laptop, next to twelve other tabs, you cannot. It is the reason "the model fits" and "the conversation fits" are different sentences.

There are two ways through: make the cache smaller, or stop keeping all of it on the GPU. We measured both this week, and both measurements surprised us.

---

Measurement one: 4-bit KV error gets better with context

Quantising the KV cache to 4 bits is the obvious lever. Each K and V row gets packed to nibbles with a per-row f16 scale:

scale = roundF16(max|x| / 7)
raw   = clamp(round(x / scale) + 8, 0, 15)

Dequantisation is (raw - 8) × scale. Symmetric, one scale per row, 4 bits per element. Counting the packed nibbles and the per-row scales, that same position drops from 4,096 bytes per layer to 544 — about 15 KB per token instead of 112 KB. The 300 MB prompt becomes roughly 40 MB, and it now fits alongside the model rather than dwarfing it.

Cheap. The question is what it costs in quality, and the intuition everyone has — including us — is that quantisation error accumulates: more positions, more noise, worse output. Under that theory, 4-bit KV is fine for a chat message and dangerous for a long document.

So we measured it. Same query, same cache, computed twice: once over the true f32 K/V, once over the K/V round-tripped through 4-bit. Then we varied context length and looked at mean absolute error, max error, and — the one that actually matters — whether the argmax changes, because that is what changes a token.

Error does not accumulate. It shrinks by roughly 12× across a 174× increase in context.

Once you see it, it is obvious, and that is the best kind of result. Attention output is a softmax-weighted average over positions. The quantisation error at each position is independent — it comes from that row's own scale, not from anything that happened earlier. Average more independent errors together and the mean error goes down, exactly like any other sampling average.

Which inverts the intuition completely: long context is the safe end of this axis for 4-bit KV, not the dangerous one. If you are going to worry about 4-bit quality, worry about short prompts.

Why the first version of this table was worthless

The first run of that measurement produced a beautiful, clean, completely fake result: byte-identical error at every context length. 0.04922855290753129 at 16 tokens and at 2,787 tokens. Five rows, one number.

That is not a finding, that is a fixture bug — and it is worth naming the shape, because it is a trap anyone measuring attention will hit. The harness used one query token at position base zero. Causal attention means a query at absolute position p attends to [0, p]. With posBase = 0 and one token, p = 0, so the loop ran over exactly one cache position regardless of how much cache we had built. It measured the same single-position computation five times and reported it as a trend.

The tell was the identical digits. Real measurements are not identical to seventeen decimal places. When your data is too clean, the fixture is usually answering a different question than the one you asked — and a conclusion drawn from it ("error does not accumulate") can be accidentally correct and still worthless, because nothing measured it.

The fix was one line: posBase = kvLen - 1, so the query actually sits at the end of the cache and attends to all of it. Every number in the table above comes from after that.

---

Measurement two: chunked attention is bit-identical

Making the cache smaller buys you a constant factor. It does not change the shape of the problem — 4-bit KV at 32k context is still hundreds of megabytes.

The other move is to stop keeping the whole cache on the GPU: hold the master copy in ordinary system RAM, stream a window of positions into VRAM, compute over that window, and move on. Host RAM is measured in gigabytes and is not competing with the compositor.

This only works if attention over a sequence of windows equals attention over the whole thing. And people are usually nervous about that, because it sounds like an approximation.

It isn't. The kernel uses online (flash-style) softmax, and the entire per-(token, head) state is three values:

• m — the running maximum score
• l — the running softmax denominator
• acc — the running weighted sum of V, unnormalised

Nothing else from earlier positions is consulted. So carrying (m, l, acc) across a window boundary without normalising leaves the sequence of floating-point operations completely unchanged — same additions, same multiplications, in the same order. The result is not "close". It is identical, bit for bit.

We assert exactly that: every window size from 1 to the full length, across four GQA shapes, compared with strict equality — no tolerance.

The strictness is the point. A tolerance would have passed for the three ways this actually goes wrong:

1. A dropped position at a boundary.
2. A double-counted position where windows overlap.
3. Normalising early — dividing by l at a boundary and continuing.

Each shifts the result by an amount a comfortable epsilon absorbs on a short sequence, then compounds on a long one. In other words: passes in the test suite, breaks in production, at exactly the context lengths the feature exists to serve. So the suite injects all three bugs and asserts they are detected.

Number three deserves special mention, because it is the one you write by accident. Normalising at a boundary produces a correct first window. Any test with a single window passes. It only diverges once there are two, which is the first case a naive test does not cover.

---

Making it land everywhere

A runtime improvement that only reaches one surface is half a feature. Bonsai ships in several places — the OS at aitherium.com, the tenant products, a browser extension — and the shared package that feeds most of them had drifted into being a different program: no tool calling, no device-lost recovery, no 4-bit KV.

So propagation became a build step rather than a habit. A sync computes the transitive closure of the runtime's relative imports, preserves the source directory layout so every relative path resolves unchanged, and — the part that matters — verifies its own output: any copied file importing something that was not copied is a hard failure naming the file.

That post-condition exists because the closure walk was wrong twice while we built it. First it missed a sibling package entirely. Then, once that was fixed, it missed two modules reached only through dynamic import() rather than a static from. Both times the miss showed up a whole build away, as a type error in the consuming package, long after the sync had cheerfully reported success.

A sync that can lie about being complete is worse than no sync, because the staleness gate goes green over a tree that does not build. Checking your own output is not paranoia when the alternative is a green light over a broken thing.

---

What this actually buys

Two numbers, one architectural consequence:

• 4-bit KV cuts per-token cache cost by about 7.5×, and its error decreases with context rather than compounding. The long-context worry was backwards.
• Chunked attention is exact, so the master cache can live in host RAM and stream to VRAM in windows without changing a single output bit. The ceiling stops being your GPU budget and starts being your system memory.

Neither is a heuristic. Both are properties you can assert in a test that fails when they stop being true — which is the only reason we are comfortable building on them.

The model fitting in the tab was never the hard part. Making the conversation fit is where the engineering is, and it turns out the answer is less about compression than about noticing which quantities are averages and which are sums.

Bonsai runs in your browser — no install, no account, no server round-trip. Weights land in your tab and the tokens come out of your own GPU. The four sizes range from 236 MB to 3.6 GB; start with the smallest.