NVIDIA Made KV Caches Portable Between Models. We Shipped It The Same Day — On A CPU.

Prefill is the tax you pay before a model says anything. Every turn, the model re-reads the entire context before emitting a single token. Prompt caching softens it — hold the KV cache for a stable prefix, bill a hit at a tenth of the rate — but there's a catch that quietly shapes the whole economics of LLM routing:

The cache only works on the model that produced it.

Keys and values are functions of that model's weights. Route the next turn to a different model — cheaper, smarter, less loaded, whatever — and everything you accumulated is dead weight. You pay full prefill again. Which means every routing decision is secretly a question about how much context you're willing to throw away.

NVIDIA's recent result treats that as a representation problem rather than a law of physics. Prefill's only output is the KV cache, so: learn to convert one model's cache into the format another expects. Four of six tested pairs retain 73–98% of the receiving model's standalone accuracy, and conversion runs 3–25× faster than reprocessing the context.

We read it Thursday. Here's what Aitherium did with it before Thursday was over.

---

The constraint, stated up front

One person. No budget. The GPU — an RTX 5090 — was already 23 GB into its 32 GB serving other things, and the host torch install is CPU-only.

Everything below ran on CPU. The models were whatever was already sitting in the HuggingFace cache. Zero downloads. Marginal cost: roughly the electricity.

That constraint picked the experiment, and it picked a harder one than the paper's:

The paper is explicit that every pair it tested shares KV-head count across scales, and that mismatched head configurations are untested. We couldn't afford the easy version, so we ran the open question instead.

First, check whether the claim is even coherent

Before writing a line of implementation, one piece of arithmetic — because "25× faster" is either load-bearing or marketing, and it's cheap to find out which.

The mapper is one small matrix per (target layer, target head), reading the concatenated keys of the top-k source layers. For the paper's Qwen3-14B → 32B pair:

mapper:      in_dim=8192  params=1.07B   2.15 GFLOP/token
prefill 32B:                            64.00 GFLOP/token
FLOP ratio :                            29.8x cheaper

A 30× ceiling — and the paper measures 3–25×, below it. That's the signature of an honest memory-bound number. A claim of 100× would have ended the investigation right there.

The same arithmetic surfaces a cost the write-ups skip: the mapper is 1.07 billion parameters, 2.15 GB resident, to convert into a 32B model. Good to know before you architect around it.

Three ways to get this silently wrong

Implementing the idea is easy. Implementing it correctly is the work, because every wrong version produces code that runs, fits, and looks fine.

Keys carry a rotation, and you have to take it off. RoPE has already rotated every key by an angle proportional to its position. A single fixed matrix is being asked to learn a representation composed with a per-row rotation it cannot see — the best it can do is average over the rotations in training. So: strip the source model's rotation, fit in position-free space, re-apply the target's rotation at inference. Since the two models disagree on rope_theta, that's not a round trip. It's now a test in our suite — the de-rotated fit must beat the rotated one by >0.2 R², or the step is ceremony. It does.

The rotation convention is a coin flip that fails invisibly. HuggingFace pairs dimension i with i + d/2. The original RoPE paper and GGML pair adjacent dimensions. Both are "RoPE." Pick wrong and the rotation is still orthogonal, still invertible, still correctly shaped — it round-trips perfectly, preserves norms, and passes every test you'd naturally write, while the fitted mapper reconstructs noise. The only thing that catches it is one hand-computed rotation with a mutation guard implementing the rival convention, asserting the test can tell them apart.

Both models must tokenize identically, and nobody says so. The map sends position i of the source to position i of the target. Different tokenizers mean position i isn't the same token, and you're regressing one document onto a misaligned other. Shapes agree, the fit converges, and the mediocre R² reads as "this pair doesn't transfer well" rather than "these rows don't correspond." Every published pair shares a tokenizer by being one family — which is exactly why the requirement goes unwritten. In our implementation it's a hard refusal, by vocabulary digest and by comparing token ids per document at capture.

The trick that turned a month into an afternoon

The fit needs, per target layer, one regression per candidate source layer to rank them, plus one on the winner. Naively: 31 × 24 = 744 passes over the activations.

But all of those regressions are submatrices of the same two accumulators — X'X over all source layers, and X'Y for that target layer. Build them once, and ranking 30 candidates plus sweeping 7 ridge strengths is linear algebra on slices.

One data pass per target layer. The hyperparameter sweep is free. The full 24-layer fit runs in minutes on a CPU, which is the difference between an experiment you run once and an experiment you can iterate on.

What it found

Reconstruction quality, held out, mean over 24 target layers:

(the paper's reference, on its considerably easier pair: 0.56 → 0.79)

The core finding reproduces: combining source layers beats the best single source layer on every target layer, and by more for values than keys.

Then the part nobody asked for.

The mapper discovers depth correspondence unsupervised. There is no alignment prior anywhere in the fit — each target layer independently ranks all 30 source layers by held-out R² and keeps the best 8. What it chose:

target layer  0  ->  source [0, 1, 2, 3, 4, 5, 6, 11]
target layer  9  ->  source [12, 13, 14, 15, 16, 17, 18, 19]
target layer 20  ->  source [19, 21, 22, 23, 24, 25, 26, 28]

A clean monotonic march. Two models of different depths, trained separately, and a ridge regression recovers the depth mapping between them with no supervision and no hint. That's not a plumbing result — that's evidence the two models are building recognisably the same thing at recognisably the same relative depths, and it fell out of the fit for free.

Does the big model actually use it?

Reconstruction R² tells you the map learned a mapping. It does not tell you the receiving model can work from the output. So we measured what the 1.7B actually predicts from a translated cache, on a disjoint corpus, against three reference points:

The control arm is what makes this readable. A mapper that quietly learned the target's average statistics and ignores its input would score respectably against reference alone.

Against the floors, the verdict is clean: 3.5× above control, and the control arm is worse than having no cache at all — meaning the target model is genuinely attending to the translated cache, not routing around it. Real cross-model information, transferred between two models that share nothing but a tokenizer and a training recipe, in the regime the literature hasn't tested.

It isn't production-grade on this pair yet, and because the fit is cheap we could just ask why instead of guessing. The obvious suspect is under-training — 16,384 tokens against a 1,536 → 2,048 dimensional regression is about 10:1. So we swept it:

Saturated. An 8× increase buys +0.057; the final doubling buys +0.003. More data is not the lever — the limit is that the source model's values are 192 dims per layer against the target's 2,048. You cannot regress your way out of an information bottleneck, and it's exactly why V saturates lower than K.

That's a genuinely useful negative: it says don't spend a week on a bigger calibration run, change the geometry. Qwen3-0.6B → Qwen3-4B — matched KV heads, matched head dim, shared tokenizer — is next. The easy regime, on a platform that already ran the hard one.

Meanwhile the platform won't let a half-good pack ship. Our loader refuses any mapper without downstream acceptance evidence above a floor, over a large enough sample to mean anything. Frontier labs have reviewers; a one-person lab needs gates that refuse — including refusing us. Ours did, on the first artifact we fed it, which is precisely the behaviour you want standing between an interesting R² and a production cache.

The bonus finding: distributed KV is nearly free at our scale

A related paper this week — Prefill-as-a-Service (arXiv 2604.15039, Tsinghua/Moonshot) — tackles the neighbouring problem: shipping a cache between machines running the same model. Its central anxiety is bandwidth. Dense models at 32K context emit KV at 33–60 Gbps per instance; hybrid-attention models cut it to 3.9–4.7 Gbps, and the paper's whole architecture exists to schedule around what's left.

We ran their metric on our own fleet. DeepSeek-V4-Flash, IQ3, prefilling at 6.18 tok/s:

52.7 MB of KV per 5,061 tokens, produced over 819 seconds
  = 0.515 Mbps of egress

Roughly 7,500× below the cheapest case in that paper. Not through cleverness — because our prefill is slow, so the cache accumulates at a trickle.

Which flips the conclusion completely for small operators. That 5,061-token cache crosses a 1 Gbps LAN in 0.42 seconds. Recomputing it takes 819. The elaborate bandwidth-aware scheduling that makes cross-datacenter KV viable at H200 scale is solving a constraint a small heterogeneous fleet simply does not have.

Distributed KV cache is not a frontier-scale capability you're locked out of. It's nearly free, and the hard parts are the unglamorous ones we'd already built: content addressing, mesh-wide warm-prefix discovery, and a manifest check strict enough that a cache is never restored into the wrong model.

Why this took a day

Not cleverness. Infrastructure.

A content-addressed KV cache plane with a manifest check was already there. A corpus was already in the repo. A quality-gate culture where every checker has to prove it can still fail was already there — so the mapper shipped with mutation guards that reproduce the wrong implementation and assert the tests can tell the difference. A commit protocol that holds a lease so parallel sessions don't clobber each other was already there.

The gap between "a frontier lab published this" and "we have measured it ourselves, with controls, on our own hardware" was one session and about zero dollars — because the boring parts were already solved.

That's the whole thesis of Aitherium. Build the unglamorous plane once, and a one-person lab moves at the speed of its ideas instead of the speed of its setup.