DeepSeek's Latent Cache Is 3× Smaller. That Makes It Easier, Not Harder.

Prefill is the tax you pay before a model says anything, and a KV cache only works on the model that produced it. Route a conversation to a different model — cheaper, smarter, less loaded — and everything you accumulated is dead weight. You pay full prefill again.

Cross-model KV transfer fixes that: learn a map that converts one model's cache into another's, so the receiving model skips prefill entirely. We built it, measured it, and shipped it open source today as aither-kvcache v2.4.0.

pip install "aither-kvcache[transfer]"

Along the way we hit the thing everyone treats as the hard boundary, and it turned out not to be one.

---

Reading the config instead of reasoning from the architecture

The technique fits one small matrix per (layer, head) pair. DeepSeek's Multi-head Latent Attention doesn't cache per-head keys and values — it caches a compressed latent that each layer decompresses on the fly. So the natural conclusion is that the two don't compose: no per-head tensor, no per-head map.

We wrote exactly that into our own pair-checker, as a hard refusal. Then we opened config.json:

num_key_value_heads   1
head_dim              512
qk_rope_head_dim      64
num_hidden_layers     43

That's a real, dense, well-defined per-token tensor: 512 latent dims plus a 64-dim decoupled RoPE key, per layer. Run the arithmetic against a conventional 8B model's cache for the same context:

DeepSeek-V4-Flash MLA:   43 × (512 + 64)      =  24,768 dims/token
conventional 8B:         36 × 8 × 128 × 2     =  73,728 dims/token
                                              →  3.0× SMALLER

MLA isn't an absent target. It's a compressed one — which is what a linear map likes best. And the design gets cleaner, not messier: the latent is position-free by construction (that's exactly why DeepSeek decouples the rope key), so only the small 64-dim tensor carries a rotation to strip and re-apply. On a conventional model you have to de-rotate every key.

The correction that mattered wasn't the code, it was the category. Our pair-checker had two verdicts — works and impossible — and MLA got filed under impossible. It now has three:

Run it across our whole model fleet and zero pairs are impossible. Every blocked pair now names the capability that would unblock it, with a count. That's a backlog. "Impossible" is a full stop, and filing work as physics is how a tractable capability never gets built.

The mapper finds the layers by itself

Here's the result that made us trust the fit.

Two models have different depths — 30 layers and 24, or 40 and 64 — so there's no natural pairing between them. Each target layer independently ranks every source layer by held-out R² and keeps the best handful. There is no alignment prior anywhere in the procedure.

What it picks:

target layer  0  ->  source [0, 1, 2, 3, 4, 5, 6, 11]
target layer  9  ->  source [12, 13, 14, 15, 16, 17, 18, 19]
target layer 20  ->  source [19, 21, 22, 23, 24, 25, 26, 28]

A clean monotonic march. Two models trained separately, and a ridge regression recovers the depth correspondence between them unsupervised. That's not plumbing — it's evidence the two networks build recognisably the same thing at recognisably the same relative depths, and it fell out for free.

The part we're actually proud of: it refuses

Anyone can implement the ridge regression from the paper. The reason we think this library is worth using is the layer almost nobody ships: it will not load a mapper that hasn't proven itself.

This matters more here than in most systems, because of how a converted cache fails. It doesn't throw. It doesn't 500. The receiving model produces fluent, on-topic, confidently wrong text — with a green healthcheck, a normal latency profile and nothing in a log. Every cheap signal says fine.

So load_pack() refuses on six separate rules, and the load-bearing one is that a mapper must carry a downstream measurement — not just a reconstruction score. And that measurement has four arms:

The control arm is the whole game. A mapper that has quietly learned the target's average key and value statistics — and ignores its input entirely — scores respectably against reference alone and looks like a working feature. Only a control tells you whether the cache content is doing anything. Without floors, "68% agreement" is unreadable; it could be excellent, or exactly what a dead mapper gets.

We also enforce a minimum sample size. Top-1 agreement is a proportion, so a mean over a couple dozen positions carries a standard error near ten percentage points — and once it's written into a manifest, it is indistinguishable from a mean over ten thousand. A number that can't tell pass from fail is not evidence, and it passes a threshold check exactly as well as a real one.

Two more preconditions are definitional rather than tunable:

• The two models must tokenize identically. The map sends position i to position i. Different tokenizers mean row i isn't the same token, and you're regressing misaligned data. Nothing downstream can see this — the shapes agree, the fit converges, and the mediocre score reads as "this pair transfers poorly" rather than "these rows don't correspond." Every published pair shares a tokenizer by virtue of being one family, which is exactly why the requirement goes unwritten. We check it by vocabulary digest and by comparing token ids per document at capture time.
• The RoPE schedule must be exactly reproducible. Keys carry a position rotation, so the map is fitted in position-free space — stripped with the source schedule, re-applied with the target's. Not a no-op: the two models usually disagree on rope_theta. We implement default, linear and YaRN, and refuse anything else rather than approximate it, because a wrong frequency schedule corrupts the fit silently.

That last refusal has teeth. YaRN scales cos and sin by an attention factor in some configurations, which makes it a scaled rotation — and its inverse is then not "negate the sine." Strip-and-reapply would come back wrong by a constant factor on every single key, uniformly, with nothing to notice. We refuse those configs rather than ship an inverse that's quietly incorrect.

Shipping it found a leak — and that's the system working

Publishing is where a codebase gets audited whether you like it or not.

Our publish pipeline has a gate that scans every payload for secrets and internal artifacts before anything leaves. It passed. Then a manual grep of the payload found five internal references our own gate had waved through.

The gate hunts credentials. It had no rule for a docstring citing an internal path, or a directory layout, or an internal tracking id. Those aren't secrets — they're the shape of the platform, which incidents happened and roughly how the tree is organised — and they ship silently under a green light.

That's now a check, with a self-test pinning both halves: strings that must trip it, and strings that must not, because a rule that fires on the word "docs" floods and a flooding rule gets switched off within a week.

Turning it on immediately found references that were already public — including one in a shared stylesheet that gets inlined verbatim into every generated docs site, quietly republishing the same string across four repositories. All cleaned. Internal tracking ids now get stripped from the published copy automatically while staying in our source, where they're the traceability that makes a comment useful.

And the new tool was born gitignored — the directory it lives in is ignored with an explicit allowlist — so the pipeline would have called a script that didn't exist in CI, failing at publish time. A different existing gate caught that within a minute.

None of this is a bad day. This is three independent controls catching what one human grep found, and converting it into something that can't recur. The measure of a release pipeline isn't that nothing was wrong. It's how fast wrong becomes impossible.

Then we pointed it at ourselves

Shipping a new engine into a library raises an obvious question about the ones already in it: are those claims measured, or inherited from a paper abstract and a README written a year ago?

So we audited them against real keys and values pulled from a live model — not gaussian noise, which flatters vector quantizers because it has no outliers and no per-channel structure — and compared against the dumbest baseline anyone could write instead: per-vector uniform min/max quantization at the same bit width. If a compression scheme ties ten lines of round(), it isn't a contribution, it's a dependency.

TurboQuant came out clean. The ratios match the README to two decimals, and the quality margin is real:

At 2-bit on keys the naive baseline posts a relative MSE of 1.31 — worse than predicting zeros. TurboQuant holds 0.114 at cosine 0.941. And keys are exactly where you want the margin, because key error goes through a softmax.

TriAttention needed correcting, and the reason is the interesting part. Its compression is better than advertised — 14–26× rather than "~10×". But we measured the wrong thing for years by measuring reconstruction:

F=12 (the shipped default), uncalibrated model, 64 query directions
  cosine similarity   0.913   <- looks healthy
  top-32 overlap      0.41    <- fewer than half the attended tokens match

Spectral truncation drops components that are small in norm and large in discrimination. Cosine similarity flatters it; the ranking moves. And attention is a ranking operation — reordering which tokens get attended to isn't bounded approximation error, it's a different model.

The actual defect wasn't the number, it was the silence: which frequency pairs carry the energy is a property of the model, and the config helper fell back to generic selection for any unrecognised model without warning — precisely when it matters most. It now raises, naming the measured figure, with two tests pinning it: unknown models must warn, and calibrated models must not. That second one is the mutation guard. A fallback that warns unconditionally passes the first test while training everyone to ignore the warning.

The README and the docs site now carry the measured numbers instead of the round one.

That's the standard we're trying to hold: the same skepticism aimed at our own shipped claims that we aim at a new one. A library that refuses unproven mappers while quietly advertising an unproven compression ratio isn't rigorous, it's selective.

What it took

One person. No budget. A GPU that was already busy, and a torch install that can't see it — every measurement in this post ran on CPU.

Paper to measured result to shipped open-source release with regenerated docs, in a day. Not because of cleverness: because the boring infrastructure was already there. A content-addressed KV plane. A publish pipeline with a truth gate that derives claims from the payload instead of trusting the copy. A culture where every checker must prove it can still fail.

That's the whole thesis. Build the unglamorous plane once, and a one-person lab moves at the speed of its ideas instead of the speed of its setup.

github.com/Aitherium/aitherkvcache · docs · pip install "aither-kvcache[transfer]"