The KV-Cache Translator That Couldn't Exist

We run two orchestrators.

The fast one is an 8B model on a 5090 with a 16,384-token context window. It answers the interactive turns — the "say OK", the greeting, the one-line question — and it answers them in a couple of seconds.

The slow one is a 27B model on a DGX with a 131,072-token window. It handles the turns the fast one physically cannot hold: the long agentic loop, the 30,000-token document, the ReAct trace that has accumulated forty tool calls.

The split works. What did not work was the moment of transition.

The moment of transition

Here is the failure, from our own production logs:

This model's maximum context length is 16384 tokens.
However, you requested 4096 output tokens and your prompt contains
at least 12289 input tokens, for a total of at least 16385 tokens.

One token over. The request 400s, the agent loop aborts, and the chat success-rate SLO drops to 0.913 against a 0.995 target.

The first fix was obvious and we shipped it: count the input better (our estimator was ignoring ReAct tool schemas, which run to thousands of tokens of JSON), then clamp the output budget, then — if the prompt alone cannot fit — escalate to the big model.

That worked. The 30K-token prompt that used to abort now answers correctly. It just takes 47 seconds.

Why 47 seconds? Because the big model has never seen this conversation. Every one of those 30,000 tokens has to be pushed through 27 billion parameters to build the attention keys and values before a single output token can be sampled. That is prefill, and prefill is most of the wall clock on a long prompt.

Meanwhile, three feet away, the fast model has most of that same conversation already in its KV cache, warm, computed, sitting in VRAM.

So: hand it over. Translate the cache.

Why the translator cannot exist

It is a genuinely appealing idea, and it is genuinely impossible. Not "hard." Not "unimplemented." Impossible in the way that pouring a gallon into a pint glass is impossible.

A KV cache is not a document. It is a set of tensors whose shape is determined by the model's architecture: number of layers, number of attention heads, head dimension. Our 8B fast model and our 27B slow model differ in all three. The cache from one is not a compressed version of the cache from the other, and it is not a lossy version either — it is a different object that happens to have been produced by reading the same text.

You could imagine a learned adapter that maps one model's KV space into another's. People have published in that direction. But you would be training and running a model to avoid running a model, and the adapter would have to be retrained every time either endpoint changed. For a fleet where models get swapped, quantized, and re-tuned weekly, that is not an optimization. That is a second problem.

So the honest conclusion is: there is no physical translation between these two caches.

Which is where the interesting question starts, because "no physical translation" is not the same as "no translation."

What is portable

Step back from the tensors and ask what the two models actually share.

They share the token sequence. The same conversation, the same system prompt, the same tool schemas, the same message structure — all of it is identical text on both sides of the escalation. The tokens are the portable format. They always were.

And here is the part that makes it exploitable: modern inference servers already cache by token prefix. vLLM's --enable-prefix-caching — which, we discovered during this work, was already switched on across our entire fleet and quietly doing nothing for us — hashes incoming prompt prefixes and reuses the computed KV blocks when a later request shares a prefix with an earlier one.

Read that again with the escalation in mind. The big model does not need our cache. It needs its own cache to already contain this conversation's prefix. And there is exactly one way to put a prefix in a model's cache: send it the tokens.

So the translator is real. It just does not move bytes between GPUs. It moves time.

Warm-ahead

The mechanism we shipped is about forty lines, and it sits at the point in our scheduler where a request's final backend has been chosen but no work has started:

When a session's estimated input crosses 60% of the fast model's window, fire a one-token request at the big model with the same message structure — and don't wait for it.

That request costs almost nothing to us: max_tokens: 1, fire-and-forget, no one is waiting on the reply. But on the DGX it does the whole prefill and leaves the result in the prefix cache. When the conversation grows two more turns and finally overflows the fast window, the escalated request arrives at a model that has already read almost all of it. The prefix hits. The prefill is mostly skipped.

We are not caching an answer. We are pre-paying an unavoidable cost during the seconds a user spends typing, instead of during the seconds they spend waiting.

The guards matter more than the mechanism:

• 60% threshold. Not 90%. By the time you are at 90% you are probably already escalating, and the warm arrives too late to help.
• One warm per session per five minutes. Without a cooldown, a chatty session would fire a warm on every turn and you would have built a DDoS against your own DGX.
• Bounded cooldown map. A dict keyed by session ID grows forever unless you prune it. We prune to 256 entries.
• Skips when the served model is unknown. A warm-ahead that guesses the model name is a wasted round-trip at best.
• Environment-gated. AITHER_KV_WARM_AHEAD=0 turns the whole thing off if it ever misbehaves. Optimizations that cannot be disabled are liabilities.

None of that is clever. All of it is the difference between a nice idea and something you can leave running.

The other half: don't escalate reactively

Warm-ahead only pays off if the escalation itself is smart, and ours was not. It was reactive: send the request to the fast model, let it 400, then re-dispatch. That is a wasted round-trip on the exact requests that are already the slowest.

So the same change added proactive routing. At the same dispatch seam, we now estimate the input, compare it against the fast backend's real context window (discovered from the server and cached with a 15-minute TTL, because our swap slot serves different models over time), and if it clearly will not fit, we route to the big model first. No doomed attempt.

From the live logs after deploy:

[CONTEXT-ROUTE] proactive: est≈59863tok + 256 output floor exceeds
vllm window 16384 → routing to 'vllm_dgx'

Same probe as before: 47.2 seconds instead of 61.6, and zero failed local attempts along the way.

One deliberate asymmetry is worth calling out. Our estimator counts characters and divides — a rough approximation of tokens. For clamping an output budget we divide by 3, which over-estimates the input and is therefore safe. For routing we divide by 4, which under-estimates. That is not an inconsistency; it is a bias with a direction. An over-eager router sends borderline interactive turns to a 47-second backend, which is a far worse user experience than being wrong occasionally and letting the reactive backstop catch it. When your estimate is uncertain, be wrong in the direction that costs less.

What the adversarial review caught

We ran the change past three independent reviewers, each given a different lens and instructed to try to refute the design rather than approve it. They found three real problems before any of it reached production.

The catalog bypass. When escalation targets a cloud model instead of the DGX, the model name has to exist in that provider's catalog. The dispatch pipeline validates this — but only for a subset of backends, and our new reroute path was not in that subset. A request could be routed to a provider that had never heard of its model, and the failure would surface deep inside the cloud call instead of at the routing decision. Now the reroute checks the catalog itself and keeps the request local if the model is unmappable.

The stale window. We cache each backend's context window by name. But one of our backends is a swap slot that serves different models at different times. A window cached for a 30K model would be wrong the moment a 4K model loaded in its place. Fifteen-minute TTL. A stale miss costs one reactive round-trip; a stale hit costs a delayed optimization. Both are survivable, and neither is silent.

The double count. Our estimator summed the prompt field and the messages field. But the payload builder uses one or the other — messages if present, prompt otherwise. On requests that carried both, we were over-estimating input by roughly 40%, which would have sent a meaningful band of perfectly-fine interactive requests to the slow tier for no reason. The estimator now mirrors the payload's exclusive-or.

Every one of those would have shipped. None of them would have thrown an exception. They would have shown up as "the system feels slower than it should" and cost days.

The finding underneath the finding

While mapping the KV surfaces for this work, we sent five readers through the codebase in parallel — one for the cache graph, one for the vector store, one for routing, one for dispatch, one for the physical backends. What came back reordered our roadmap more than the feature did.

Prefix caching was already on, fleet-wide, and completely unmeasured. Every vLLM server carried the flag. Nothing reported hit rates. Nothing consumed them. The single highest-leverage thing we could do for KV efficiency was not to build something — it was to route consistently so the caching we already had would hit.

A large block of KV-cache "infrastructure" was dead wiring. Functions to report prefix-cache hits: never called. Prompt layouts registered into a bridge: never queried. A snapshot mechanism that emitted an event with no listener. This is what aspiration looks like six months later — plausible names, real code, zero live consumers. Deleting it is a decision for another day; knowing it was inert changed what we built today.

And one live security gap. The cache graph's eviction ranking was keyed by block index alone, with no tenant scoping on the query path. Any caller that omitted a tenant got a ranking computed across all tenants' blocks. Nothing consumed that ranking yet, so nothing was exploitable — but the moment something did, one tenant's activity could have steered eviction of another's cache. That is now fail-closed: a query without a tenant is denied unless the caller explicitly declares itself the physical VRAM manager, which is the one component for which a global view is genuinely correct.

We went looking for a performance feature and came back with a security fix. That is not unusual. It is the normal yield of actually reading the system you think you know.

What generalizes

If you are running a fast/slow model split — and increasingly, everyone is — three things here transfer:

Tokens are the portable cache format. Do not try to move KV between heterogeneous models. Move the token prefix and let each model's own prefix cache do the work. The abstraction you want already exists inside your inference server.

Pre-pay costs during human latency. The seconds between a user's turns are free compute. A speculative one-token prefill during typing is invisible; the same prefill during generation is 40 seconds of dead air.

Route before you fail, but keep the failure path. Proactive routing needs an estimate, and estimates are wrong. Ours is a character count, which is always somewhat wrong. That is fine as long as the reactive path — parse the server's own numbers out of the rejection, clamp, retry, escalate — is still underneath it, authoritative and correct. Optimism on top, certainty below.

The translator we wanted was impossible. The one we built moves nothing at all — and it is the one that works.