Five Ways to Shrink a 2.78-Trillion-Parameter Model

Kimi K3 is 2.78 trillion parameters. Ninety-three layers, 896 experts per layer, top-16 routing, 1.56 TB of weights on disk. The published way to run it is a rack.

I want it answering at 5 tokens per second on a DGX Spark and an RTX 5090 — hardware that fits under a desk and draws less than a hair dryer.

Five research angles are open at once. Four already have numbers against the real weights. This is what each one is, what it measured, and what it buys.

---

The shape of the problem

Split the model in two, because the two halves fail differently.

The trunk is small and always hot. The experts are enormous and mostly cold. A token touches roughly 105.9B of the 2.78T parameters — so the model is far cheaper to run than to store, and the entire engineering problem is getting the right 82 GB in front of the arithmetic units fast enough.

Measured on our own hardware: 12.8 seconds per token, with expert reads streaming off NVMe. To hit 5 tok/s that has to become 200 ms. That's a 64x gap, and no single trick closes it — which is why there are five angles.

---

Angle 1 — Ternary trunk distillation

The trunk is 108.81 GB in bf16 and 55.86 GB as int8. The question is how much further it goes before the model degrades.

Naive rounding to ternary — every weight becomes −1, 0 or +1, about 1.58 bits — is a disaster. Measured on a real K3 trunk matrix: 51.04% output error. The model would be noise.

But rounding isn't the only option. You can train the ternary weights to match the original layer's behaviour, learning a per-group scale alongside them. On the same real matrix:

Training loss fell from 0.02683 to 0.001247. 11% error at 1.58 bits is roughly what int4 delivers at 4 bits — the same accuracy for 2.5x fewer bits.

What that buys: a ternary trunk is about 10.7 GB against int8's 55.86 GB. At the DGX's measured 240 GB/s that's 45 ms of trunk reads per token instead of 233 ms. The trunk stops being part of the problem.

The subtlety worth stating: the learned group scale is what makes this work. An earlier tool that optimised the shadow weights alone diverged — reporting negative recovery, worse than the rounding it was meant to beat. Learning the scale jointly is the difference between 51% and 11%.

---

Angle 2 — Does error compound across 93 layers?

This is the objection that kills aggressive quantization on paper. If each layer introduces 11% error and there are 93 layers, the output is garbage — 1.11⁹³ is about 14,000x.

So we measured it. Feed a layer a clean input and a perturbed one, and see whether the perturbation grows or shrinks by the output. Call that ratio A. If A > 1 error compounds; if A < 1 it contracts.

Measured across all three layer families — MLA attention, KDA linear attention, and MLP — with the real norms and residual connections in place:

A = 0.485 to 1.001.

Error contracts. Over 93 layers the cumulative factor lands between 0.12x and 0.32x rather than exploding. RMSNorm is doing the work: it rescales the hidden state every layer, and a perturbation that grows in magnitude gets normalised straight back down. The residual path dilutes what's left.

This matters more than any single quantization result, because it converts "we can't afford error per layer" into "we have an error budget and here is its size." Every other angle spends against that budget.

The measurement was deliberately run across all three families rather than one. Sixty-nine of the 93 layers are KDA, not MLA — a result from attention layers alone would have described a quarter of the model.

---

Angle 3 — Expert pruning

896 experts per layer, 16 fire per token. If routing were uniform, every expert would be equally hot and there'd be nothing to exploit.

Routing is not uniform. Measured on the live engine: 94.9% cache hit rate with 24 GB resident out of a 1.45 TB pool. A working set of 1.6% of the expert pool serves 19 of every 20 token-expert lookups.

That's not a caching curiosity, it's a statement about the model: the overwhelming majority of those 83,328 experts are almost never selected. A pruned K3 that keeps only the hot set would be a fraction of the size and read a fraction of the bytes per token.

This is the angle with the largest headroom and the least work done so far, which is why it's being measured properly now: the exact skew per layer, the size of the top-K set, and — the part that decides whether it ships — what accuracy costs when a rare expert is routed to and isn't there.

---

Angle 4 — A bit-exact CUDA trunk kernel

The trunk GEMV is bandwidth-bound, not compute-bound. That single fact reshapes the kernel design, and it produces a result that looks wrong at first glance.

fp64 arithmetic costs 1.8x, not 64x. On consumer Blackwell the fp64 rate is 1/64th of fp32, so the textbook expectation is catastrophe. But if the kernel spends its time waiting on memory, the arithmetic rate barely enters — and you get double precision for under 2x. Bit-exact determinism becomes affordable.

The measured speedup for the coalesced CUDA trunk kernel over the CPU path: 17.5x.

The coalescing detail is the whole ballgame. A thread-per-row kernel has each thread striding across memory; a warp-per-row kernel has 32 consecutive threads reading 32 consecutive floats, which the memory system coalesces into one transaction. Same arithmetic, an order of magnitude apart.

Determinism isn't decoration here. Fixed summation order means a quantization change can be attributed with certainty — if the output moved, the quantization moved it, not floating-point reassociation. Every measurement above depends on that.

---

Angle 5 — Distilling K3 into smaller models

The other four angles make K3 itself faster. This one makes everything else smarter.

Take K3's outputs as a teacher signal and train the smaller models we already run — a 12B perception model, a low-bit dense chat model, a reasoning finetune — to match them. Standard knowledge distillation, with one correction that matters:

Expert weights cannot be copied across. Different hidden dimension, different tokenizer, different residual space. There is no weight-space mapping between K3 and a 12B dense model. It has to be output distillation — the student learns to reproduce the teacher's logits, not to inherit its parameters.

There's direct precedent in-house: a 27B model distilled from a larger sibling, now serving in-browser at 102 ms to first token and 50 tok/s on WebGPU. The pipeline exists; this is a new teacher for it.

---

What ties them together

Every one of these needs the same thing: real activations from a fully-resident K3.

Ternary distillation fits weights against the actual distribution a layer sees — gaussian noise won't do, because the whole result depends on matching real behaviour. Expert pruning needs real routing traces. Cross-model distillation needs the teacher's real outputs.

Locally, K3 streams 1.45 TB of experts per token at 12.8 s/token, so collecting a useful corpus takes years. On a machine where the model is resident in RAM it takes hours. That one dependency is why the capture step exists at all, and it's the current focus.

---

Where this lands

Four of five are backed by numbers on the real weights. The fifth has a working pipeline waiting for a teacher.

None of them alone gets to 5 tok/s. The trunk work takes the trunk out of the equation; expert pruning attacks the 82 GB that remains; the kernel makes what's left fast. That's the shape of the path, and it is measured rather than assumed at every step so far.

The interesting part isn't any single number. It's that a 2.78-trillion-parameter model turns out to be mostly cold — 1.6% of its experts doing 95% of the work, error that shrinks rather than grows with depth, and a trunk that survives 1.58 bits when you train it properly. A model that large is not 2.78T parameters of essential complexity. It's a much smaller model wearing a very large coat.