Running a 2.78-Trillion-Parameter Model Across Two Desks

Kimi K3 has 2.78 trillion parameters. The checkpoint is 1.56 terabytes. The usual answer to "can I run this at home" is a laugh track.

Last night it answered "The capital of France is" with " Paris." — on a desk. Tonight it runs split across two desks, with the experts of a mixture-of-experts model living in three tiers of an RTX 5090 box across the room from the machine doing the thinking. This post is the build log: what we measured, what broke, the one bug that made the whole thing slower before it made it faster, and why the model is now in the loop improving its own serving stack.

Standing on shoulders first: the engine is kimi-k3-in-c — Fareed Khan's astonishing portable-C99 implementation that streams the whole model through as little as 8 GB of RAM, byte-identical output at every memory budget. We ported it, split it, and taught it to use a GPU across the room. Everything below builds on that foundation and keeps its central discipline: a speedup that changes the output is not a speedup, it is a broken engine.

Act I: first tokens on an ARM box the engine never targeted

The engine targets x86-64 Linux with AVX2. Our serving box is a DGX Spark — GB10, aarch64, 20 Grace cores, 121 GB of unified memory, no AVX anything.

So the first job was a NEON port of the hot kernels, under the engine's own bit-exactness oracle: an FNV1a hash over every output float. Same hash or it does not ship. The port passed exact and the first real tokens came out at:

A 20-core ARM desk unit beating a 124-core EPYC on efficiency, running a model 12x larger than the machine's entire RAM. As far as we can tell these are the first Kimi K3 tokens ever generated on a DGX Spark.

Two supporting pieces landed the same night. A content-addressed prefix cache — session states keyed by the hash of the token sequence — made follow-up turns 3.9x faster. And every request now feeds an expert-heat histogram, because MoE routing is a measurable, exploitable signal — hold that thought.

Act II: the design — experts live on the other machine

Here is the architecture decision that makes this interesting. Kimi K3 is mixture-of-experts: 82,432 routed experts, 17.55 MB each, of which only 16 per layer fire for any token. 93% of the checkpoint is experts that are asleep at any moment.

The engine streams them from NVMe. Our Spark's NVMe is fast (9.5 GB/s at the expert access pattern) — and still the expert term dominated: 6.64 s of every token was expert math and expert I/O, against 4.44 s of trunk.

Across the room sits a box with an RTX 5090: 32 GB of GDDR7 at 1.8 TB/s, 128 GB of DDR5, a 24-core CPU, and a big NVMe. The design: that box becomes the expert tier. Not a fallback, not a cache — the place experts live:

• VRAM tier: the hottest experts, dequantized to bf16, GPU computes them
• DDR5 tier: the next set, bf16 — and the 24 CPU cores compute these directly, in parallel with the GPU arm. RAM is serving capacity, not a waiting room.
• NVMe tier: a 58 GB heat-ranked slice — 3,500 experts covering all 92 MoE layers — extracted byte-exact from the checkpoint.

The trick that makes the wire disappear: Kimi K3's experts run in a latent space of width 3,584. So per MoE layer, the Spark ships one activation vector — 14.3 KB — and gets back the weighted sum of every expert the far box holds. Weights never cross the network. One round trip per layer, overlapped with local compute: the Spark fires the request, computes its own share of experts while the wire and the far GPU work, then joins.

Measured on the real lane:

That last pair matters as much as the speed. Bit-exactness across two machines is mathematically impossible (floating-point summation order), so the cross-machine gate is a measured numeric tolerance against an independent oracle — and both compute arms sit at fp32 noise. We found and killed a real precision bug on the way: the first GPU path ran matmuls in bf16 and failed its own tolerance gate at 5.8e-03. The fix was not loosening the gate — it was computing in fp32 with bf16 storage. The gate held; the code moved.

Act III: the bug that made it slower, and the contract that fixed it

First full split benchmark: the split was slower than local. Warm tokens at 21–22 s, spikes to 59 s, against 12.5 s steady local. Something was deeply wrong, and it was ours.

The expert server, asked for an expert it hadn't loaded yet, would read it from NVMe and dequantize it while the engine waited — 400 to 1000 ms. The Spark computes that same expert locally in 4.5 ms. Shipping a cold expert was a hundred times worse than not shipping it.

The fix is a contract, not a patch: the tier may only help or stand aside — in latency, not just correctness. A cold expert now comes back instantly as "missing" (the engine computes it in milliseconds, exactly as it would if the far box didn't exist), and the server warms it in the background for the next token. The failure mode collapsed to "no worse than local"; the win path stayed.

We could prove the stand-aside half brutally: with the lane deliberately dead, the split run produced bit-identical logits to the local run — all 163,840 of them, max diff 0.000e+00. The tier cannot corrupt a token by absence. That is what fail-soft means.

The heat is real

Remember the histogram? Quantile-balanced MoE routing is deliberately cache-hostile — flat usage defeats LRU, and the engine's own author measured LRU dead-flat at every size. But measured heat still beats no heat: replaying our real traffic traces, pinning the hottest experts hits 48.7% where pure LRU manages 36.2%. Every serving request sharpens the histogram, the histogram builds the pin lists, the pin lists decide which experts live in GDDR7 versus DDR5 versus NVMe — on both machines. The system literally learns where its own weights should sleep.

Act IV: the model is now in the loop

The newest piece: a self-improvement loop where Kimi K3 is the research brain for its own serving stack. Each cycle, the serving model reads its own live metrics — tokens per second, tier residency, heat coverage, its last few experiments — and picks one bounded experiment from a fixed menu: re-rank the pin list, deepen the far tier's warm set, stage speculative decoding, rebalance the trunk/cache split, or simply re-measure.

The loop's rules are the whole point:

1. A proposal is never a result. Every experiment ends in a measurement through the real serving surface, or a bench scorer that refuses to emit a metric at all if the output hash changed.
2. Deltas under the noise floor are refused in both directions. We once measured a 14.6% "regression" caused entirely by a background download. An optimizer that learns from noise learns confidently and wrong.
3. Wins are staged, never auto-applied. Serving restarts are a deliberate human act.
4. A cycle that cannot measure fails loudly. Silence is not a pass.

The objective is pinned in the config: 1.0 second per token. And it's arithmetic, not a wish: the far tier collapses the expert term from 6.64 s toward 0.64; speculative decoding multiplies by its measured 60–85% draft acceptance; the last step — moving the trunk's bf16 matmuls onto the GB10's own GPU — takes the 4.44 s trunk term toward ~0.5. Multiply it out and the floor lands between 0.4 and 0.7 s/token, on hardware that fits on two desks.

Where it stands tonight

Serving is live at ~12.5 s/token steady on the Spark alone. The expert tier is proven at the engine level — correctness gates green on both arms, fail-soft proven bit-exact, the warm path measured at layer level — and the first clean end-to-end warm benchmark of the fixed contract is queued. The honest number to watch next is steady-state split s/token; the projection says single digits, and the projection is made of measured parts.

The through-line, same as everything we build: every claim above is a number someone can re-measure, every gate can fail, and the system that serves the model is also the system improving it. A trillion-parameter model should not require a trillion-dollar rack. It requires two desks, a network cable, and the discipline to never call a difference smaller than the noise a result.