Two Machines, One Model: The Mesh That Splits DeepSeek

Two machines in a room are running a 284-billion-parameter model that neither of them can hold.

Not a distilled version. Not a 7B stand-in. DeepSeek-V4-Flash, 91 gigabytes of weights, a frontier-scale mixture-of-experts model — served through one API endpoint at 17.9 to 22.3 tokens per second, which is faster than you can read this sentence. The DGX Spark holds one contiguous range of layers. The RTX 5090 holds the next. They pass a token forward and answer as a single system.

The machine that could hold the whole model manages 23.0 tok/s. Splitting it across two machines costs about ten percent.

That number is the headline, and it is worth sitting with for a second. The received wisdom about distributed inference is that the network eats you alive — that the moment a model spans machines you are in a different performance regime, paying interconnect tax on every token, which is why this work is supposed to require NVLink and a datacenter. Two consumer boxes on ordinary 1 GbE Ethernet lose ten percent.

Eight kilobytes

Here is the fact the whole thing rests on.

When you split a transformer between two machines at a block boundary, the only thing that has to cross the wire is the hidden state: one vector, 4096 dimensions, two bytes each. 8 KB. At 23 tokens per second that is 184 KB/s against a link that carries 125,000 KB/s — 0.15% utilisation. A 1 GbE round trip is 0.2–0.5 ms against a 43 ms token budget, so one crossing costs about 1% of a token.

The network is not the bottleneck. The network is barely awake.

That is the counter-intuitive heart of it: the expensive-sounding part of distributed inference — moving data between machines — turns out to be nearly free, provided you cut the model in the right place. Everything else in this post follows from choosing that place correctly.

Granularity is the whole game

The same two machines, the same weights, the same network, cut two different ways:

An order of magnitude between the top and the bottom, from nothing but where you cut.

The bottom row splits inside a block, at the tensor level. That means the expert weights themselves stream across the link on every single token — gigabytes of traffic to compute one word. The top row splits between blocks, and ships the 8 KB activation. The two message sizes differ by roughly six orders of magnitude, and that difference is the entire spread in the table.

Cut the model where the tensor is small. Transformer block boundaries are exactly those places.

Stage splits

The architecture treats each machine as a stage, not a tensor recipient:

• The coordinator plans contiguous layer ranges — this node owns layers 0–18, that node owns layers 19–36.
• Each device loads only its stage. The model is pre-split into per-layer fragments, so a joining node downloads a few gigabytes, not 91.
• Downstream stages start first; stage 0 publishes its route only once every stage reports ready.
• Activations cross the wire once per token.

Measured through the mesh: 17.9–22.3 tok/s with the RTX 5090 carrying layers 19–36, over a direct peer link at 4–20 ms, with a public relay fallback so a node behind a firewall still joins.

What it took to package a 91 GB model

Three real traps in the toolchain, each a lesson a fresh node should never have to re-learn:

1. --keep-quant was broken. It preserved the scratch directory but still re-ran the quantizer, which for 2-bit tensors demands an importance matrix and aborts. Fixed in three lines so an already-quantized model keeps its fragments as-is.
2. A stale static archive broke the linker with an undefined symbol that only a forced re-archive resolved.
3. The shard layout is specific — source shards must live in a directory named after the model prefix, not flat.

None of this is in the mesh-join path a user sees. That is the point.

The part that scales

The mechanism that carries this model across two machines is the same one that carries it across two hundred. Adding a node means: it joins the mesh with an invite token, the coordinator re-plans the contiguous layer ranges across the now-larger pool, the new node downloads only its assigned stage's fragments, and stage 0 publishes a new route. The topology planner does this automatically from a live view of each device's resources and the link latency between them.

Nothing in that loop knows or cares whether it is arranging two machines or two hundred.

The onboarding — the thing a non-technical user should never think about — is being reduced to a single command. Platform detection, bundle download, container-or-native runtime, the join, the layer fetch, registration: all handled. The deployment matrix that cost days to learn (Blackwell GPUs need a CUDA 13 runtime the Windows build doesn't ship; glibc 2.38; a CUDA runtime image that omits libcudart) is a decision tree the skill hides.

What this actually means

The rule has been that running a frontier-scale model requires a machine that can hold a frontier-scale model, and those machines are not for sale to ordinary people at ordinary prices. That rule is what makes serving these models the business of a handful of organisations with datacenters.

It turns out to be a rule about packaging, not about physics. A 91 GB model is a sequence of transformer blocks. Each block is a few gigabytes. Between any two of them the traffic is 8 KB. Nothing requires those blocks to sit in the same box, and the measurement says you lose about ten percent when they don't.

Two desktop machines, one 1 GbE cable, 284 billion parameters, 22 tokens per second. No datacenter, no NVLink, no cloud bill, no permission from anybody.

For a model that genuinely fits on one box, one box is still the right answer — stage traffic is pure overhead there. The split earns its keep exactly where it is needed: when the model is bigger than any single device you own. Which is the direction every model is going, and the reason this was worth building.

The next measurement is the one that matters: the 91-gigabyte DeepSeek-V4-Flash, split, on the mesh, with a swarm of concurrent requests across peers — the aggregate-throughput number that proves whether adding machines makes the system faster, not merely bigger. That experiment is armed.