Two Machines, One Model: The Mesh That Splits DeepSeek

A 91-gigabyte mixture-of-experts model split across two machines answers through a mesh at roughly the throughput of a single machine — because each machine runs a contiguous block of the model on its full compute, and only a few kilobytes of activations cross the wire per token. The DGX Spark holds the first half of the layers. The RTX 5090 holds the second. They agree on a token, pass it forward, and serve it as one API.

This is the second chapter of a story that started with "The CPU Was Never the Passenger": one ARM64 machine serving DeepSeek at 23–30 tokens per second. The question this time was simpler and harder at once — what happens when the model no longer fits?

The wrong answer taught us the right granularity

The first attempt at using a second machine was the obvious one: offload a few layers to the 5090 over ggml-RPC, the per-tensor remote protocol. It crashed. Not a config error — a real defect in the RPC backend's buffer allocation, one that aborts the coordinator on the first decode. We fixed it, and then we measured it: 1.9 tokens per second. The 5090 was doing real compute, and the wire was the bottleneck — nine round-trips per token, one per expert matmul, each a network hop through Docker's port proxy. The GPU was fast. The granularity was wrong.

Stage splits: the coarse-grained answer

The architecture that actually works treats each machine as a stage, not a tensor recipient:

• The coordinator plans contiguous layer ranges — this node owns layers 0–18, that node owns layers 19–36.
• Each device loads only its stage — the model is pre-split into per-layer fragments, so a joining node downloads a few gigabytes, not 91.
• The downstream stages start first; stage-0 only publishes its route once every stage reports ready.
• Activations cross the wire once per token, a few kilobytes each.

The result, measured through the mesh: 17.9–22.3 tokens per second across two machines, with the RTX 5090 carrying layers 19–36. The peer link is direct and low-latency — 4–20 milliseconds — with a public relay fallback so a node behind a firewall still joins.

What it took to package a 91 GB model

Turning DeepSeek-V4-Flash into a stage-splittable package exposed three real traps in the toolchain — each one a lesson a fresh node should never have to re-learn:

1. --keep-quant was broken. It preserved the scratch directory but still re-ran the quantizer, which for 2-bit tensors demands an importance matrix and aborts. The fix is a three-line change so an already-quantized model keeps its fragments as-is.
2. A stale static archive broke the linker with an undefined symbol that only a forced re-archive resolved.
3. The shard layout is specific — source shards must live in a directory named after the model prefix, not flat.

None of this is in the mesh-join path a user sees. That is the point.

The part that scales

The mechanism that carries this model across two machines is the same one that carries it across two hundred. Adding a node means: it joins the mesh with an invite token, the coordinator re-plans the contiguous layer ranges across the now-larger pool, the new node downloads only its assigned stage's fragments, and stage-0 publishes a new route. The topology planner does this automatically from a live view of each device's resources and the link latency between them.

The onboarding — the thing a non-technical user should never think about — is being reduced to a single command. Platform detection, bundle download, container-or-native runtime, the join, the layer fetch, registration: all handled. The deployment matrix that cost days to learn (Blackwell GPUs need a CUDA 13 runtime the Windows build doesn't ship; glibc 2.38; a CUDA runtime image that omits libcudart) is a decision tree the skill hides.

The honest numbers

For a model that fits on one box, one box is still the right answer — stage traffic is overhead. The split earns its keep exactly where it's needed: when the model is bigger than any single device, which is the direction the models are going. And the throughput penalty of the split is what you pay for the privilege of running something you otherwise couldn't run at all.

The next measurement is the one that matters: the 91-gigabyte DeepSeek-V4-Flash, split, on the mesh, with a swarm of concurrent requests across peers — the aggregate-throughput number that proves whether adding machines makes the system faster, not just bigger. That experiment is armed.