24 Tokens a Second: A 284B Model on a Desk-Sized Box

Every claim in this post is a number we measured this week on hardware we already own — a 121 GB unified-memory DGX Spark and an RTX 5090, joined by plain Ethernet. No rented cluster, no cloud bill, no spec-sheet math.

The headline

DeepSeek-V4-Flash — 284B parameters, IQ2 at 90.9 GB — served from one desk-sized box at 24.2 tok/s median, with a 0.6B DSpark drafter proposing and the 284B verifying. The drafter's acceptance is not a rumor: draft_n 248–390 across the benchmark runs, spec decoding engaged and working.

The measured ladder — every rung this week

The ladder is the story, and the answer is not the one the tok/s column alone would pick. Without a drafter, the split loses 43% to a single box — the RPC pipeline serializes each token (~56 ms/token with both GPUs half-idle, zero I/O, sub-millisecond latency). With a drafter, speculative decoding batches the verification step and amortizes that serialization away: the split's 17.9–22.3 is real, and it is why the claim that "a 135 GB model doesn't fit on one Spark" is wrong by construction.

The part nobody puts in the benchmark post

Every number above assumes the hardware is yours alone. Ours isn't — it runs a platform. The 284B pool shares the big box with a 12B vision model, and the second card is shared with the chat orchestrator, an on-device model runtime, and an embeddings lane.

So we measured the thing that actually decides the config: who else is resident. The second card is 32.6 GB and the platform's own lanes hold 27.9 of it — 4.3 GB free, about two of this model's layers. Offloading two layers frees nothing worth having on the big box, and a two-layer split measures worse than no split at all. The split earns its keep at ~12 layers (~25 GB), and that is a card the platform would have to give back.

That is not a knob, it's a decision, and the honest version of this post says so: you can have the drafter, or the co-resident vision model, or the split — any two of the three, on this hardware. Tonight the box runs the 284B with the vision model beside it at 14.2 tok/s (five runs: 14.21 / 13.68 / 14.45 / 14.61 / 14.24), because a memory gate measured 2 GB of headroom and refused the drafter rather than thrash. The 24.2 number is real and repeatable; it costs the vision lane, while the 14.2 number is the baseline with the vision model co-resident.

The drafter is the multiplier in both configs

The drafter is a 10.9 GB dense 0.6B model that shares the DeepSeek tokenizer. Co-located, it proposes up to 3 candidate tokens per step; the 284B verifies them in one forward pass. That's the 1.77× — and it costs only memory, not a second machine.

The memory gate is the whole decision: model + drafter + process ≈ 115 GB on a 121 GB box leaves 3–8 GB of real headroom — the proven working fit. The arithmetic has to measure MemAvailable, not MemTotal; sizing against total RAM "proves" 27 GB of headroom while a co-resident 28 GB tenant makes the true figure negative — and the OOM-kill is the receipt for that mistake.

The platform it lives in

The pool is one lane of AitherOS, the local-first AI platform this runs on. The same stack follows you: one URL installs the omnibox on a fresh laptop, retrieval and sessions answer from home through a tunnel, browser agents run on-device (Bonsai, WebGPU, fully offline in a tab), and every session event is sealed and pushed to a git repo in real time. Nine in ten turns never leave the house; renting the cloud is a decision, not a default.

When the box is crowded, the pool heals itself: a watchdog on cron judges the RPC session socket — never /health alone, which stays green while the decode path is dead — and converges the lanes by measurement, with every artifact committed and every decision carrying its number in a comment.

The recipe is public

The whole thing — flags, traps, the seven failure classes we hit so you don't have to — is written up as a runnable recipe: whole-block offload placement (never per-tensor), the --spec-type silent-noop trap, the -np 1 shared-KV rule, and the durability pattern that survives reboots. Measure first; the numbers that look obvious are the ones that were wrong.