The CPU Was Never the Passenger

A 284-billion-parameter mixture-of-experts model running on an ARM64 machine answers structured queries at 28–30 tokens per second. No data center. No cloud API calls. The stack is measured, local, and designed to scale to a second tier of machines without changing what the application sees.

Red Hat recently published "The CPU is back: Rethinking the CPU-GPU split for LLM inference." They propose that GPUs should not handle every phase — prefill and memory-bound KV cache operations belong on modern CPUs so GPUs focus on compute-heavy decoding. An ARM64 unified-memory part like the DGX Spark GB10 makes that question architectural: the CPU and GPU are the same die. The split is not about which chip, it is about which tier of the memory hierarchy.

We have already rethought this. Not hypothetically. The stack is running. Here is what it looks like, measured to the millisecond.

---

Speculative Decoding Delivers Measurable Speedup

DeepSeek-V4-Flash (13 billion active parameters via MoE, 284 billion total) quantized to 2.7 bits per weight (UD-IQ2_M format) resident on the DGX Spark GB10. The coordinator launches a paired drafter — DSpark, the vendor's own lightweight model, quantized to 8 bits — which predicts the next few tokens. The main model scores those predictions. When they match, the batch advances by the draft count at nearly no cost.

Measured throughput:

Structured output hits 30 tokens per second consistently. Prose — where the draft has less agreement with ground truth — trades some speed for correctness. The speed is not constant; it is predictable and tied to a measurable, observable parameter (draft acceptance). That matters more than a headline number.

The full model plus drafter occupies 94 GB of the DGX's 121 GB unified memory, leaving 27 GB headroom. We ran measurements with headroom: the fully-resident, fully-warmed state. The throughput equation is not just the model; it is the memory state.

---

Red Hat's Thesis, Delivered in Arithmetic

Red Hat's central claim: as workloads become more agentic, the CPU:GPU ratio shifts from 1:8 (pure training) toward 1:1 or 4:1. CPUs are better for prefill, for attention compute on memory-bound operations, for the logic between model invocations.

On an ARM64 unified-memory architecture, that distinction dissolves into a tier question. The DGX Spark's 121 GB is DRAM, the fastest tier. An RTX 5090 node (32.6 GB) holds models that fit. A desktop with 125.6 GB DDR5 holds larger frozen checkpoints. The question is not which chip computes first; it is which tier holds a block right now, how fast can we move it if we need it, and do we need it based on what the MoE router just selected.

The prefetch window arithmetic decides everything.

A 43-block MoE model processes one token at a time. Each block is a forward pass. At 28.55 tokens per second, one token takes 35 milliseconds. One block takes 35 ms / 43 = 0.81 milliseconds.

Now ask: how far ahead can a tier prefetch to cover one block's worth of compute?

Intra-pass window: A DRAM-class tier (GB10 unified memory or desktop DDR5) can wait 0.81 ms and fetch a block inside that window. Fast.

Full-pass window: An SSD cannot. SSD latency is 50–100 ms, comfortably faster than a full forward pass (35 ms × 43 = 1,505 ms). An SSD can fetch blocks on a "next pass" schedule: prefetch all 43 blocks while the current pass runs, so the next pass finds them resident.

The ratio: 1,505 ms / 0.81 ms = 1,857×. The same model, the same forward pass count, split across tiers of different latency classes, requires two different prefetch strategies.

The CPU-GPU split Red Hat asks about is really tier-assignment. Which phase (prefill, token generation, attention over KV cache) runs on which processor is less interesting than: which tier can hold which blocks such that promotion latency stays inside the required window?

So the real lever is not "should the GPU decode" — it is "can we predict which block the MoE will use next and have it staged before the token arrives." That is a learned problem, not a policy problem.

---

Three Tiers, One Scheduler

The serving plane spans three machines connected by an AitherMesh overlay (WireGuard, 100.64.0.x). The host firewall blocks the LAN path; the mesh path is the only working route.

• Tier 0: DGX Spark GB10 (121 GB unified memory, ARM64). Holds the frontier reasoning model and handles structured output workloads. Measured latency ~0.1 ms intra-rack.
• Tier 1: RTX 5090 (32.6 GB VRAM). Holds on-demand models — vision, image generation, smaller tuned variants. Orchestrator and perception roles live here. Latency ~2 ms over mesh.
• Tier 2: Desktop DDR5 (125.6 GB). Holds frozen, reference checkpoints and CPU-side inference. Latency ~5–10 ms. Capacity-limited backend, not latency-optimized, but usable for pass-wide prefetch (the 1.8-second window, not the 1-millisecond window).

MicroScheduler (:8150) routes every LLM call across 21 registered backends, of which 11 are currently healthy. A "backend" is a role ↔ hardware mapping decided at config, not at request time. The model assignment plane (config/model_assignments.yaml, bind-mounted) decides: reasoning → DeepSeek-V4-Flash on tier 0; orchestrator → vLLM on tier 1; fast → Bonsai 27B on tier 1. Change one line, restart one service, and reasoning traffic moves from cloud API to local hardware. No rebuild.

role_to_model_assignment:
  reasoning: deepseek-v4-flash-pool         # Local: DGX tier 0
  reasoning_local: deepseek-v4-flash-pool   # Same
  deliberative: bonsai-27b-5090             # Tier 1
  orchestrator: vllm-orchestrator           # Tier 1
  coding: gpt-4-turbo                       # Cloud fallback
  embedding: vllm-dgx-embed                 # Tier 0
  perception: vllm-dgx-perception           # Tier 0

---

Nineteen Capability Gates Verify Nothing Is Silent

A 200 OK on /health does not mean the service can do its job. We learned this at scale: a Discord bot with a valid token and healthy container status that never once logged in, a Nexus instance that returned 200 for every /search call and returned [] for every one, a secret vault that reported readable and was unreachable.

The platform wires 19 capability checks into a single gate that runs every 4 hours:

ISA (Inference Stack Agreement) 001–013: Five config planes declare what serves what. Do they name the same models? Do those models actually exist in the catalog? Are the launch scripts on the DGX the ones we checked in? Does the coordinator respond? Is the KV cache healthy? Do the served checkpoints have non-finite tensors? Found one live: lm_head.weight rows 139020–139135 with 131 NaN values, one row literally Inf (absmax 1e38). Detection: per-tensor and per-row scanning of all served checkpoints, ~1–3 GB read per model. Turns "something is wrong with this model" into "rows 139020–139135 need a targeted fix."

POOL001–POOL006: The speculative decoding launcher accepts a --spec-draft-model flag. Does it require a paired --spec-type flag? Measured failure: launcher accepted draft-model alone, loaded cleanly, reported healthy, drafted zero tokens. Do the pool scripts on the DGX match the canonical repo versions (SHA256 parity)?

Fleet Capabilities: Can the secrets vault actually return a restricted secret? Is the output directory writable? Is the Discord bot logged in (not just token-set)? Is MicroScheduler reachable?

Each gate returns one of three states: Verified (exit 0, the invariant holds), Violation (exit 1, the invariant is broken), or Dead (exit 2, could not judge — a timeout, a closed port). A probe that cannot emit a verdict is dead, not passing. Exit 1 pages and pages again until fixed.

---

Deployed vs. Designed

Deployed (production traffic, verified by gate):

• Speculative decoding on DGX Spark (28.55 tok/s verified 2026-08-07)
• Model assignment plane (role→model→backend routing via config)
• Three-tier memory hierarchy (all tiers reachable; traffic measured at <0.2% link utilization)
• AitherMesh overlay (100.64.0.x WireGuard)
• 19 verification gates (ISA + POOL + Fleet capability probes)

Designed (architecture document complete, components built, not in serving path):

• Multi-tier block prefetching with window-aware demotion
• RPC pipeline split across tiers (block sharding + event streaming)
• Learning-based block prefetch (AitherGraph predicting router decisions)

The single-box GB10 is the serving path today. The multi-node pipeline is reachable and the components are there, but the split does not carry production traffic yet. That sequence is deliberate: measure each layer's assumption before adding the next.

---

Welcome to the Game

Red Hat is right that the CPU is part of the inference equation. On this machine, CPU and GPU are the same die, so "CPU-GPU split" becomes "memory-tier split." The prefetch window arithmetic (1 ms intra-pass, 1.8 seconds full-pass) makes that split precise and measurable. A tier that can serve the intra-pass window is fast enough for streaming; a tier that serves the pass window is useful for background promotion.

This is the inference stack that runs on a desk. 284B parameters, 30 tokens per second, fully measured, fully local, ready to scale. It exists. It is operating. The gates run unattended every four hours and report what they find — including, right now, three role-to-backend disagreements they are naming precisely. A gate that never has anything to say is a gate nobody has watched fail.

The frontier is not in the cloud. It is sitting here, answering questions at the speed that matters for an agent that thinks between turns.