The Fleet That Sees Two Minutes Ahead
AitherOS is a large local-first service fleet, and for a while now it has had something like a nervous system. AitherPulse turns failures into weighted pain. Genesis heals. Watch restarts. Sense feels affect and lets it color how the system talks. It works — and every bit of it is reactive. The fleet feels pain only after the damage has already landed. A container wedges, the health check fails, pain registers, remediation starts. The nervous system is real; it just lives entirely in the past tense.
The Prospector program is the attempt to give it a future tense. As of this week, a model watches the fleet's own telemetry and predicts the trajectory of service health — and it has fired its first real alarm, roughly two minutes before the window it warned about. This post is about what it took to get there, and — more interesting, we think — the harness we built so that a young model can be wrong safely.
Retargeting a game world model onto ops telemetry
The model itself is not new. We built a learned world model for ARC-AGI-3 game rollouts: given a state and an action, predict the next state, unroll, plan against the imagined futures. At some point the obvious question surfaced: the fleet emits a state stream too. Why does the world model only get to imagine games?
The retarget looks like this:
- State: a 171-dimensional vector, binned every 30 seconds from the live Redis event bus — service health transitions, restart counts, pain weights, queue depths, the usual telemetry suspects, all normalized into one fixed-width snapshot per bin.
- Dynamics: a residual-MLP forward model, trained with backprop through time over K=8 unrolled steps. Feed it the recent window, and it rolls the state forward — a short imagined future of fleet health instead of a game board.
- Head: a supervised classifier over the rollout that answers one question: is a service failure coming?
None of that is exotic, and that's the point. The model is small, local, and cheap enough to score every bin forever.
The twist: numbers weren't enough — the state needed perception
The honest middle of this story is that the pure numeric state vector didn't work. Counters and gauges are what the fleet did; a lot of the predictive signal lives in what the fleet said — the raw event text scrolling through the bus.
So we added 5 percept dimensions. Each 30-second bin, a small local LLM
(gemma) reads the raw event text and scores it: error_tone,
restart_activity, cascade_suspect, novelty. Machine perception,
distilled to a handful of floats and concatenated onto the state vector like
any other feature.
The ablation is unambiguous. Without percepts, the supervised head fails the quality gates everywhere. With them, it passes: precision 0.81 / 0.70 offline, with roughly two minutes of lead time before a service failure. The cheapest way to summarize it: the world model needed senses, not just instruments.
LLM-grounded labels: vote, then adjudicate
Supervision had its own problem: what counts as "an incident window" in months of historical telemetry? Hand-labeling was never going to happen, and any single model's judgment bakes in that model's blind spots.
So labels were LLM-grounded too. Two local models — qwen and gemma — independently vote on each historical incident window. Where they agree, the label stands. Where they disagree, the window goes to a third, stronger cloud vote (deepseek) for adjudication.
The disagreement-only tiebreak cut measured label noise from 26.9% to 14.4% — nearly halved, while keeping the cloud model out of the hot path for the majority of windows where the two local voters already agreed.
First alarm: wrong service, right storm
Prospector is now live. An observer container subscribes to the event bus, assembles the 171-dim state every 30 seconds, rolls it forward, and scores the window. And it has fired its first real alarm.
Verdict: false positive on attribution. It named the wrong service. But it alarmed during a genuine window of fleet churn — the storm was real, the finger pointed at the wrong cloud. We're deliberately not spinning this. It's the first data point in the earned trust ledger, and the ledger is the product.
The harness: advisory pain, and trust you have to earn
Here's the part we'd argue matters more than the model. A predictor that can page you — or worse, act — on day one is a liability. The wiring that's landing now makes every forecast advisory first:
- Advisory pain. A forecast becomes a
predicted_service_failurepain point in AitherPulse — with severity capped below the interrupt threshold. A prediction can never halt anything, structurally. The cap is in the pain type, not in a policy someone could forget. - The fleet feels unease. Advisory pain auto-forwards to Sense, so the fleet literally feels apprehension about its own near future — the affect channel that already existed for real damage now carries the forecasted kind, at lower intensity.
- Reflex context, not reflex action. The prediction injects context into Reflex — prepare, don't kill. Warm what's cold, drain what's draining, and leave the trigger alone.
- Genesis triage. The named service gets health-checked first and evidence gets gathered while the window is still open. Any restart proposal is human-approval-gated through Steward — the model can suggest surgery; it cannot hold the knife.
- A promotion gate. A prediction class earns autonomous soft actions only after ≥20 live alarms at ≥0.6 measured precision. Not vibes, not offline numbers — live, counted, per-class track record.
Advisory-first, escalate by earned trust — and the reactive loop stays the referee. If a prediction is wrong, nothing happened that the old nervous system wouldn't have survived. If it's right, the fleet met the failure with warm spares, gathered evidence, and a restart proposal already sitting in a human's queue.
Prediction is cheap. Trust is earned.
The model is the least interesting part of this system — a residual MLP and a small LLM reading logs could be assembled in a weekend. The engineering that took real thought is the harness: severity caps that make predictions structurally incapable of causing harm, an attribution ledger that scores every alarm against what actually happened, and a promotion gate that converts a track record into authority one prediction class at a time.
That first alarm — wrong service, right storm — would be an embarrassment for a system that claimed foresight. For a system designed to earn foresight, it's exactly what week one is supposed to look like: the ledger opened, the first entry honest, and the fleet, for the first time, feeling something about its future two minutes before it arrived.