97.5% Accuracy On A Tenth Of The Model Calls: The Decision Loop, Measured

Verdict: stop paying a model for a decision you have already made. The industry pattern is a function — send the question, call the model, pay, repeat, forever. We built the loop instead: ask once, act, report what happened, and the next identical decision is answered from evidence. Measured on one desktop:

• 97.5% accuracy against 72.0%, from the same brain, on a tenth of the model calls — 40 instead of 400, in 14 seconds instead of 120.
• 35,177 ms → 9.0 ms on a repeated decision, live on an ARC-AGI-3 fork, with zero model calls in the second answer.
• A judge that scores 100% at zero calls, matching an LLM judge that needed 24 calls and 523 seconds for the same 24 items.
• Probabilities that are actually probabilities: calibration error 0.226 → 0.029, where the common failure is a confident 0.99 on a 60/40 coin.

And it knows its own limits. On a decision outside what it was taught, the door declines by name and the call goes to the brain — it answers 0.9965 of what it is competent for and hands back the rest. A fast path that guesses is a liability; a fast path that says "not mine" is infrastructure.

What to do after reading this: put bounded, repeated decisions behind /decide and keep your model for the ones the door hands back.

104 seconds. Every number below, measured, with the command that reproduces it.

The repeat, measured live on an ARC-AGI-3 decision — the hardest subject we have, where every board is new:

Thirty-five seconds to nine milliseconds, because the second time it had already been told. That is the whole product in three lines.

The same shape on a judging workload — 24 labelled items, graded against known labels:

Identical accuracy. Twenty-four fewer model calls, and 523 seconds back.

Two more numbers before you wire anything up. Our calibration error on an unfair coin is 0.029, down from 0.226, so a 60/40 coin reads 0.60 rather than collapsing to 0.99 — you cannot threshold a confidence that is always 0.99. And compacting tool output before the model reads it removes 74.3% of the tokens at 100% recall: every failure line survives.

The four numbers, and the command for each

Every one of these runs on one desktop.

The gate that decides serve-or-decline

The door answers from the cheapest rung that has evidence: an exact match it has seen, then a near neighbour, then a small learned head, then a model. The learned head is the interesting one, because it is the only rung that can be wrong in a way that looks right.

It ships with four gates, and a head has to clear all of them before it serves anything: it must beat always-guessing (a head that reproduced the majority class exactly measured a lift of +0.0000); its accuracy needs a Wilson 95% lower bound above the floor, not a point estimate, so a perfect head on 23 forks is held and the same head on 260 goes through; it must record which criteria it was trained on and return nothing outside that set; and it must hold up on a benchmark whose forks appear nowhere in its training data.

Our current head clears the first three at 0.9965 and fails the fourth at 0.7059, where a model gets ~0.96. So it is a candidate today, not a serving rung, and cold judgements go to the brain. That is the honest state of it and it is also the design working: the door serves what it is good at and says so when it is not. Each gate was added because a measurement forced it, and each is asserted by a test that fails when the gate is removed.

Against a decision service that does not learn

The interesting comparison is not against a bigger model. It is against the same model wired the ordinary way: one call per decision, no memory of the last one. 400 decisions over 40 distinct states, brain right 70% of the time at 300 ms a call:

Ten times fewer calls and 25 points more accuracy, from the same brain. And it is not a cache: a cache makes the same 40 calls and lands at 78.8%, because it stores the model's first answer and has no way to hear that it was wrong — 10 of 40 states stayed frozen on a wrong answer for the whole run, served at zero cost and full confidence. The door has an outcome channel. By the third batch of 50 it is at 1.000 and stays there:

accuracy per 50: 0.88  0.94  1.00  0.98  1.00  1.00  1.00  1.00
served by:  engine 90.0%   llm 10.0%

Batching helps on top: 13 cold questions take 4.37 s sequentially and 0.31 s batched — 14.3× — but 400 batched calls are still 400 calls.

Reproduce: python tools/decide_bench.py

It is wired into the hard case

The ARC-AGI-3 solver picks an action, and one of the places it does that is a yes/no on whether to abandon a queued plan — until now decided by a hand-tuned constant. That seam now asks the door, in shadow: the door answers, the answer is logged next to the heuristic's, and the heuristic still decides. Ground truth goes back through /decide/outcome on the next turn.

Shadow rather than live because ARC states are overwhelmingly novel and the learned rung is held by the out-of-distribution gate above — the benchmark whose whole point is novelty is the last place to make an exception. Shadow buys an agreement rate and a journal of real forks to learn from.

Where it is wired

POST /decide, /decide/batch and /decide/outcome on the world model; /v1/decide* on the public gateway; decide, decide_batch, judge and decide_outcome as MCP tools through the local gateway on 127.0.0.1:8182; from adk.choose import decide, outcome if you are on awdk. Ask once, act on the answer, post the outcome. The next identical decision costs nothing.

Compaction ships as compact_output and is the same shape: keep-or-drop per line shape, so a 1,490-line build log becomes 292 lines with every failure intact, in one batched call.

The one number to watch

Coverage, not accuracy. Measure what your fast path declines, and make sure something good catches it.

The compaction row is the door-taught leg — what compact_output actually runs. The bench also prints two (diagnostic) legs that switch the rule pass off to isolate the door's own contribution; the best reads 77.9% and is not a configuration anyone runs.

Reproduce the whole set in one command:

python decision_proof.py

It runs calibration, judge, compaction, routing, exploration, latency and reliability, exits 0 only when every bench clears its own check, and exits 2 rather than 0 when it cannot run one at all.