Decisions You Already Made Should Never Be Paid For Twice

Verdict: a bounded decision does not need a frontier model, and once the door has learned one, it should never cost a model call again. On our judge bench the door answers 24 of 24 judgements it has been taught at 0 model calls, 0.0 seconds, $0. Read that row precisely: those are the same 24 items it was taught, so it measures the cost of a repeat, not accuracy on unseen work. We have since run the held-out version, and it says something narrower and more useful than the headline: the door is a cache with a fail-closed miss, not a judge that generalises. On a criterion it has been taught it is right ~99% of the time for nothing; on one it has never seen it abstains rather than guesses, and the call falls through to the model. Section 3.1 has the table. On a scan of an unfair coin the reported probability tracks the coin (0.60 for 60/40, not 0.99) once outcomes have come in; on our own journaled forks it holds up at ECE 0.091 over 137 pairs — but only after we found that the recorder had been writing confidence into the probability column, which made the first honest run of this check look far worse than the door actually is. Every number below is from one box, one day, and one command anyone can rerun, including that one.

1. A bounded decision does not need a frontier model

Most of what an agent platform decides all day is bounded: pick the faster of two branches, grade an output against a rubric, route a request to the right model. These are not open research questions — they are decisions with a small answer space, made under the same conditions, over and over. Routing every one of them through a frontier model is paying full price for a question you already know the answer to.

The AitherOS decision door (decide / judge — MCP tools, awdk's adk.choose, and the world model's own /decide and /judge HTTP surface) is built around that observation: answer from what it has learned at this fork when it can, fall back to a model only when it cannot, and say which it did. On the judge bench below, "door warm" is the same 24 judgements the LLM judges graded, after the door had been taught each one: 100% accuracy, 0 model calls, 0.0 seconds, $0. That is a cache-hit measurement and we label it as one. What it proves is the cost of the second sighting of a shape, which is the shape of the win — not a smarter grader, a grader that stops asking the same question twice. What it does not prove is accuracy on work the door has never seen; for that, read the "door, cold" row, which is 87.5% against 95.8% for the same local model asked directly. The door pays 8 points and 146 seconds on the first sighting to pay nothing on every one after it.

2. Fast is not the same as right

Speed is worth nothing if the number underneath it is wrong. We scanned an unfair coin from p=0.05 to 0.95, 100 flips per point, and asked the door to report its own confidence — a 60/40 coin should come back reading close to 0.60, not collapse to "the likelier side, with certainty."

Cold, the door behaves like a Jev-shaped collapsing stub: always the likelier side, reported at 0.99 regardless of how close the coin actually is — the exact failure mode publicly measured in Jev's own choice calibration (~0.4 ECE) and noul (undershooting by up to 9 points). Twenty outcomes of feedback bring our error down by roughly 4x; a hundred bring it near the noise floor for this test (100 flips carries roughly ±0.05 sampling noise on its own, so 0.029 is close to as good as this particular scan can prove). The point isn't the third decimal — it's that the number moves toward the coin's real rate as evidence comes in, instead of staying pinned at certainty.

And here is what happened when we checked that on our own journals instead of a coin — including the part where we published a wrong number first.

The first run said not calibrated: ECE 0.43 over 110 pairs, with 71 predictions sitting at a mean 0.96 where the observed frequency was 0.51. We wrote that down and shipped it, because a calibration claim you can only make on the easy bench is not a calibration claim. Then we found the cause, and it was not the model: the recorder wrote confidence and never probability, so the journal contained zero probability pairs and the only thing anyone could score was the evidence-strength number being read as if it were a probability. Two different quantities, one column.

With probabilities actually recorded, the same check on the same box:

Calibrated within our own 0.1 line, on real forks. Two caveats we are not going to bury: 100 of those 137 pairs come from a single demo domain, so this is thin evidence about a handful of forks rather than a production reliability diagram; and the legacy confidence-versus-outcome row (ECE 0.43) is still in the tool's output, labelled as what it is — not a probability. The check that caught all of this ships with the kit and is the one bench we most want you to run.

3. What the judge actually costs

Eight agent outputs, three criteria each, 24 judgements, labels known in advance, same local hardware for every non-cloud row:

Honest caveat: these 8 cases are easy enough that a cheap cloud model also hits 100%. This table is not an accuracy win for the door — it's a cost, latency and correction-loop win. The door starts behind a fresh cloud judge and catches up to it through outcomes, then answers every repeat of a case it has already seen for nothing. A judge you pay a fraction of a cent for per item is cheap; a judge you pay nothing for the second time is a different category.

3.1 The held-out number, which refutes the flattering reading

The table above teaches the door on the same 24 judgements it then grades. We said in the opening that the held-out version was owed. Here it is, on 134 labelled cases — 433 criteria harvested from real pytest, ruff, curl, git, podman, CI and service-log output — with the door taught on one half and graded on the other, every leg in its own process:

Read the coverage column, not the accuracy column. 46.7% of the held-out criteria had been taught verbatim, and measured coverage was 47.2%. Those two numbers tracking each other is the finding: the door answers the forks it was taught and abstains on the rest. On the 247 criteria that appear nowhere in its training in any form, it answers 0.8% of them — it does not guess, which is the behaviour we want and the opposite of what a confident wrong answer would cost.

So the claim this post is named for survives exactly as written, and a claim we never made but that the warm row invites — that it judges new work for free — does not. The saving is your repeat rate. For an eval suite re-run on every commit, that rate is close to one; for a judge seeing novel criteria every time, it is zero. Both are worth knowing before you wire it in.

Getting to this number cost two bench bugs of our own, and both made us look better:

• Leave-one-tool-out was not held out. The door keys a fork on the criterion text, and 128 of 375 "held out" criteria were shared verbatim with the training tools — the command succeeded appears 46 times across 9 of them. We now drop those from the test set and report what remains.
• Every leg after the first shared one store. The checkpoint directory is read at module import, so the second and every later leg kept writing to the first leg's scratch dir. Fold 1 scored 0.0% coverage — the honest answer — and folds 2 through 9 scored ~100%, because fold 1's training set had already taught every later fold's test family into the store they all shared. The tell was that the hard split (98.9%) beat the easy one (38.3%). Each leg now runs in its own process, and the bench's self-test runs two legs back to back and asserts the second, taught nothing, answers nothing.

That is three of our own published numbers now corrected by our own tooling before anyone outside had to find them. We would rather ship the narrower claim.

3.2 The same measurement three days later, with a shape key and a learned rung

The 47% coverage above was the door keying a judgement on the criterion text. A judge that only recognises a criterion it has seen verbatim is a cache, and we said so. Two changes since: the judge now keys on a feature descriptor of the output — traceback present, error and pass/fail counts in buckets, exit code, keyword overlap with the criterion, length bucket — so "the tests passed" on a log it has never seen lands on a shape it has; and a neural rung (linear heads over hashed embeddings, temperature-scaled, with an out-of-distribution gate that makes it decline outside its competence) sits under the table. The dataset grew to 463 labelled rows / 1,747 criteria. Split by row so no output text is in both halves, stratified by shape, 30% held out, five seeds, and the neural rung fitted on the training split only, per seed:

The floor is the two-line baseline any learned rung has to beat, and the engine rung alone ties it exactly — as it should, because the engine is a lookup on that key. The neural rung beats it on every one of the five splits (+0.2 to +2.2 points), which is a small, consistent lift on ~1,500 answered criteria per split, not a different machine. Coverage is identical because both abstain on a criterion they have never seen in any form; that 14% still falls through to a model, and we would rather it did. The command is python tools/judge_holdout_bench.py --seeds 5 --neural; the artifact is tools/judge_holdout_2026-09-21.json.

So the reading moves from "a cache with a fail-closed miss" to "a cache with a fail-closed miss that generalises across outputs of the same shape" — 47% → 86% of unseen judgements answered, at 98% when it does. It is still not a judge of criteria it has never been shown.

4. Where our own retrieval was broken

We ran a harder test on ourselves: 30 "where is <symbol> defined" questions over this monorepo, an 8-directory-open budget, one seed.

Asking a model to navigate the tree fork by fork found nothing in 41 minutes and cost 217 model calls doing it. But the more interesting row is the one right after it: our own name-only landmark map also found 0.0% — fast (5.0s) and completely wrong, because it had indexed a Next.js static export and never once saw lib/ or services/, the two trees where the answers actually live. A fast, confident, empty index is worse than a slow one, because nothing about the 5.0-second wall clock told us it was broken.

Fixing what the map indexed took it from 0.0% to 10.0%. Adding a real symbol index — 101,803 symbols across 15,511 files, built once in 157 seconds into SQLite, one indexed query per lookup instead of a directory walk — took it to 76.7%. Letting the decision door learn from outcomes on top of that took it to 80.0%, at roughly a third of the directory opens. Owning that our own retrieval was the broken part, not the model, is the paragraph in this post we'd want a reader to trust the rest of it for.

5. What you can run yourself

The door's own benches ship inside the package, so the tables above reproduce on your own hardware, against your own brain:

pip install awdecide
awdecide door-bench calibration
awdecide door-bench judge
awdecide reliability

Each bench exits 0 only when it clears its own verdict, 1 when it fails it, and 2 when it could not run at all — never a silent pass. --json prints the numbers the tables are built from. The exploration bench is the one we cannot hand you: it asks "where is this symbol defined" over our own tree, so its questions do not travel; run the other three and hold us to them. The reliability bench is the one that currently returns NOT CALIBRATED on this box's journals, and we would rather you found that by running it than by taking our word for the good numbers.

The door's own service tree stays private — it shares a repository with our ARC world model and fleet deploy — so what ships is the door on its own, inside the awdecide package: the engine, the bench scripts, the compaction corpus and the generated half of the labelled dataset (408 rows; the 55 rows transcribed from our fleet's logs stay home). No server, no fleet, stdlib only: awdecide door-bench judge-holdout is section 3.2, door-bench compact and door-bench cache-cost are section 6, door-bench calibration is section 2, each exiting 0 only when its own verdict holds. On the shipped 408 rows the held-out judge reads 94.0% coverage and 98.4% accuracy when answered against a 97.3% lookup floor — higher coverage than the 463-row table above because the transcribed rows it lacks are the hardest ones, and we would rather you knew that than noticed it.

To put the door in front of your own agent: it's an MCP tool (decide, decide_batch, judge, decide_outcome) through the gateway at 127.0.0.1:8182, a Python call (from adk.choose import decide, outcome) if you're on awdk, or a plain HTTP surface at /v1/decide* and /v1/judge* on the public gateway. Ask once, act on the answer, post the outcome — the next identical decision is the one that costs nothing.

6. Compaction: where it happens is the whole argument

The same door drives tool-output compaction — a 1,000-line pytest run reaching the model as the dozen lines that decide what happens next, chosen per line shape (decide.compact.pytest), so the second run of the same kind of command is compacted from evidence in ~30 ms with no model call. A fair critique of per-tool-call compaction went round this week, and its five points are worth answering with numbers, because our design already lives on the right side of each of them and we want to say precisely why.

Compaction is not a filter, and history editing is expensive. Prompt caching is prefix caching. Delete message 2 from 1,2,3,4,5,6 and 3..6 are re-written at the cache-write price (1.25× input); leave it and they are re-read at 0.10×. A compactor that edits history pays for every token after the edit, every time it edits — and it discards the reasoning blocks the API only replays against an unchanged history. Ours never edits history: it shrinks a tool result before it is appended, at the one point every result passes (ToolRegistry.execute in awdk, the awsh tool-result event, the /compact command, the MCP tool). Nothing above it changes, the prefix stays cached, the reasoning stays. We modelled a 40-turn session on real tool-output sizes from this box with a prefix-cache accounting and Sonnet 5 list prices (python tools/cache_cost_bench.py):

History editing saved half a percent and wrote 66% more cache than doing nothing, while invalidating 47 reasoning blocks. Append-time compaction saved 43% with rules alone and 59% with the taught door. The arithmetic is the critique's; the design is what it recommends.

It knows what it is deciding on. Our compactor decides on the result, line by line, with the whole result in view — not on the tool call before the result exists. And the lines that stop an agent looping — the command, the exit code, the error, the summary (1 failed, 39 passed) — are kept by rule, never sent anywhere and never dropped: on the seven-item corpus (pytest pass and fail, ruff, git log --stat, a podman build, curl refused and 401) the always-keep rules alone drop 39.8% of lines and 53.0% of estimated tokens at 100% recall of the hand-labelled must-keep lines; the taught door drops 80.4% of lines and 74.3% of tokens, still at 100% recall (python tools/compact_bench.py, exits 1 on any lost must-keep line). A shape the door has never seen is kept, not guessed at.

It does not replace the model's own compaction. Frontier models are tuned on their compaction flows and we leave that alone; this runs at the tool-result boundary and makes the history the model later compacts smaller to begin with.

One caveat on the table, in the same spirit as section 3: "taught door" is measured after one teaching pass on the same corpus, so it is the second-sighting number. The rules-only row is the one that needs no teaching at all.

short version

A bounded decision doesn't need a frontier model twice: our decision door answers a judgement it has been taught at 0 model calls, 0 seconds, $0 (a cache-hit measurement, labelled as one; cold it is 87.5% against the same model's 95.8%). Held out by row on 463 labelled cases, it answers 86% of unseen judgements at 98% accuracy and abstains on the rest rather than guessing — a cache with a fail-closed miss that generalises across outputs of the same shape, and a learned rung that beats the lookup floor on every split. Compaction happens where it is cache-neutral: at append time, never by editing history — 50 cents against a dollar twenty for the history-editing hook on a modelled 40-turn session. It also reports a 60/40 coin as 0.60, not 0.99, after learning from outcomes. And when we pointed it at our own codebase, our retrieval map was broken (0.0% found); fixing the map got 10.0%, and a real symbol index got 76.7%. The benches that travel ship in awdecide (awdecide door-bench calibration|judge, awdecide reliability); the exploration bench asks questions about our own tree and stays here.