Deciding Is Not Spending: An Autoscaler That Never Holds the Credit Card
Every number in this post was measured on our own fleet this week.
The rule
A library may never spend money on its own.
That sounds like a policy. It is actually an architecture, and it decides where every line of an autoscaler goes.
awrun is our priority-aware queue for CI work. It is the only component
that sees the job queue and the runner pool in the same breath, so it is the
only thing that can answer are we saturated, and by how much. It ships
publicly on PyPI, which means it runs on machines that have no cloud account,
no credential, and no quota — and on ours, where it has all three.
So awrun owns the decision. It never owns the act.
What that split buys
The obvious design is one component that notices saturation and launches
instances. It is also the design where a pip install puts a thing on a
stranger's laptop that can bill their employer.
Splitting it moves three hard problems out of the library and into the host, where they already have answers:
- A credential. The provisioner reads AWS keys as a host process. That is not an implementation detail, it is the boundary — see below.
- A quota. Instance count is capped by real vCPU headroom, read from the account at plan time.
- A spend gate. Launching is off by default and stays off until a human arms it.
The library still tells you the truth when it cannot act. awrun capacity
reports the gap whether or not a provider is registered — measurement never
waits on the money decision.
The lane
awrun (decides) -> AitherComet -> deploy-executor -> AitherZero playbook -> EC2
in a container on the host
Nothing crosses the container boundary except a decision. Here is the whole
thing answering, live, with dry_run: true:
ci-runner-provision: DRY RUN (pass -Execute to launch)
credentials: from the local cache (vault not consulted)
account: 286801154241 region: us-east-1
quota: this plan needs 24 vCPU; 140 of 192 are free
plan: m5.xlarge in us-east-1, labels=self-hosted,Linux,X64,aws
cost: ~$0.192/hr -- BILLS UNTIL TERMINATED
Credentials resolve. Quota is real. Nothing launched.
Why the boundary is not an implementation detail
The provisioner fetches its AWS credentials as a bare host process. Call the same function from inside a container and it does not fail fast — it loops:
credentials: no engine answered (attempt 3/6) -- retrying in 20s
An autoscaler that runs in a container and provisions from a container will therefore assess correctly, decide correctly, and provision nothing, forever. Every log line will look like a retry. This is why the split is drawn where it is, and why "just call the tool" is the wrong shape even when the tool is right there on disk.
The part worth your attention
Our capacity routine was enabled, on a five-minute interval, wired to page on error. It had never once added a runner.
Not because the logic was wrong — because a path resolved one directory too high in a container, so the import that reads the queue never loaded. The routine ran, caught the failure, reported it faithfully, and every run looked like a run.
When we fixed it, it answered on the first tick:
{"queued": 53, "pool": 7, "idle": 0, "saturated": true, "want": 26}
Queue saturated (want 26 runners), dry-run only
53 jobs queued against 7 runners, none idle. That had been true, unread, for a long time. The lesson is not check your paths. It is that an automation which reports its own failure faithfully is indistinguishable, in every dashboard, from one that has nothing to report. The only thing that separates them is asking it for a number and looking at the number.
Growing without shrinking is a spend generator
The reaper runs in the same invocation as the provisioner. Not as a separate schedule, not as a follow-up job — the same call, because two schedules is how one of them quietly stops existing.
It also refuses the obvious mistake:
reap: 0 -- 31 run(s) still queued, not shrinking while work is waiting
Scaling down while jobs wait is the failure that turns an autoscaler into a thrash machine. It is one line to prevent and impossible to notice once it starts.
A rebuild is the real test
While we were here, we asked a harsher question: if this machine died, how much of it comes back from the repository?
We now measure it. Every scheduled task the platform owns must launch a
payload that git ls-files knows about. First run:
57 of 61 tasks would not survive a rebuild
55 launch payloads outside the repository
2 launch files that do not exist
Those last two had been firing on schedule for weeks, running nothing, showing green. That is the same silence as the autoscaler: the system works, every probe agrees, and the capability is gone.
The number is pinned and ratchets down only. It cannot grow without failing the gate, and it convicted our own newest task first — a scheduled job written by hand an hour earlier, which is exactly the kind of thing that feels like progress and is actually a future outage.
The shape to steal
If you are building something that spends money on its own:
- Put the decision where the information is.
- Put the act where the credential is.
- Make the two talk over a boundary you can audit.
- Default the act to off, and let the decision run anyway.
- Ship the shrink in the same call as the grow.
- Ask it for a number regularly — a silent automation and a dead one look identical.
awrun is on PyPI. The decision half runs anywhere; the act half is yours.