An Agent Orchestrator Without the Cluster

Verdict: the primitives an agent workload needs — isolation, an egress allowlist, suspend and resume, declarative apply — do not require a cluster. awrun 0.2.0 ships all four as a pip install, and adds the two a cluster orchestrator leaves to you: why a run exists, and who was allowed to touch it. If you already operate Kubernetes at scale, Google's new AX is aimed squarely at you. If you have one machine and agents you would like to stop babysitting, start here.

AX describes itself as "a high-throughput, declarative orchestrator to run billions of autonomous agent workloads in a cluster". Its model is four resources — Task, Workspace, Gateway, Model — applied as manifests, with ax suspend and ax resume for state. We think that is the correct shape, and it is reassuring to see it arrive from that direction. Here is the same shape at the other end of the scale.

The same four, on one machine

The last row is the difference. The queue is one JSON file per run, and a run's state is which directory that file sits in. Claiming, suspending and resuming are each a single atomic rename, so two workers — or two machines sharing the directory — cannot both win. There is no daemon to keep alive and nothing to install beyond Python.

Resume means "continue", and you can count it

Measured on a desktop, 200 runs in the queue: suspend 2.3 ms median, resume 2.8 ms (p95 3.2 and 3.9). That is the queue's half — moving a run in and out of the claimable set.

The half that matters for agents is what happens next. A --kind flow run is a journaled workflow, and its run id is its journal id. Every model call is appended and synced to disk when it completes, so the journal is the checkpoint. When a suspended flow resumes, the calls it already finished are handed back from the journal; only the rest are made.

We test this by starting a real child process, stopping it mid-workflow through the queue, and counting calls. Of four model calls, the two that had finished were made exactly once across suspend and resume. The one in flight at the moment of the stop was made again — one repeated call, not four. For a long research or coding run, that is the difference between a pause and a refund request.

pip install 'awrun[flow]'
awrun submit --kind flow --script research.py --name nightly-research --priority 5
awrun suspend r-7f3a9c2e     # parks it; a running flow is asked to stop
awrun resume  r-7f3a9c2e     # same id, same priority, same place in line

Because replay trusts the journal, the journal is digested at suspend and checked at resume. One that changed while the run was parked is not replayed.

A limit that cannot be enforced fails the run

A manifest that says memory_mb: 4096 while the process runs unconfined is worse than no manifest. So the rule is one line: a run asking for something its runner cannot enforce is failed before it starts — never run unconfined.

timeout_s is enforced on any local run. cpus, memory_mb and egress need isolation: container, which starts the run with every capability dropped and no new privileges. egress.hosts: [] means no network at all. A non-empty allowlist puts the run on an internal container network whose only exit is awrun egress-proxy — and awrun asks the runtime whether that network really is internal before it starts anything.

The proxy matches the name the client asked for, before any lookup. *.example.com does not match evilexample.com. Listing a host opens 80 and 443, not its admin port. Every denial is logged, because the destination that was turned away is the interesting one.

Two things a scheduler does not tell you

Why is this running? Every run carries lineage — intent, goal, expedition, flow, plan, notebook, parent_run. awrun queue --lineage goal=G-42 answers "what is executing for this goal right now" without asking the thing that planned it.

Who touched it? Suspend, resume, cancel, apply, export and import are written to a hash-chained audit trail before they happen, so a change that was turned down still leaves a line. Name your operators and those changes also require a verified identity on that list. Turn on AWRUN_AUDIT_REQUIRED and a change that cannot be recorded does not happen. These are awiam, awbac and awdit — optional installs, each usable alone.

Moving a run between machines

awrun export bundles a suspended run with its journal and closes the local copy — a run lives in one place. awrun import lands it suspended on another queue under the same id. Import verifies before it unpacks: against the digest the exporter printed, or against the exporter's public signing key via awseal. A bundle offered with neither stays out. That is a workload migrating between nodes with a USB stick as the control plane.

Which one to use

• You run Kubernetes and need thousands of concurrent sandboxes: AX is built for that, and awrun is not a cluster scheduler. It has no multi-host placement; a run moves when you move it.
• You have a workstation, a homelab or an edge box and agents that run for hours: pip install awrun. You get suspend and resume that does not re-bill you, limits that fail closed, an egress allowlist, a declarative queue, and an audit trail — today, with the network cable out if you like.

Every number above comes from the package's own test suite and self-test: awrun self-test, and pytest in the repository.