Every Save Is a Backup: Real-Time Differential Git Backups in AitherOS

The way most people think about backups is wrong.

You take a snapshot every N hours, ship it somewhere, hope the destination is still there when you need it. If a file changes ten times between snapshots you lose nine of those states — including whichever one you actually needed to roll back to. If the snapshot lands in a black hole (see the March 15 audit) you don't find out until the disaster.

Git already solves this. It is a content-addressed, immutable, integrity-checked, remote-replicated, incremental storage system that ships with every developer laptop on the planet. Every commit is a snapshot. Every diff is a delta. Every git push is a replicated write. The tooling to restore, browse history, compare states, revert a single file — already written, already trusted, already installed everywhere.

We already used it as object storage. This week we added the missing half: use it as a real-time backup daemon.

The primitive

lib/backup/MicroCommitBackup.py is a poll-based watcher. You point it at a directory ("watch_dir"), you point it at a bare git repo somewhere ("repo_dir"), you give it a remote URL, and you go on with your life.

Every second (configurable) it rescans the tree, diffs against the fingerprint it remembers, and marks changed files as pending. It does not immediately commit — that would spam the remote with a commit per keystroke. It waits until either (a) the writes go quiet for 5 seconds, or (b) the oldest pending change has been sitting for 60 seconds. Then it stages the batch, commits with a stable author, and pushes.

That's it. That's the mental model.

from lib.backup import MicroCommitBackup, MicroCommitManager

watcher = MicroCommitBackup(
    name="my-notes",
    watch_dir="~/notes",
    repo_dir="~/.aither/mirrors/notes",
    remote="https://user:token@github.com/me/notes-mirror.git",
    branch="main",
    debounce_seconds=5.0,
    max_defer_seconds=60.0,
)

manager = MicroCommitManager()
await manager.register(watcher)
# ...write to ~/notes as normal. Nothing else to do.

Kill the machine. On the next box, git clone the remote. You have every state of every file, timestamped, diffable, integrity-verified.

The pattern that already existed

We shipped this on top of infrastructure we already had, because it turns out the hard parts had all been solved for a different problem: mirroring a 3.6 GB model weight to GitHub without a paid storage tier.

That work produced two libraries:

• GitHubChunkedBackup — the low-level chunker. GitHub caps release assets at 2 GiB. So we split at 1.9 GB, upload each part as its own asset, and stitch on read via HTTP Range requests. Never buffer a whole part in RAM.
• LargeFileStore — a threshold-based auto-chunk layer on top. Below 90 MiB, upload as a single asset. Above, split-and-manifest. Return a LargeFileRef dataclass the caller persists in place of the raw bytes. Fully self-describing — to_dict() gives you enough to get() the bytes back.

MicroCommit uses the same primitive for anything that would otherwise be too heavy for git. When a change is committed, files above the threshold get routed to LargeFileStore.put() — which returns a LargeFileRef. We write that ref to disk as a tiny JSON pointer:

{
  "aither_lfs_pointer": 1,
  "version": 1,
  "ref": {
    "backend": "github-chunked",
    "repo": "aitherium/lfs-mirror",
    "tag": "aither-lfs",
    "base_name": "model-weights.bin",
    "size": 3617841152,
    "sha256": "d5c3...",
    "chunked": true,
    "parts": [...]
  },
  "original_size": 3617841152
}

The pointer is a few hundred bytes. Git happily commits it. The real bytes live in the release-asset store, addressable and integrity-checked. Restore inflates any pointer during checkout.

This gives us a single homogeneous surface — one git clone — that carries files from 12 bytes to 5.7 GB with identical semantics.

Fleet endpoints

AitherRecover on port 8115 (compound aitheros-security-core) now exposes four auth-gated endpoints:

• GET /recover/microcommit/status — every active watcher, counters, last error. Remotes are always redacted before they hit the reply.
• POST /recover/microcommit/register — start a watcher.
• POST /recover/microcommit/unregister — stop it. Idempotent.
• POST /recover/microcommit/flush — force an immediate commit, bypass the debounce window.

The manager keeps watchers in memory only. AitherRecover restart clears every registration — deliberately. Ownership of the fact that a backup should exist stays with the caller. When we wire this into AitherStrata for tenant-scoped continuous backup, Strata re-registers on boot from its own persistent catalog. That boundary is explicit by design.

The trap that nearly hid this

The first bake looked clean. docker buildx bake reported success. docker inspect showed the expected mount points. And the container came up shipping LargeFileStore.py — which had been committed hours earlier — but not MicroCommitBackup.py, which had been committed minutes earlier.

The COPY layer in the intelligence build stage said CACHED.

Reading the Dockerfile:

FROM base AS intelligence

# CACHEBUST must be declared IN THIS STAGE — ARG is stage-scoped, so the
# compose arg reached only app-code-full and was inert for every service
# target that actually ships (D-1737 follow-up).
ARG CACHEBUST=1
COPY --chown=aither:aither AitherOS/*.py /app/
COPY --chown=aither:aither AitherOS/lib/  /app/lib/

That comment is right about the stage-scoping trap. It is wrong about the fix. BuildKit does not rekey a layer just because an ARG was declared — it rekeys a layer when the ARG is referenced. A bare ARG CACHEBUST=1 sitting alone busts nothing. The scratch stage two hundred lines up gets this right; it follows the declaration with a LABEL aitheros.cachebust="${CACHEBUST}". The intelligence stage forgot to do that. So the arg went in, went nowhere, and the COPY next to it cache-hit off the previous build's context.

The fix is one line:

ARG CACHEBUST=1
LABEL aitheros.cachebust.intelligence="${CACHEBUST}"
COPY --chown=aither:aither AitherOS/*.py /app/

The LABEL needs no shell (works even under FROM scratch), references the ARG, and forces every subsequent layer to rekey when CACHEBUST changes. The next bake showed #8 DONE 2.2s and #9 DONE 4.3s where the previous one had shown #8 CACHED and #9 CACHED. MicroCommitBackup.py landed in the image.

There are probably eleven more stages in that file with the same silent bug. That's a follow-up.

What the tests catch

Seventeen tests. The interesting ones:

Debounce is deterministic. The watcher takes a time_fn hook that defaults to time.monotonic(). In tests we pass a _FakeClock and .advance(dt) between checks. Two writes 100 ms apart with a 5-second debounce produce exactly one commit — not two, not zero. Same suite proves that six seconds of silence flushes even without an explicit trigger.

Trigger bypasses debounce. The first version of this test failed — the loop was clearing the flush event before _should_commit() could see it. The fix moved the clear into the commit path and into _tick() when nothing is pending. This is the class of bug where the module imports, the healthcheck stays green, the routine reports "running", and every scheduled commit still fires on time. Only an on-demand flush silently drops. Worth a test.

LargeFileStore failure leaves the file in _pending for retry. When the store put() raises LargeFileStoreError, the file stays queued and no pointer is written. Next tick tries again. Zero data loss during a network blip. The alternative — retry on the next scan by rediscovering the file as "changed" — would work by accident, and would silently stop working if the fingerprint algorithm ever got tightened.

Push failure populates last_error but keeps the local commit. A remote-unreachable case still produces a valid local commit; the counter commits_made bumps; last_commit_sha gets set; last_error is populated so the status endpoint tells you. Nothing gets rolled back. Next push retry catches up automatically.

Remotes are redacted everywhere. The _redact_remote() helper is called before every log line, and every status() reply runs its remote through it. Tokens embedded in a URL never reach the wire.

What this replaces

Every ad-hoc scheduled rsync. Every cron job that tars a directory and uploads to S3. Every "I meant to commit that" moment. Every state-you-had-two-hours-ago that you can no longer reconstruct.

For AitherOS specifically, this is the primitive we needed to close two open loops:

1. AitherStrata cold-tier backup. Snapshots of tenant working sets that today go to a local disk. Point a MicroCommit watcher at the tier root; every snapshot is a commit; the remote is a GitHub org the tenant controls. The LargeFileRef pattern already handles the big files. Restore is git clone + inflate.
2. AitherNexus large-artifact ingest. Currently a bespoke upload path. The /store/put gateway shipped alongside this covers it — Nexus stores a ref, not the bytes. Fetching is symmetric.

Both are follow-ups. Both are unblocked by what shipped today.

The economics

GitHub gives you free unlimited private repos and unlimited release assets. There is no bandwidth cap on release-asset downloads. There is no storage cap for release assets. If you have a GitHub account you already have every-file every-state backup capacity, and you already have five geographic replicas.

We are storing the boot state of a 208-service AI operating system on the same infrastructure people use to distribute a JavaScript library. It costs zero dollars. It restores in the time it takes to git clone.

That is the whole thesis of AitherOS in miniature — take an existing, boring, ubiquitous primitive that everyone already has and use it for the thing they thought they needed a bespoke system for. The bespoke system was Git. It has been sitting on our desk the whole time.

---

Ships in commit 3f64974297 on feat/tunnel-phone-coding. Endpoints live at https://<host>:8115/recover/microcommit/{status,register,unregister,flush} on any AitherOS node running the security compose profile. Manager is in-memory; consumers own re-registration. 17 tests pass; full backup module at 46/46.