microembedder: An Embedding Lane That Runs Entirely in the Browser

An agent that runs in your browser needs memory that runs in your browser. Not a round trip to a fleet, not a model that hogs the GPU the decoder is already using — a small, fast retrieval lane that captures what the agent sees, keeps it locally, and answers "where did I see that?" in milliseconds.

We built that lane. We named it microembedder. And then we spent a rented GPU proving what doesn't make it better — which is the part that matters most.

The lane: capture → chunk → embed → query

The design constraint came first: embeddings never touch the WebGPU queue. Decode is memory-bandwidth-bound; the moment embeddings share the device, every token slows down. So the lane is three lanes:

• GPU worker — decode and prefill only, never starved.
• CPU workers — chunking, tokenization, tool-result parsing, and the embeddings themselves: a tiny model in WASM on a separate thread.
• Storage — IndexedDB, session-shaped (a few thousand chunks, brute-force cosine is instant).

The capture adapter takes whatever the session actually produced — a page snapshot, a browse payload, live DOM text — normalizes it, chunks it on paragraph boundaries, embeds the chunks, and stores them with recency-aware scoring. Query returns ranked results where content wins and recency breaks ties.

Two details we're proud of:

Diff-indexing. Re-capturing the same page costs zero embed calls — a stable content hash means unchanged chunks are skipped before the model is even asked. One section changed? Only that chunk gets re-embedded, and chunks the page no longer produces are removed, not left to rot as stale retrievable units. Reindex cost scales with what changed, not with the corpus.

Refusal, not stubs. Until a real model is registered on the weight mirror, the lane refuses — a loud error, a visible unavailable in stats. It never returns hash-based placeholder vectors that would look like real retrieval. A fallback that returns plausible garbage is worse than a refusal: every downstream signal would agree the feature works while it retrieves nothing.

Verified the honest way: 32 unit tests, and a live probe that runs the compiled store against a real Chromium tab's IndexedDB and asserts every operation end to end — because jsdom can't give you a real database.

The part we measured instead of guessed

Here's where the story gets interesting. We had a hypothesis: distill a tiny student from a big embedding teacher — a standard move — and the student would beat the stock model on our code-search eval. We ran it properly: 8 epochs on a rented A10, monotone convergence, teacher agreement climbing from essentially zero to +0.57.

The distilled student lost. recall@10: 0.667 vs 0.722 for stock — one row on an 18-row eval. The fail-closed gate refused to publish; nothing shipped.

And the interesting part is why: at baseline, the teacher's own relevance scores had zero correlation with retrieval success on this eval (teacher_cosine = −0.01). The student faithfully learned the teacher's opinion — and the teacher's opinion was wrong for this task. You cannot distill a signal the teacher doesn't have. The rental bought us a measured answer, not a guess: the model axis is not the lever for this corpus.

That's the system working, not failing. The gate refused. The artifact was preserved. And the number that killed the bet was available for free at baseline time — the tell was sitting in the log before the first epoch ran.

What a real answer needs: a real eval

The deeper problem the run exposed: an 18-row eval cannot measure anything. One row flips the verdict — it just did. So we built the measurement:

• Harvested real queries from actual session traces — 286 traces → 100 deduped questions → 99 grounded to directories by rule (agent names, subsystem tokens), never by running retrieval against itself.
• The eval grew from 18 to 99 rows — and the pipeline's own corpus gate accepts it unchanged. The next embedding bet runs on real questions people actually asked, with a number that means something.

What's next

The lane ships in awkit, our browser SDK — the same shared-brain worker that already serves the on-device models. The tiny embedding model lands on our weight mirror next, and the lane arms itself the day it does — no code changes, just a config.

Beyond that: the routing half of our LLM plane is being extracted as awrouter — the "OpenRouter for your own fleet" — model resolution, backend discovery and failover, context-window fitting, streaming, standing alone with zero monorepo imports so anyone can run it against their own backends. Registered, planned, and on the way.

The through-line of all of it: spend on measurement before you spend on models. The lane is real, the eval is real, and the next bet — whenever anyone makes it — will be decided by data instead of hope.