The Code-Search Embedder Now Runs in Your Browser Tab

The result

Earlier today we published a code-search embedder that is 13x smaller than the model it was distilled from and retrieves the right directory more often. We said the browser build would follow. It has.

The same student, quantized to Q4_K_M, is a single 396 MB GGUF on our public weights mirror. A page fetches it once, keeps it in the browser cache, and embeds locally from then on. Measured in headless Chromium against the live mirror:

That last pair is the one that matters for search. The gap between "relevant" and "unrelated" is the whole signal a retriever ranks on, and it survives four-bit quantization and a wasm runtime with minimal degradation.

What it plugs into

Our on-device agents already carry a full retrieval engine: a vector retriever and a tag retriever, fused, with an IndexedDB cache for document vectors and per-turn telemetry. Until now the vector half needed an embedding server on the machine. Now the page answers its own embedding requests. The request shape is the OpenAI embeddings API, so the retriever did not change at all; it just started getting vectors back.

Two things gate the download, and both are deliberate. The visitor has to have opted into on-device models on that origin, and a background retrieval call is not allowed to open that dialog on its own. If consent has not been given, the route answers honestly and the retriever falls back to tags. Phones are excluded outright: a 0.6B model in a phone tab is a reboot waiting to happen, and the honest answer for a phone is the local server, which runs the same weights with memory mapping and no tab budget.

How the file was made

The path from the trained checkpoint to a browser-runnable file is short, and two of the steps are the kind nobody writes down:

1. Convert the Hugging Face checkpoint to GGUF with llama.cpp's converter in f16.
2. Quantize with llama-quantize to Q4_K_M for the browser (396 MB) and Q8_0 for servers (639 MB).
3. Write the pooling type into the GGUF metadata.
4. Verify every artifact against the full-precision model on the same texts.

Step 3 is the detail that decides whether the file works. This embedder pools the last token, and the converter does not record that. Without a pooling type in the metadata, llama.cpp refuses embedding requests outright, and the browser runtime we use exposes mean and CLS pooling in its API but not last-token pooling. Baking pooling_type = LAST into the file means every consumer gets the right vector with no flags at all. We measured it: with the metadata set and no pooling option passed, the Q4 file agrees with the full-precision model at 0.989.

The second detail is the end-of-sequence token. The tokenizer appends one, and the model pools on it. The GGUF carries add_eos_token = true, so llama.cpp reproduces that behaviour, which is why the browser vector and the training-time vector line up. If you convert your own embedder and the vectors look plausible but disagree with your training code, check those two fields before anything else.

Fidelity at each stop

A fidelity number is worth more than a retrieval number here. Retrieval metrics tell you the model ranks well; fidelity tells you the file you shipped is the model you evaluated. Any model of the same width produces 1024-dimensional unit vectors with plausible similarities, so "the dims match and the cosines look sane" proves nothing. Compare against the original weights on the same text, or you do not know what you deployed.

What is not in this build yet

This is the CPU build. It runs the model through llama.cpp compiled to WebAssembly, single-threaded, because static hosting cannot send the isolation headers that multi-threading needs. Roughly a second per document is fine for a retriever that caches document vectors and embeds one query per turn. A WebGPU build of the same file would make bulk indexing in the tab practical, and that is the next pass.

The recipe is the same as the first post. Train a small embedder on your own questions and answers, hold out whole directories, verify every quantized artifact against the full-precision model, and ship the one that fits where you want it to run. This one fits in a tab.