Our 0.6B Distilled Embedder Beat the General-Purpose Ones on Real Documents

We train small embedding models that know one corpus: a large teacher scores your documents, a 0.6B student learns from the teacher's margins and your corpus's own structure, and the student ships quantized with the eval that proves it works. On code search that already beat a model ten times its size.

The fair objection is obvious: code is not documents. A student trained on a code corpus has learned the wrong distribution, and the honest thing to do is test it on prose before claiming anything. So we did.

The setup

One real customer workspace, of the kind this platform actually runs: 26 content chunks from six files — a firm profile, a home page, and four project profiles. The questions were not invented for the benchmark: seven of them are questions users had actually asked of that corpus, taken from the conversation store. The other twenty-one were written against the document text so every file gets covered evenly.

Four embedders, all served the same way (llama.cpp, GGUF, Q8_0, GPU), each with the query convention its authors document — nomic gets its search_query:/search_document: prefixes, Qwen3 and the student get their instruct-style preambles, and we also ran the raw forms. Metrics are document-level: a query counts as a hit when its gold document is what the top-ranked chunk came from, so a chunker cannot flatter or punish a model.

The result

The student wins on prose, and it wins on the seven real user questions too (p@1 0.714, tied with nomic, ahead of Qwen3 at 0.571 and MiniLM at 0.429).

Four things we did not expect

The code prefix does nothing on documents. Identical rankings with and without the Instruct: Given a code search question… preamble. No penalty, no benefit. The distillation appears to teach retrieval geometry that transfers; the specific instruction wording is a code-corpus artifact.

The corpus was being served by the weakest model of the four. The workspace runs with the local embedding mode, which selects the sentence-transformers fallback — all-MiniLM-L6-v2, 384 dimensions. Its real-question p@1 is 0.429 against the student's 0.714. A production RAG was answering with the offline fallback while a better option sat unreleased.

Truncation costs real questions. The volunteer-compute plane stores and verifies 256-d vectors. Truncating the student to 256 dimensions drops real-question p@1 from 0.714 to 0.429. Narrow vectors are the right shape for verifying that two peers agree and the wrong shape for ranking, and the two uses should not share a width.

Throughput is the trade. nomic embeds ~1.6x faster than the student on the same GPU (91.6 vs 58.6 chunks/s), at a quarter of the parameters. Single-query latency is 11.6 ms vs 17.1 ms. For interactive retrieval both are noise next to an LLM turn; for a bulk re-index the difference is real.

What this does not prove

Twenty-eight queries over six documents. A two-query gap is directional, not significant, and one corpus is one corpus. The claim this supports is narrow and worth stating narrowly: a student distilled on code transferred to ordinary prose well enough to beat the general-purpose embedders we compared, on this corpus, including the one serving it. It does not say the student is universally better, and we would not retire a general-purpose embedder on this table alone.

Rerun it on your own corpus

This is now a stage, not a script. awembed compare is black-box over OpenAI-shaped /v1/embeddings endpoints, so your student, the teacher, and any third-party model are measured by the same code on the same rows:

awembed compare --corpus docs.jsonl --queries queries.jsonl \
  --endpoint student=http://127.0.0.1:18101 \
  --endpoint nomic=http://127.0.0.1:18103 \
  --qprefix nomic="search_query: " --dprefix nomic="search_document: " \
  --out compare.json

It refuses to score when the gold column names nothing in your corpus, when an endpoint returns the wrong number of vectors, or when a model is unreachable — because a table of zeros that reads as "all four models are bad" is worse than no table.

What's next

The community side matters here: the same student is the model volunteers embed with in the compute pool, where peers verify each other's vectors, and verified batches earn credit. Running this benchmark told us exactly where its narrow form is safe to use — and where it is not.

The general-purpose embedder on this box is not being retired by this result. The offline fallback that was serving those documents is.