13x Smaller and It Beats the Teacher: A Code-Search Embedder for $3

Every number in this post is from one run on 2026-09-02: a single rented A10 (24 GB, $1.29 an hour), one seed, one corpus hash, every stage writing a sidecar you can diff. The whole thing is reproducible from a single command, and we show the command.

The result

We wanted an embedding model that understands our codebase — one that, given a plain-English question like "where is the retry logic for the model scheduler", returns the right directory summary at the top of the list. Small enough to run on a laptop or inside the browser, good enough to trust.

We started from Qwen3-Embedding-0.6B and taught it with NV-Embed-v2, a 7.85B general-purpose embedder that topped the MTEB retrieval leaderboard when it was released. Then we scored all of them on 1,214 held-out questions over 1,576 directory summaries the student had never seen.

Read the first column of numbers. Ask a question, take the top hit: the student is right 80 times in 100. The model thirteen times its size is right 66 times. The same student before training was right 62 times. Recall at ten — is the right answer anywhere in the first page — went from 72% to 90%.

And the int8 export is not a compromise. At 1.06 GB on disk it scores within a tenth of a point of the full-precision student on every metric, with a measured cosine fidelity of 0.9993 against it.

Why a small model can beat a big one here

This is the part worth understanding, because it is not magic and it is not a leaderboard trick.

The teacher is a superb general embedder. It has never seen this codebase. Its notion of "the passage that answers this question" is the one it learned from the whole internet, and on our directory summaries that notion is right about two thirds of the time.

The student is trained on two signals at once:

• The teacher's judgement. For every training question we captured the teacher's similarity to the correct directory and to three hard negatives — directories that look plausible and are wrong. The student is trained to reproduce the margin the teacher sees between right and wrong. That is the distillation half.
• The corpus's own labels. Each training row also says which directory is actually correct, and the student is trained contrastively (InfoNCE) to rank it above the hard negatives. That is the in-domain half.

The second signal is what the teacher never had. So the student learns the shape of this codebase — its naming, its layering, what a summary of a service directory looks like versus a summary of a config directory — on top of the teacher's general sense of relevance. A 0.6B model with the right supervision beats a 7.85B model without it. The lesson generalises: for retrieval over a corpus you own, in-domain training is worth more than parameters.

The eval split is disjoint by directory. No directory that appears in a training row appears in an eval row, so these numbers say the student generalises to parts of the codebase it was never shown — not that it memorised the ones it was.

What the run actually did

Four stages, one script, one GPU. Each stage refuses to run on missing input and writes a sidecar with what it measured.

Capture. Spin up the teacher as a local embedding server, embed 4,275 distinct questions and 3,027 distinct directory summaries, and write one row per training example: question, correct directory, three hard negatives, and the teacher's similarity to each. The teacher's mean margin between the right answer and the average wrong one came out at +0.13 — a real but narrow signal, which is exactly why the in-domain half matters. Embeddings are sharded and checksummed, so a rerun resumes rather than re-embeds.

Distill. Two epochs over 4,862 training rows at batch 16 — 608 optimizer steps, about nine minutes on the A10. Loss went from 1.55 over the first ten steps to 0.48 over the last ten. fp32 master weights with bf16 autocast, AdamW, 5% warmup, linear decay, gradient checkpointing so the full declared batch fits a 24 GB card.

Quantize. Every linear layer's weights rounded to int8 with one scale per output channel; embeddings and norms stay in full precision. The export is 2.24x smaller than fp32 and embeds within 0.9993 mean cosine (0.9991 worst case) of the model it came from, measured through the exact loader the eval stage uses. The gate on this stage is 0.98; it passed with room.

Eval. All four systems — teacher, baseline, student, int8 — scored on the same 1,214 questions with the same metric code. The stage's own gates require the student to beat the baseline on both p@1 and recall@10 and the int8 export to hold 98% of the student's recall@10. All passed.

The recipe is one command

python embed_train.py --corpus code_search_2026-08.jsonl

The driver takes one argument, the corpus. It rents the cheapest GPU with at least 24 GB under a price cap, tracked against a rental ledger; waits for ssh; ships the corpus, the four stage scripts and the teacher server; installs the token the gated teacher needs; builds two pinned environments (the teacher and the student want different library versions, so each gets its own); and starts the pipeline detached with a log you can poll. --status tells you RUNNING, DONE or DEAD and what the GPU is holding. --from-stage quantize reruns just the tail when you change the export format. --collect lands the report and every sidecar in the repo under a dated folder, so the record of the run is in git next to the code that produced it.

Total rented time for the run described here, including the environment builds: about two and a half hours. About three dollars.

What we ship, and what stays home

The student ships. It is Apache-2.0 (Qwen3-Embedding's licence), 0.6B parameters, 1.06 GB as int8, and it is the model we are wiring into our code-search lane.

The teacher does not ship. NV-Embed-v2 is CC-BY-NC-4.0, and we treat that seriously: it ran on our rented box, its similarity targets live in our training record, and nothing derived from those targets reaches a customer surface except the weights of a model that is licensed to. The lineage is recorded in the run's manifest.

What this means if you have a codebase

You do not need a 7B embedding model to search your own code well. You need a small model, a few thousand questions paired with the right answers and some plausible wrong ones, a big model to borrow judgement from for an afternoon, and an evaluation that holds out whole directories rather than random rows. The result is small enough to run on-device.

We will follow up with the browser build: the same student running on-device through WebGPU, embedding your question locally before it ever leaves the tab.