One Embedding File, Every Lane: 7 Milliseconds on the Server, a Tab in the Browser
The result
Three posts in one day about one small model, and this is the one where it stops being a project and becomes infrastructure.
Our distilled code-search embedder now runs in production from its quantized GGUF through llama.cpp. The full-precision engine that served it this morning is retired. We measured the two side by side on the same GPU, same texts, same reference, before touching anything:
| probe | GGUF Q8_0, llama.cpp | bf16, full engine |
|---|---|---|
| agreement with the original fp32 weights (cosine) | 0.9997 | 0.9999 |
| agreement between the two lanes | 0.9996 | |
| query vs relevant / unrelated directory | 0.766 / 0.088 | 0.766 / 0.089 |
| latency per embedding | 7 ms | 31 ms |
| time from start to healthy | 9 s | ~90 s |
| GPU in use, whole card | 30.0 GB | 30.7 GB |
The retrieval numbers are the point. A search engine ranks on the gap between "relevant" and "unrelated", and that gap is identical to three decimals across the two runtimes. Everything else on the table is a gift: four times faster, ten times quicker to start, and roughly a gigabyte of a busy GPU returned to the models that need it.
One file, three homes
The artifact is the Q8_0 GGUF from this morning's browser post. Nothing about it changed. It now lives in three places:
- the server, where llama.cpp serves it on the GPU behind the same OpenAI embeddings endpoint the previous engine answered, under the same model name and the same hostname, so no consumer noticed the switch;
- the browser, where the Q4_K_M sibling of the same conversion runs in a tab through llama.cpp compiled to WebAssembly;
- the mirror, where anyone can fetch either file and run it with any llama.cpp build.
That is what the two metadata fields from the browser post buy you. Because
pooling_type and the end-of-sequence rule are baked into the file, every
consumer computes the same vector with no flags to remember. The server lane
was configured with one line: the model path. Pooling, tokenization and
normalization came from the file.
How the switch was made safe
Two checks decided whether the switch could happen, and both ran before and after it.
The first is fidelity against an independent reference. We keep a probe sentence and its embedding from the original full-precision weights, computed in plain PyTorch, pinned in the repo. Whatever is serving the model has to reproduce that vector at a cosine of 0.99 or better. Quantization of the same weights lands between 0.985 and 0.9997 on that probe. A different model of the same width lands near 0.1. The floor sits in the wide gap between those two, so it separates "same weights, different arithmetic" from "not the model you evaluated", which is the only question that matters when you swap a runtime.
The second is that the served path has to be the mounted checkpoint. Serving engines are helpful; several will quietly fall back to a default model if the one you named is not where they looked, and answer under the name you gave them. The gate reads the path the engine reports it loaded and refuses anything that is not the file we mounted. With llama.cpp that path comes from the server's properties endpoint rather than the models list, and the gate reads both.
Both checks pass on the new lane. The consumer that embeds every code-search query was restarted and its own end-to-end probe passed against the unchanged hostname.
The rollback that was never needed
The switch ran side by side first: the new server on a spare port, the old one still answering. Only after the table above was measured did the old unit stop, its definition moved out of the service directory rather than renamed in place, and the new unit take over the port and the hostname alias. Putting the old definition back is one move and one restart. It stayed in its retirement folder.
What this means if you run a small embedder
Quantize it, bake the pooling and token rules into the file, and serve the file with llama.cpp. Pin one reference vector from the original weights and make every runtime prove it reproduces that vector before it takes traffic. Do the same check on the path the engine says it loaded. Then measure the switch on the same GPU with the old lane still up. For a 0.6B embedder the reward is four times the throughput, a tenth of the start time, and one artifact that runs in a tab, on a laptop CPU, and on a production GPU, producing the same vector in all three.