The Checkpoint Ships Its Own Reference: Porting DeepSeek-V4.1-Flash to llama.cpp

DeepSeek-V4.1-Flash is 475.3 GiB across 96,085 tensors, and llama.cpp has no architecture for it. We set out to write one. Rungs 1 through 5 — tensor loading, dimensions, the engram tables, the model header — were mostly bookkeeping. Rung 6, the compute graph, is where it got interesting, because V4.1's attention does something the V4 code already in llama.cpp does not.

The thing that unlocked it

The single most valuable discovery was not in the weights. The checkpoint ships its own reference implementation — inference/model.py — alongside them.

Reading it settled in one pass what four hand-written probes had failed to pin down. We had been reverse-engineering the architecture from tensor shapes and getting plausible, wrong answers. The reference answered the actual questions directly, with comments explaining why.

If you take one thing from this post: when a checkpoint ships its own inference code, that file is the specification. Read it before writing a line of kernel C++.

What the architecture does

V4.1 compresses attention state and shares it across layer groups. The reference's own docstring is the contract:

What attention layers hand down the stack instead of recomputing. Layers run in order and every source writes before its consumers read, so one slot each is enough and nothing needs resetting between forwards. Sources: compress_kv and index_k from kv_source_layers, topk_idxs from index_source_layers, candidates from candidate_source_layer.

Four slots. Three different author sets. That one sentence is the whole of the graph work, and it is easy to read past.

In this model kv_source_layers is [2, 8, 14, 20] and index_source_layers is [2, 8, 14, 20, 24, 28, 32, 36]. Compression ratios are 2 on blocks 2–19, 1 on blocks 20–39, and 0 on the first two. So eighteen blocks carry a ratio that means "compressed", but only three of them own a compressor. Thirteen blocks read state they do not produce.

V4 never hits this. Its compressed ratios sit on exactly the blocks that own the corresponding tensors — every compressed block is its own source. That is why the existing V4 code looks like it should work and does not.

Three things that are easy to get backwards

The index-key owner is the KV list, not the index list. In the reference: self.owns_k = layer_id in args.kv_source_layers. The reason is in the comment above it — the index keys are derived from the compressor's latent, so only a layer that compresses its own KV can produce them. Blocks 24, 28, 32 and 36 are index sources that own no key.

The top-k selection is authored by a third set. It is not the key owner's. Each index layer computes its own selection from the shared keys, with its own query projection and its own weights, because different layers select differently.

Ratio 1 is not a special case in the arithmetic. We spent time planning a separate code path for it before noticing that a softmax over a single value is 1.0 — so pooling over a group of one is the plain projection. The only difference is the gate, and a ratio-1 compressor has none. Three ratios, two code paths.

Two segfaults, and what they were actually saying

Our first graph attempt crashed twice in the same function: build_lora_mm on a null weight, first on the compressor, then on the indexer. Both were the same sentence from the docstring, arriving as a null pointer instead of an error message.

That is the general lesson. A crash on a null weight reads like a missing tensor or a loader bug. It was neither — it was the architecture telling us the state is per-group, and a parameterisation was never going to express that.

What running found that reading did not

A shape-correct fixture — six blocks, about a gigabyte, filler weights — is enough to prove the arithmetic executes, and it caught two things that no amount of reading had:

A tensor marked optional that the graph dereferenced anyway. The loader marks the head's hyper-connection weights TENSOR_NOT_REQUIRED for V4.1, and the checkpoint indeed has 258 of those keys — 43 blocks times six — and zero at the model level. The graph called it regardless. ggml_mul_mat on a null pointer is a segfault, and the flag saying "this may be absent" was never honoured.

A flag carrying two unrelated jobs. One boolean in the cache planner selected the read scheme — genuinely different between V4 and V4.1 — and also gated the padding that keeps a state-index tensor at a fixed width. V4's slots happened to be on the right side of it. V4.1's were not, so on every other decode step a list came out empty and the graph asserted.

Neither was reachable by reading. Both took one run.

What this does not prove

The fixture has filler weights. The numbers it produces — 31.4 tokens/second prefill, 30.8 generation, on CPU — describe six blocks, not a model, and say nothing about how V4.1 performs. What is verified is that the compressed-attention graph executes end to end and that a ratio the graph cannot run aborts by name rather than silently falling through to full attention.

We also know of one confirmed defect we have not fixed: the block-level hyper-connection collapse uses the wrong mix. The reference computes its mixing coefficients and then collapses with a different one than it just computed — the carry from the previous sub-block. Our port does both in one call and uses its own. It is ledgered, not hidden, and it is a graph-structure change rather than a parameter.

What's next

The full 475 GiB checkpoint, converted and quantized down. The engram tables alone are 189 GiB — 39.8% of the file — and they are FP8, not the BF16 we first assumed; measuring corrected us. They are also a pure hash lookup, so a token touches a few hundred of 384 million rows. That makes them a candidate for living somewhere other than RAM, which is the next question worth answering with a measurement rather than an estimate.