Mixed-precision compression of DeepSeek-V4-Flash

Assigning precision per tensor family compresses DeepSeek-V4-Flash-0731's 284B weights to 80.76 GiB, with all 256 routed experts intact, running at a physical context of 262,144 on a single 128 GB machine.

What matters here is not the compression ratio but the asymmetry of the allocation. The model activates only 13B parameters per token, yet almost all of its bytes sit in the routed experts: 43 layers, 256 experts each. Two-bit precision goes to that one family and nowhere else. Attention projections, shared experts, the router, embeddings, the output head, norms and scales all stay at Q8 or higher. The narrowest bit widths land exactly where redundancy is highest, and nothing on the path every token must traverse loses precision at all.

Compression got the model into the machine. It did not get it to the target speed. Closing the remaining gap requires speculative decoding, and the second half of this report is what we found there: in this continuous implementation, the lossless guarantee does not hold. Same prompt, temperature 0, three verification depths, three mutually different but individually stable outputs.

The recipe

Precision is assigned by tensor family, not by layer.

Routed expert gate/up go to IQ2_XXS and down to Q2_K. Attention projections, shared experts, and the output head stay at Q8. Embeddings, the compressor, and the indexer stay at F16, as does the router. Norms, scales, and critical scalars stay at F32.

Four things are pinned and may not move: keep 256 of 256 routed experts, keep top-6, skip no layers, leave the sparse attention mask alone. These are not the same class of operation as quantization. Quantization changes how weights are represented; these four change the model function itself. Once any of them moves, comparison against an official baseline stops meaning anything — and pruning experts or lowering top-k are precisely the two allowances most easily taken while compressing.

The DSpark drafter, extracted from the same source revision, converts separately into a 5.64 GiB sidecar. Same revision is a hard requirement: identical structure does not imply identical draft distribution, and mixing versions changes acceptance rates and the inputs to residual correction while remaining unauditable. We ran a tensor inventory comparison between 0731 and the preview release; every difference falls in the mtp.* namespace. The preview carries the older MTP structure, 0731 carries a redesigned DSpark, and even the tensor names differ. The no-mixing rule therefore has structural evidence behind it, not just discipline.

Conversion auditing is per tensor: every source tensor must land in exactly one target tensor or carry an explicit reason for not converting; shapes, strides, scales, expert ids, and layer ids are all checked; and dequantization error is sampled by quantization type — 660 IQ2_XXS and 316 Q2_K samples on the sidecar side, and full coverage of BF16, F8_E4M3, and F32 to each target type on the target side.

The engine overruled the recipe on paper

The first full conversion followed the architecture document literally and explicitly forced embeddings, compressor, and indexer to q8_0. The smoke load refused it outright: tensor token_embd.weight has type q8_0, expected f16. The engine hardcodes an expected layout for those tensors, and what the recipe says is irrelevant.

Reverting to the template-native recipe loaded fine. The only deviation from the paper recipe is that compressor, indexer, and embeddings sit at F16 rather than Q8 — higher precision, not lower — and 80.78 GiB still fits under the 81.5 GiB cap. This kind of conflict is worth recording on its own: the feasible region of a quantization recipe is defined by the engine, not by the document, and you only find out by actually loading once.

The second conflict was a build flag. With CUDA_ARCH empty, make cuda-spark produces D2R IQ2_XXS/Q2_K kernels that are arch-gated and return an error at runtime; two CUDA regressions failed. Building natively for sm_121a turned the regression suite green, with fused D2R parity at 0.00000000. The sm_121/sm_121a requirement in the architecture document is not a suggestion.

The imatrix chicken-and-egg

Calibrating an imatrix requires running calibration text through the runtime, and the runtime requires an already-converted artifact to start. Our ordering was: convert a bootstrap artifact using synthetic importance (the tool's fallback, sum of squared weights), bring the engine up, collect an imatrix specific to those weights, then reconvert the real target.

Collection hit another tool boundary. The fork's --imatrix-dataset is Metal-only and errors out under a CUDA build demanding --metal. We collected with llama.cpp instead (GGML_CUDA, ARCHS=1210), which reads the ds4 GGUF format fine and emits a standard .dat. The calibration dataset hash goes into the manifest, bound to the artifact id.

One observation worth recording: the synthetic sidecar produced zero speculative speedup. At 32K context, temperature 1.0, 300 tokens — speculation off measured 12.38 tok/s, K=1 measured 11.87, K=5 measured 11.91. Acceptance was too low and verification overhead consumed the entire gain. That is not speculative decoding failing to work; it is an uncalibrated drafter. The target's imatrix is the governing variable.

Capacity

The nameplate 128 GB is decimal. Measured on the machine, MemTotal is 127.6 GB — about 119 GiB — and pure read bandwidth measures 250.3 GB/s.

The final target artifact is 86,720,111,552 bytes, or 80.76 GiB. Add the 5.64 GiB sidecar, 12 GiB of hot cache, roughly 6 GiB of KV and index for a 262K context, about 2.5 GiB of CUDA graphs and workspaces, and about 4.5 GiB for the OS and resident daemons: the design point lands a little over 115 GiB, leaving three to four GiB of headroom against 119.

Measured: with a warm page cache, 80.76 GiB maps in 15.6 seconds. A single prefill of 245,769 tokens ran at 704 tok/s, generated 65 more tokens, and exited cleanly. The lowest MemAvailable at any point was 2.24 GiB.

That run does not count as a formal capacity gate. The frozen prompt was 245,760 tokens; the engine reported 245,769, because the HTTP layer adds nine protocol tokens to the raw prompt. Those nine tokens disqualify it as formal evidence. The 4 GiB memory floor has to be adjudicated on the 220K warm case, which has not been run.

What compression does not solve

Single-stream decode is bandwidth-bound, not compute-bound. Every generated token requires a full read of the active weights — roughly 11 GB at this artifact size — and at 215–250 GB/s of effective read bandwidth the physical ceiling is 19–22 tok/s, with real implementations landing lower. Our measured plain decode is 15.9 tok/s at short context and 12.4 tok/s at 32K.

Compression has already pushed that number as far as it goes. Compressing further costs quality rather than bandwidth: two-bit precision already went to the family holding the most bytes, and every remaining family combined cannot free up another order of magnitude.

So the remaining gap has exactly one path: commit more than one token per sweep of the target weights. A small model proposes K tokens and the large model verifies them in one pass. The official 0731 checkpoint ships that small model attached at mtp.0..2, over target layers 40, 41, and 42. The lever looks free.

Speculative decoding is advertised as lossless, and the guarantee rests on one line: accept with probability min(1, p(x) / q(x)), and on rejection sample from a normalized max(p − q, 0), which preserves the target's own distribution exactly. Provided p really is the target's distribution.

Width drift

We fixed one greedy request — temperature 0, top_p 1, so sampling randomness is off the table — and swept verification depth against an isolated public sidecar. With speculation off, the model wrote 71 tokens and stopped. At K=1 and K=2, 84 tokens. At K=3 and K=4, 128 tokens. Each cluster is byte-stable across independent restarts and different from the others; every finish reason was stop, and the response SHA-256 values are on record. Two independent restarts with speculation off produced identical bytes, so this is not environmental noise.

The second round of evidence changed both artifact and memory path: the final imatrix target on the production device-cache path, three different coding prompts per depth, 100 tokens each. At K=1 all three matched speculation-off byte for byte. K=2 through K=5 all diverged.

Only both rounds together give the conclusion. K=1 cleared a limited comparison on the production path, which reads easily as "K=1 is safe" — but on the public sidecar, K=1 also diverged from the plain target. Clearing one limited comparison is not exactness. We locked production at K=0 rather than K=1 on that basis.

The failure sits before acceptance, not in rollback. The implementation packs the current token plus K drafts into a single target forward of width 1+K, and that forward is not the same numerical path as a width-1 decode: cross-row MoE reduction order differs, floating-point addition is not associative, and CUDA dispatches a separate vectorized tier for widths 2 through 8. The first speculative token is taken directly from that wide forward, so the deviation has already happened before acceptance, rollback, EOS handling, or usage accounting get any chance to intervene. The target at width `1+K` is not the target at width 1.

The most direct evidence is in the deployed source. Its own gate comments state that width-1+D batched verification is not bit-identical to a width-1 decode, attribute this to cross-row MoE floating-point reduction, and mark accept(A≠E) as informational. It then substitutes a weaker invariant for "matches plain decode": the token stream need only be invariant between two forced rejected-draft values at the same verification width. That proves causality of the rejected tail, not equivalence to width-1 decoding. The rollback comment is equally direct — committed cache values carry the wide-forward differences inherently, so a value difference there is expected.

Rollback can drop the rejected rows. It cannot change one thing: the row that was accepted was computed on the other numerical path. Greater depth makes it easier to observe, but this is not a bug that begins at depth 3. It is a property that exists from depth 1 up.

We classify this as an engine defect rather than acceptable quantization noise. Speculation is only allowed to affect speed. If turning it off changes the output, the implementation is not exact, and the fix belongs in the engine — the drafter is not a quality knob.

Why this is not a host-side patch

Exactly one exact repair is available off the shelf: sequential verification, running every draft through the established single-token path. It is correct, and it is deliberately slow — the author of another fork labels the switch exactly that way in the documentation. What it gives up is the amortization of the weight sweep, which is the entire source of the gain. In the current implementation, exactness and speedup are opposing line items in one budget.

The real repair is new kernels: read each layer's weights once while bit-exactly reproducing width-1 arithmetic, routing decisions, cache write order, and output head. That is not something a host-side change to acceptance or rollback logic can deliver.

Even exact, the payoff is unproven. We first fixed sidecar load compatibility — the legacy naming schema would not bind against the newer loader, and a capacity guard made depth 1 silently emit zero drafts, logging drafts=0 without an error. After that, K=1 reached 96.77% acceptance: 31 drafts, 30 hits, with output bytes, finish reason, and usage identical to speculation off. Speed was 14.3 tok/s on against 21.1 off. In a separate sweep against the final artifact, K=1 measured 15.5 against 15.4 off. We have not once observed speculative decoding produce a net gain.

If you run speculative decoding locally

Three things you can do on your own deployment.

Sweep verification depth with a fixed greedy request rather than testing one depth. Temperature 0, top_p 1, one prompt, K from 0 up to the maximum you intend to use, comparing output byte for byte. Testing a single K lets you read "this depth happens to match" as "the implementation is exact."

Reproduce every depth across independent restarts. Without that, you cannot separate a deterministic implementation deviation from environmental noise. Our data supported a conclusion only because two independent speculation-off restarts were byte-identical.

Do not treat acceptance rate as evidence of correctness. They measure entirely different things: a high acceptance rate says the draft model guesses well, and says nothing about whether the verification path is numerically equivalent. We hold a sample with 96.77% acceptance and divergent output.

What we do not claim

• We do not claim this two-bit artifact is token-for-token equivalent to the official weights. The quality of mixed-precision compression has to be adjudicated by an evaluation gate against a frozen official baseline, and we have not run that gate once. Until we have, everything above is engineering observation, not a quality conclusion.
• We do not claim to have localized the first numerical divergence to a layer or operator. That requires layer-by-layer logit and hidden-state traces, which we have not done. The conclusion does not depend on it: the deployed source already declares wide verification non-bit-identical, and the end-to-end greedy output demonstrates the consequence.
• We do not claim width drift applies to all speculative decoding implementations. It applies to those that pack drafts into one wide target forward where width changes kernel dispatch or reduction order. Sequential verification that runs each draft through the single-token path is not affected.
• We do not claim K=1 is safe. It was exact in a limited comparison on the production path and divergent on the public sidecar.
• We do not claim the 4 GiB memory floor holds under a 220K warm load. That case has not been run.

Acceptance work on this machine is far from finished: of the eight gates we set for it, one has passed, and the rest are missing full deployed-runtime evidence, queued, or paused.

On compression itself the conclusion is clean: 284B, every expert, a 262K context, resident and running inside 119 GiB. A 97% acceptance rate says the drafts guess well. It does not say the verification is right.

Appendix A: the memory situation

Steady state on a machine like this is 117 GiB used and 3.8 GiB free, which means the system's own OOM daemon treats the model as the anomaly.

earlyoom triggers at 6% by default — about 7.3 GB — while the production path idles between 4 and 8 GB. The engine hardcodes oom_score_adj=1000, volunteering itself as the first sacrifice. We patched it to accept -900 and spent two days before spotting one line of stderr: Permission denied. The kernel rule is that an unprivileged process may only raise its own oom_score_adj, never lower it, so the patch's premise had never held.

Load transients were worse. A 0.2-second poll showed 4.4 GB; journalctl recorded an actual floor of 1.6 GB. earlyoom worked down the badness ordering and took out a Gitea instance as collateral. Pre-evicting 86.4 GB of page cache did nothing — file pages at the trough total only about 7.5 GB, so the trough is structural. Sacrificing a 10 GB balloon process did nothing. Lowering the weight cache limit failed outright, because one span is mandatorily resident and there is no graceful degradation path.

The clean fix was two things: add the model process to earlyoom's avoid list, and mount 16 GB of swap. The model is CUDA-pinned and never touches swap, but its existence breaks earlyoom's second trigger condition, and the steady state goes quiet.

Appendix B: KV prefix hits depend on the bytes the client echoes

At 850 tok/s, a cold 220K prefill is 259 seconds. In a multi-turn tool session, KV prefix reuse is therefore not an optimization but a precondition for the product being usable.

The server's on-disk checkpoint matches on token-text prefixes, and how much hits depends entirely on the byte fidelity of the history the client echoes back. Same conversation, three ways of echoing it: drop reasoning per the official default and send canonical DSML blocks, and 363/363 cache, stopping at the first turn's prompt boundary with everything after recomputed; keep an empty think marker and replay the exact sampled DSML block, and it caches 363 plus the increment; echo reasoning_content and replay the sampled DSML bytes verbatim, and it caches 428/448, writing only 20 new tokens and saving roughly 95% of the continuation prefill.

The reason is mundane: clients normalize the model's tool calls into JSON before sending them back, and if the bytes the server re-renders differ by one character, the prefix breaks at that character.

So the gateway defines two client contracts. First, put the response's reasoning_content back into the next turn's assistant message unmodified — those are raw sampled bytes and the parser does not strip them. Second, echo the tool_call id we assigned; we use it to retrieve the original sampled DSML block and substitute it at encode time by canonical-block uniqueness. Not by ordinal position: the tools template itself contains example DSML blocks, so ordinal alignment substitutes into the wrong slot, and a regression test holds that line.

An honest boundary: those three hit counts come from server logs. The gateway-side replay wiring is currently verified against a mock transport, and the live end-to-end path has not been closed within a single run, so we do not treat it as passing evidence.