Quantized KV Cache Helps Six of My Nine Models and Actively Hurts the Two Newest Ones

The usual advice is to quantize your KV cache because it is nearly free. That holds for six of the models I keep on this box, and it is wrong for the two with the newest attention designs, one of which loses 37 percent of its prompt speed.

Every few weeks somebody asks what actually runs on a 128GB Strix Halo box and how fast. I have been answering that from memory, or from numbers measured on different builds weeks apart, which is not an answer at all. So I put every model I keep on this machine through the same matrix on one binary in one night: three KV cache types, two context depths, identical flags.

The build is the one my production services run, 10283, commit b7b85da9, cut from a community fork whose flash-attention work I wrote about in part four. Fedora 44, kernel 7.1.5, Mesa 26.1.5, Vulkan only, IOMMU off. Flash attention on everywhere, batch and micro-batch taken from each model's production preset where one exists. The whole run took an hour and thirty-nine minutes and the drift check at the end came back at 0.16 percent, so nothing below is day-to-day noise.

Two honesty notes before the numbers. The four largest models ran with a single repetition each to keep the night finite, so their rows carry no standard deviation and small differences on them should not be read closely. And one model, Ling 3.0 Flash, does not load on this build at all; I measured it separately and I will come back to that.

What runs, and how fast

All of these are f16 KV cache at 32k of context, sorted by generation speed. Weight quantization and micro-batch size differ between models, so both are in the table: the quantization is whatever I actually keep on disk for that model, and the micro-batch comes from each model's production preset. Neither is normalized, because the point is what these models do as I run them.

Three things about that table before you read speeds out of it. Micro-batch is not the same everywhere, and it changes prompt processing a lot, so the three models running at 128 are not directly comparable to the ones at 512 on that column. Active parameter counts come from what llama.cpp reports; for Ling and DeepSeek V4 it reports none, and the 21 billion for Hy3 comes from its model card rather than from the file. And Hy3 is a hand-mixed quantization that llama.cpp labels Q8_0, which is not what it is.

The bottom two rows are the useful surprise. A 19 GiB dense model generates at the same speed as a 90 GiB mixture-of-experts model with 295 billion parameters, because the dense model activates every one of its weights for every token while the big one activates a fraction. If you are shopping for a box based on parameter counts, that pairing is worth staring at for a minute.

The other thing worth noting is that everything here fits. A 90 GiB model with a 32k context sits inside the 124 GiB window this box gives the GPU, with room left over. That is the actual argument for unified memory, and it is not visible in any speed number.

What quantizing the cache does

The standard advice is to run your KV cache at q8_0 or q4_0, because it shrinks the memory the cache needs and costs you almost nothing. Here is what that advice does to each of these models, again at 32k, comparing q4_0 against f16.

Each row compares a model against itself, so weight quantization, batch sizes and everything else stay fixed and the cache type is the only thing that changes.

Six models behave the way the advice says: prompt processing stays within a couple of percent, generation gets faster, sometimes by a lot. Three rows, covering two models, go the other way on both metrics at once. Ling loses more than a third of its prompt speed.

The split lines up with attention design, although with only two models on the losing side I would not call that a rule yet. The six that gain all use conventional grouped-query attention, where every layer keeps a key and value cache of the usual shape. The two that lose are the two newest architectures in the set, and each of them does something different with attention. They also turn out to lose for different reasons.

DeepSeek V4 Flash keeps a single KV head with a head dimension of 512 and pairs it with a separate indexer that decides which parts of the context to attend to. There is an open upstream issue about quantizing its cache: doing so switches on an incoherence rotation, and this model's attention dispatch needs that rotation to be absent before it will use its sparse paths, so affected layers fall back to plain attention. The issue reports that this also corrupts the output, confirmed on CPU and on CUDA.

Since that would make my two rows a measurement of something nobody should run, I went and checked, and I cannot reproduce the corruption here. With q8_0 keys and values the model answers short questions correctly and stops cleanly, the startup log allocates the same sparse-attention state buffers as the f16 run, and it still reports the indexer as enabled. That is not a clean bill of health. Two short answers are not a quality test, and losing 13 percent on both metrics still looks like something taking a slower route through the graph. I could not run the obvious isolation either, quantizing keys while leaving values at f16, because this build refuses mixed cache types outright.

The practical advice comes out the same and it is simpler than a tradeoff: on this model a quantized cache costs speed and buys nothing, so leave it at f16.

Ling 3.0 Flash is more extreme. Of its 43 layers only eight carry a normal KV cache at all, and the rest use linear-attention variants that keep a fixed-size state instead of a growing one. The practical consequence is that its cache at 32k of context needs 0.34 GiB. For comparison, Hy3 needs 10 GiB at the same depth and my dense 27B needs 8.1. When the cache is already small enough that reading it costs nothing, compressing it cannot save anything, and the unpacking work is pure overhead. That is the cleanest explanation I have for a 37 percent loss, and it predicts that the effect gets worse with depth rather than better, which is what the depth-zero rows show: at empty context all three cache types are identical to within half a percent.

What the numbers do not explain

The split into winners and losers is clean. The size of the win among the six winners is not, and I want to be plain about that rather than dress it up.

Qwen3-Coder-30B and Qwen3.5-122B-A10B need exactly the same number of cache bytes per token, 4 heads at dimension 128 across 48 layers versus 2 heads at dimension 256 across 48 layers. Same product, same footprint. One gains 30.8 percent and the other gains 4.5. My dense 27B has the second largest cache in the whole set at 8.1 GiB and gains only 6 percent, while Coder gains five times as much with a cache a third the size.

The obvious candidate is how much of each model's decode time goes into reading the cache rather than reading weights. A model with 10 billion active parameters spends more of its time on weights than one with 3 billion, so the same cache saving matters less. That reasoning fits the 122B against Coder comparison and then falls over on Hy3, which has 21 billion active parameters and still gains 28 percent. Head dimension may be involved too, since 128 and 256 behave differently in the flash-attention kernels on this hardware. I do not have enough models to separate those factors, and I would rather publish the table with an open question on it than invent a rule that the next model breaks.

The one model that needed a different build

Ling 3.0 Flash refuses to load on build 10283. Its architecture, BailingMoE3, is not supported there (yet). That support exists in an open upstream pull request, and I keep a candidate build carrying it, so I measured Ling on that binary instead, build 10289, commit 017a7247.

Measuring one model on a different binary than the other eight is exactly the kind of thing this series complains about when other people do it, so I did not just note it and move on. Before the Ling cells I ran the same reference model on both binaries. The largest difference across all four cells was 0.35 percent, which is inside the noise of this box. The Ling rows can sit next to the rest.

One caveat specific to that session: its drift check flagged 2.14 percent on the f16 prompt-processing cell at 32k, which is the number the 37 percent loss is measured against. That could change the loss by a percentage point or two in either direction and changes nothing about the conclusion.

Ling can do multi-token prediction, and these numbers do not use it. The GGUF carries the tensors for it, bailingmoe3.nextn_predict_layers is set to 1 and the prediction tensors sit on layer 42. I run it without, because the draft path stalls during first-token generation on this stack, and a speculative-decoding run that hangs is not a measurement. So treat the Ling row as its floor rather than its ceiling.

Two quants of the same model

I keep two Q2-class quantizations of DeepSeek V4 Flash, because I have not decided which one to keep. One is IQ2_XXS at 2.06 bits per weight and 80.8 GiB, the other is Q2_K at 90.2 GiB. The larger one generates 8.6 percent faster at 32k and processes prompts at the same speed.

A bigger file that decodes faster looks backwards, since decoding reads weights and more weights means more reading. The quantization families explain it, though I have not isolated it properly: the i-quant packs more weight into fewer bytes and pays for that with more work to unpack each one, while the k-quant is fatter on disk and cheaper to decode. On a box where the memory bus is usually the limit, the fatter format still wins here, which says the unpacking cost is the binding constraint for this model rather than the bandwidth.

What this does not tell me is which one to keep, and that is worth saying because it is the same trap the cache advice sets. A throughput bench measures throughput. Whether a Q2 quantization of a 284 billion parameter model still writes usable code is a different question that needs different work, and nobody in this comparison, including me, has done it.

What I run now

Current settings on the reference box, as of this post:

• f16 KV cache for DeepSeek V4 Flash and Ling 3.0 Flash. On these two, quantizing the cache costs speed on both metrics, so there is nothing to trade.
• q8_0 KV cache for the large mixture-of-experts models I run on demand, Laguna S 2.1 and Hy3, where it buys real generation speed at depth and the memory headroom matters.
• f16 KV cache on my main Qwen presets, deliberately, even though the table says q4_0 would be faster. Someone has measured what cache quantization does to output quality: oobabooga published KL divergence numbers in April that cover both of these Qwen models, and they come out well, staying under 0.04 at q8_0. My reason for keeping f16 is that these are the models I hand long documents to and I have not run that comparison on my own workload, which is caution rather than evidence.
• Flash attention on everywhere, unchanged from part four.

The maintained version of all of this is in the Setup Pack, and the per-model rows with build hashes, cache types and raw JSON paths go on the model bench page.

What I would like from anyone running these models on their own Strix Halo: the same comparison on a model I do not have, especially another architecture with linear or latent attention. If the split I am describing is real, it should show up on any model that keeps a small or compressed cache, and it should not need my box to appear. Post the numbers with the build commit and the cache type attached, and I will put them next to mine.