A Reader Fixed Quantized KV Cache on Strix Halo. I Verified It at 262k Context, and His Theory About My Dense Cliff Survived Half Its Test

Part three of the tuning series, and the most community-built one yet. A reader published two fixes that make quantized KV cache fast on this hardware. I verified them on a 128GB box, pushed them to the model's full 262k native context, and then tested his explanation for the dense flash-attention cliff from part one. The explanation came back half right, which taught me more than a clean yes would have.

Two weeks ago a reader posted a llama.cpp fork with two targeted fixes. His diagnosis: quantizing the KV cache, the memory that holds your conversation so far, is expensive on Strix Halo for a reason that barely exists on discrete GPUs. Flash attention has to convert the quantized cache back to f16 for the math, and the stock kernels do that conversion redundantly. A big-cache discrete card hides the waste; this bandwidth-limited box pays full price. His fork branch fixes the Vulkan side for prompt processing and the ROCm side for generation. He asked for independent numbers from 128GB boxes that can push deeper context than his 64GB machine. That is this post.

The verification: his fixes hold, and then some

Same discipline as always: his branch head against its upstream merge base, four binaries (stock and fixed, both backends), every row identity-logged, flash attention pinned on. Qwen3-Coder-30B as the direct comparison to his published numbers, plus my daily Qwen3.6-35B-A3B as the mild case he flagged himself.

The Vulkan gains on my box are larger than his published numbers, the ROCm gains are smaller (my guess is the different batch settings), and the direction is confirmed everywhere. The line that matters most: with his fix, q8_0 KV generation runs 23 to 53% above the f16 ceiling depending on depth, so you get half the cache footprint and more speed, instead of paying for the smaller footprint with your throughput. His fixed build also leaves f16 untouched (flat within noise), and the one blemish I found is small but real: on my MoE, ROCm prompt processing loses 0.3 to 2.4% with the fix. I flagged it to him rather than rounding it away.

The 128GB part came out different than either of us expected. The fixed q8_0 build completed the model's full native 262,144-token context on both backends. But f16 fits at 262k on this box too, so the story is not fit versus no-fit. It is headroom plus speed: at 262k on ROCm, q8_0 generates 65% faster than f16 (12.1 versus 7.3 t/s) and gives up 2.6% prompt processing. One operational note from the same session: a stock Vulkan f16 run at 131k died after 42 minutes with a device loss, same kernel signature as the micro-batch crash from part two, and it is documented in the same upstream issue. Deep Vulkan prefill on this box has a wall somewhere, and it is not his code.

Then he handed me a theory about my cliff

When I asked whether his data-layout fix could help the plain f16 path, where the dense cliff from part one lives, he checked the shader and said no: f16 already reads sequentially, there is nothing for his transpose to fix. Instead he offered a different explanation. His memory microbenchmark showed his box reads about 800 GB/s while the working set fits the GPU's last-level cache, and about a quarter of that once it spills. His hypothesis: my cliff is the per-layer KV working set outgrowing that cache. He even gave me a formula for where the spill should start, and a way to test it: run his membench on my own box, plug my models' dimensions into the formula, and check whether quantizing the cache moves the cliff the way smaller footprints should.

So I did all three. The verdict is partially confirmed, and the parts are worth separating.

The physical premise is real, and now measured instead of assumed. On my box the cache boundary sits between 32 and 40 MiB. Inside it, coalesced reads run at a median of 937 GB/s. Past it, they drop to 227 GB/s. That is a 4.1x bandwidth fall on a perfectly clean workload, no llama.cpp involved.

The footprint effect is real and immediately useful. I ran my dense model with f16 KV against q4_0 KV across depths from 8k to 65k, flash attention on, same session, same build:

One flag on the table itself: the f16 cell at 65k is noisy (its standard deviation is about 20% of the mean, kept and marked in the raw data rather than smoothed), so read that last row as "far below 100" rather than as a precise 35. The picture does not depend on it: f16 falls off the familiar cliff, and q4_0 never does. Its worst single step is a 16% decline where f16 loses 52%. At 32k, q4_0 sits 2.2x above f16 and even 13% above the flash-attention-off control from the same session. Enabled with -ctk q4_0 -ctv q4_0, the cliff has a third workaround now, next to switching backends and turning the flag off, with one big caveat that gets its own paragraph below: this is throughput, and I have not measured what it does to output quality.

The exact formula does not survive. It predicts my dense model spills at 8,192 tokens, as a boundary. What I measure is a progressive decline that starts quietly between 8k and 16k (minus 17.6% for that step) and only becomes the dramatic cliff between 24k and 49k. And q4_0 does not reproduce the same cliff shape at a proportionally later depth, as the formula implies; it smooths the curve instead. To be clear about my own earlier wording: the crossover I published in parts one and two, the point where flash attention on flips from helping to hurting against flash attention off, still stands between 16k and 32k. What this session adds is that the flash-attention-on curve itself starts sagging earlier than that window suggested.

And one thing the theory cannot explain yet. Plugged into the same formula, my MoE model should spill at 16k. But the MoE shows nothing like the dense behavior; part one measured it gaining 42% from flash attention at 32k, the depth where the dense model is deep into its collapse. Both models use the same head size; the dense model carries roughly three times the KV bytes per token through its attention (four KV heads across 65 layers versus two across 41). My working guess, and it is a guess: the MoE's prompt time is dominated by its expert layers, so a slower attention step barely dents the total, while the dense model pays the full price. That question is open, it is the most interesting one on the part-four list, and I have put it to the fix's author.

What I actually changed

The settings from parts one and two stand. New from this round, with an honest caveat attached: quantized KV cache moved from "avoid, it is slow here" to a real option. q4_0 KV is the strongest cliff mitigation I have measured on the Vulkan path, and with the reader's fixes, q8_0 becomes faster than f16 for generation at depth. The caveat: all of this is throughput. I have not measured what cache quantization does to output quality, and that test needs a different harness than a benchmark tool. Until then I treat q8_0 as the conservative option and q4_0 as an experiment, not a default. The maintained configuration, as always, lives in the Setup Pack.

One more change: the numbers from this series now have a permanent home. Two public bench pages went live this week, one tracking llama.cpp builds against a fixed reference set on this box, one collecting model measurements like the ones in this post, each row with its build hash, flags, and configuration attached. The model data from this post is on the model bench page, and both pages get a changelog line whenever something changes.

The open list for part four: the MoE asymmetry above, a controlled ROCm reboot A/B for the IOMMU setting (a reader asked, and my current ROCm comparison is directional rather than isolated), quality testing for quantized KV, and whatever the comment threads order next. Same offer as the last two parts, because it keeps working: measure any of this on your own box and reply with the numbers, with build_commit and the flash-attention mode from the benchmark JSON attached. Half of this post exists because someone did exactly that.