Part four of the tuning series. A routine platform upgrade turned into the answer to the question that has run through this whole series. The dense flash-attention collapse that cost me half my prompt processing at long context is not there anymore, and it takes exactly one patch to bring it back. Five builds from one base, and a rule from part one that no longer applies.
This week was supposed to be maintenance. Fedora 43 had reached the point where staying put costs more than moving, so I upgraded the box to Fedora 44, which brought kernel 7.1.5 and Mesa 26.1.5. The ROCm stack had to go: its version locks pinned Fedora 43 packages and blocked the release upgrade, so the machine is Vulkan only now. Every ROCm number in the earlier parts of this series stays valid as a historical measurement, but I cannot re-run it on this box anymore, and I will say so wherever it matters.
While I was in there, I rebuilt llama.cpp with a curated set of patches from the community fork I verified in part three. Nine of them, chosen rather than taken wholesale: the quantized KV work, a scratch-buffer fallback instead of an abort, a couple of layout guards and tests, a MoE prefill prepass, and two changes to the flash-attention path. I left out the experimental variants, the ones the author's own numbers showed as marginal or negative, and the bundled driver stack. Before any of it went into production I ran the correctness suites for the flash-attention and matrix-id operations the patches touch, all passing, and kept the previous binary as a rollback.
Then I benchmarked the new build against the old one to see what the upgrade bought me, and the first result was mildly annoying: my long-prompt prefill with f16 KV cache looked about 4% slower. I noted it as a possible regression and moved on. That number turns out to be wrong, and finding out why is what led to everything below.
The measurement that changed the series
The problem with that first comparison is that it changed two things at once, the upstream base and the patch set. So I built the plain upstream base my production build sits on, with the same compiler and flags, and compared the two directly. Same base, patches as the only difference.
The 4% loss disappeared. On identical code, the patched build is faster everywhere on the MoE model, by 10 to 12% for prompt processing with f16 cache and by 45 to 60% with q8 cache at 32k and 65k of context. So the 4% I measured earlier came from comparing across two different upstream bases.
Then I ran the same comparison on the dense model, and the difference was far too large to call it tuning.
| Qwen3.6-27B, f16 KV, flash attention on, Vulkan/RADV | stock upstream | patched build |
|---|---|---|
| Prompt processing, empty context | 353 t/s | 353 t/s |
| Prompt processing, 32k | 96.6 t/s | 253 t/s |
| Prompt processing, 65k | 29.2 t/s | 197 t/s |
The stock column is the cliff this series has been describing since June: prompt processing collapses once the context grows, and by 65k it has lost 92% of its empty-context speed. The patched column does not collapse. The only difference between those two columns is the patch set, and both ran the same afternoon. At empty context the two are identical, which is exactly what you would expect if the difference is in how the growing cache gets read.
Every number in this post is Vulkan/RADV, and the collapse was always a Vulkan behavior. Back when this box still had ROCm, part one measured the same dense model at 32k gaining 22% from flash attention on that backend instead of losing half its speed. So this patch fixes something ROCm users never had. I cannot re-run that comparison here anymore, because the upgrade took ROCm off the machine.
Which patch, and how I know
Nine patches went in together, so at that point I knew the cliff was fixable and nothing else. Two of the nine touch the flash-attention path: one that rearranges strided f16 KV data into contiguous form before prefill, and one that hoists a fragment load out of an inner loop. I built three more binaries from the same base to separate them. One with all nine patches except the contiguize patch, one with all nine except the hoist, and one with the contiguize patch added to an otherwise plain build.
| Build, dense model, f16 KV, flash attention on, Vulkan/RADV | 32k | 65k |
|---|---|---|
| stock upstream | 94.5 t/s | 29.8 t/s |
| all nine, contiguize removed | 135 t/s | 45.1 t/s |
| all nine, fragment hoist removed | 242 t/s | 181 t/s |
| stock plus contiguize (and its prerequisites) | 237 t/s | 180 t/s |
| full production build | 252 t/s | 198 t/s |
When I take the contiguize patch out and leave the rest in place, the collapse comes back. When I take the fragment hoist out instead, the speed stays. And adding the contiguize patch to an otherwise plain build removes the collapse again, 505% above stock at 65k. The fragment hoist is worth a further 4 to 9% on top, which is a nice gain and not the fix.
Two caveats on that table. The contiguize patch cannot be applied to a naked upstream build; it needs the earlier patches in the series for the infrastructure it hooks into, so the "stock plus contiguize" row really means "stock plus the first six patches". And my drift check on the stock cells came in at 3 to 4% that session, above my own 1.5% threshold, so I would not read the last few percent between the fast builds too closely. Neither of those touches the main result, which is a factor of six.
Generation speed, for the record, does not move at all across any of the five builds. It stays within 0.09%, which fits a fix that only touches the prefill path.
What that does to part one's rules
Part one's central finding was a four-way matrix: flash attention helped my MoE model at depth, hurt my dense model badly on Vulkan, and the right choice depended on backend, model type, context depth, and your ratio of prompt tokens to generated tokens. I published a crossover depth and a break-even calculation to go with it.
Here is the same matrix on the rebuilt box, with the patched build:
| Model | Metric at 32k, Vulkan/RADV | Flash attention off | Flash attention on |
|---|---|---|---|
| Qwen3.6-35B-A3B | prompt processing | 511 t/s | 762 t/s |
| Qwen3.6-35B-A3B | generation | 41.8 t/s | 47.0 t/s |
| Qwen3.6-27B | prompt processing | 183 t/s | 252 t/s |
| Qwen3.6-27B | generation | 7.98 t/s | 9.31 t/s |
Flash attention wins every cell in that table, and it wins at empty context too. Part one needed a matrix, a crossover depth and a break-even calculation for this question. On this build it is one line: leave it on. For the dense model at 32k, the patched flash-attention-on path is 38% faster than the flash-attention-off workaround I recommended in part one, and 160% faster than the flash-attention-on number I published for the same cell.
One thing has not changed: if you run a stock upstream release, the cliff is still there and part one's advice still applies to you. The patch is under discussion upstream, not merged. Until it lands, my recommendation depends on which build you run, and that distinction is now written into the Setup Pack rather than buried in a post.
The theory that was half right
In part three I tested the fork author's explanation for the cliff. His idea was that the working data outgrows the GPU's cache and everything past that boundary runs at raw memory speed. I measured the bandwidth wall on my box and found it exactly where he said it would be, a 4.1x drop past 32 to 40 MB. Shrinking the cache footprint with q4_0 quantization did soften the collapse, as his model predicted. But his formula for where the collapse should start did not match my curves, and it could not explain why my MoE model shrugged off conditions that flattened the dense one. I called it partially confirmed and left the second factor open.
The second factor was the layout after all. When I first asked him whether his data-rearranging work could help the plain f16 path, he checked the shader and told me f16 already reads sequentially, then added a caveat: there is a secondary path that stages through shared memory in some alignment and size cases, and he would not call the answer airtight for every configuration. That caveat was the answer. His contiguize patch exists precisely because strided f16 access does happen on this hardware, and removing it is what removes the cliff.
The bandwidth wall is real and I measured it. The collapse on my dense model came from the layout.
A smaller finding that changed how I bench
The scatter between repeated runs moved with the fix, and by more than the speed did. On the collapsing path, the same deep dense cell varies by 5 to 10% between identical repetitions. On the fixed path it reproduces within about 0.3%. That is a factor of twenty in stability, and it retroactively explains a 20% standard deviation I flagged in an earlier deep run without understanding where it came from.
It also has a practical consequence for anyone comparing numbers on this hardware: a single deep dense measurement on an unpatched build is not worth much. Run it three times before you believe it, or you will end up arguing about a difference that is just your machine's mood.
What I run now
The current state of the reference box, as of this post:
- Fedora 44, kernel 7.1.5, Mesa 26.1.5, Vulkan only. The ROCm stack was removed because its version locks blocked the release upgrade. Earlier ROCm measurements in this series remain valid as history.
- llama.cpp from the fork's branch tip, build 10283. The curated nine-patch build described above ran production for a week and did the work in every table here. I have since replaced it with the full branch tip, built against my own driver rather than the author's bundled one. It contains the same contiguize patch plus his newer work, and it went in under the same rules: correctness suites first, previous binary kept as rollback. A stock upstream build is still the reproducible baseline. It is just slower on this hardware.
- Flash attention on, everywhere. In every cell I have measured on this build, turning it off is slower.
- Kernel command line unchanged from part two, including amd_iommu=off, with the tradeoff I documented after publishing: the NPU driver cannot initialize in that configuration.
- f16 KV cache on my main Qwen presets, deliberately, as a quality precaution for long contexts. Quantized cache is fast now, and part three has the numbers, but nobody has measured what it does to output quality, including me.
The maintained version of all of this lives in the Setup Pack, and the measurements behind it, including the five-build attribution table and the new reference baseline, are on the build bench page with build hashes and configuration tags attached to every row.
Part five is already stacking up, and most of it comes from that move to the branch tip. Against the build this post describes, it adds 42 to 78% prompt processing and 49 to 55% generation on DeepSeek V4 Flash, 39 to 40% prompt processing on a 295B mixed-quant model, and 20% on a coding model at depth. A deep-context device loss that killed a 131k run for me in an earlier session did not recur either: the same case finished in under twelve minutes with an empty kernel log. Beyond that: five models measured on the community toolbox build, including a 122B and two architectures I had never run before; a controlled reboot comparison for the IOMMU setting on ROCm, which a reader asked for and which I can no longer run on this box, so it needs a second machine or a container; and quality testing for quantized KV cache, which is the biggest open question in this series and needs a harness that is not a benchmark tool.
The offer from the last three parts stands, and it has now produced a fix for the thing this series started with: measure any of this on your own box and reply with the numbers, with the build commit and the flash-attention mode from the benchmark JSON attached.
Discussion