Turning flash attention on at 32k context makes my MoE model 41% faster at prompt processing on Vulkan and my dense model 51% slower. On ROCm, the identical flag speeds up both, and doubles deep-context generation.
I run a Strix Halo box (128GB unified memory) as my daily local-inference machine, and the box has a reputation: decent capacity, modest speed, and a folk tradition of tuning tips. Kernel parameters, environment variables, driver choices, magic flags. After one very thorough reader filled the comments of last week's benchmark post with more tuning suggestions than I could answer from memory, I collected the tips, added the ones from the guides, measured what I could on a pinned build, and ranked the levers by their actual effect on my box.
The flash-attention, MMVQ, hipBLASLt and rocWMMA numbers were measured this week on one pinned build (9870, commit 2d973636e), Mesa 25.3.6 (RADV) for Vulkan, TheRock 7.13 for ROCm, kernel 7.0.11 (itself a lever, held constant here), batch sizes 2048/512, mmap off for every core-matrix run, and the host's tuned profile sat on accelerator-performance throughout. Build hash and the requested flash-attention flag come from the benchmark's own JSON for every row. Older results from my published series and third-party numbers are labeled as such. Two models throughout, the ones I run daily: Qwen3.6-35B-A3B (MoE) and Qwen3.6-27B (dense), both Q5 quants, with prompt processing (how fast the model reads your input) and generation (how fast it writes the answer) each measured at an empty context window and at 32k tokens of context.
Lever zero: know what you're actually running
I earned this one the hard way, in public. Last week's post claimed a llama.cpp update had moved my numbers by +38% and -45%. A reader challenged the attribution, a git bisect followed, and the truth came out in two layers: my "old" column had been produced by a stale, mislabeled binary from May, and the swings were not performance changes in the inference code. A single commit had changed llama-bench's default flash-attention mode from off to auto, my harness didn't pass -fa, and so my before and after columns had benchmarked two different operating modes. The full correction is on that post.
The lesson costs one line of discipline and would have saved me the whole episode: llama-bench's JSON output records build_commit and the requested flash-attention mode for every run. Log both next to every number. One gap remains: if a run uses the automatic mode (-fa auto, the default since that commit), the JSON only records that auto was requested, not which mode the run actually ended up in. The only way to find out is to bench auto once against explicit off and on and compare. That habit catches mislabeled binaries and drifted defaults, which is exactly what got me.
Flash attention at depth
That mess pointed straight at the biggest tuning lever on this hardware: the flash attention flag itself. So this time I measured it properly. Same build (9870), both backends, both models, flag off versus on.
Vulkan (RADV), 32k context:
| Cell | FA off | FA on | Change |
|---|---|---|---|
| MoE prompt processing | 474 t/s | 670 t/s | +41% |
| MoE generation | 41.2 t/s | 46.7 t/s | +13% |
| Dense prompt processing | 172 t/s | 85 t/s | -51% |
| Dense generation | 8.0 t/s | 9.3 t/s | +17% |
ROCm (TheRock 7.13), 32k context:
| Cell | FA off | FA on | Change |
|---|---|---|---|
| MoE prompt processing | 464 t/s | 567 t/s | +22% |
| MoE generation | 21.9 t/s | 41.5 t/s | +89% |
| Dense prompt processing | 164 t/s | 200 t/s | +22% |
| Dense generation | 4.3 t/s | 9.1 t/s | +113% |
With an empty context window, the flag barely matters on either backend (everything within about 2%). Once the window fills up, at 32k tokens of context, it decides more than any other switch I tested, and in ways no single rule of thumb covers:
The flash-attention penalty for dense models is Vulkan-specific. On Vulkan, turning it on halves my dense model's prompt processing at 32k. On ROCm, the same flag speeds the same model up by 22%. The community wisdom I found says flash attention "may cost a little at small context." What I measure is a 51 percent drop on one backend and a 22 percent gain on the other.
The dense penalty appears only beyond 16k context, then hits hard. A depth sweep at 0, 4k, 8k, 16k and 32k shows no gradual decline: at 16k, flash attention still speeds up dense prompt processing by 11 percent. At 32k it costs 43 percent. Up to 16k the flag helps the dense model on both axes. The MoE benefit builds steadily with depth instead (+1.7% at zero, +38% at 16k, +41% at 32k), and on ROCm every sampled depth improves monotonically, generation included. The sweep ran a day after the matrix, and its FA-on dense endpoint reproduces within 2 percent while the FA-off endpoint drifted lower, which is why the loss reads 43 percent there against 51 in the matrix. The crossover point itself did not move between the two sessions.
On ROCm, flash attention decides your deep-context generation speed. If you run ROCm with long contexts and the flag off, you are running at half the generation speed you could have, for both models. Turn it on.
Auto is a convenience, not an optimizer. In llama-bench, auto closely matches explicit on across both backends and both models here, including the case where on is the wrong choice. The JSON only records that auto was requested, so this is inferred from the matching numbers, and I haven't tested how llama-server resolves it. My dense model's 32k penalty happens silently under the benchmark default either way.
If you run dense models at depth, the backend is the limiting factor, not the flag. With flash attention on, ROCm does 200 t/s dense prompt processing at 32k where Vulkan does 85, at roughly equal generation speed. That is a 2.3x difference from choosing the right backend for that workload. For the MoE the ranking flips: Vulkan wins prompt processing (670 vs 567) and generation (46.7 vs 41.5). One box, two backends, and the right choice depends on the model class you run.
And if you stay on Vulkan with a dense model past the crossover, the right setting depends on your workload mix. Flash attention costs you prompt speed and buys you generation speed. On my numbers the break-even sits near three prompt tokens per generated token: reading long documents with short answers favors the flag off, long conversations on deep context favor it on.
The rest of the ladder
The levers from my published series, for completeness, all measured on this box:
- The build itself. Real, but smaller than folklore suggests once you hold the flags constant. The genuine build win in my series was the mid-June MTP fix, worth +7 to +16% decode with speculative decoding. Later "build gains" in my own post turned out to be the flash-attention default flipping. Pin the build, re-bench after updates, and compare only same-mode numbers.
- Speculative decoding (MTP). The 35B MoE ships a draft head; at draft depth 2 it takes decode from about 55.6 to 64.2 t/s at empty context, roughly +15%. The draft head's predictions get accepted less often as the context fills up (0.56 with an empty window, 0.51 at 64k), but I haven't measured a clean no-draft baseline at depth, so I won't put a number on the deep-context gain yet. Clearly worth it for chat-shaped work.
- Batch sizing. My production setting is 4096/512 from a June sweep I haven't published. Batch sweeps elsewhere land on different answers depending on model and quant, and the differences I've seen are single digits; worth one sweep on your own workload.
- VRAM carveout. Controls how much you can load and nothing about how fast it runs. A small carveout plus a large GTT window loads big models at unchanged speed; measured in June, and AMD's own Strix Halo tuning guide now recommends the same basic approach.
Three tips that didn't survive measurement
Negative results matter too:
- Disabling MMVQ (GGML_VK_DISABLE_MMVQ=1). MMVQ is the quantized matrix-vector kernel the Vulkan backend uses during token generation. Disabling it was suggested in the comments of the last post, with real gains measured on a Strix Point with a different MoE architecture. On my Qwen pair: +0.2 to +0.6% for the MoE, mixed and within run variance for the dense. The effect may depend on model architecture, quant, driver, or hardware, which makes it interesting upstream and irrelevant for my operating point.
- ROCBLAS_USE_HIPBLASLT=1. hipBLASLt is an alternative AMD matrix-math library that rocBLAS can hand its work to, and the guides report dramatic prompt-processing gains from it. I measure +1.2 to +3.4% on prompt processing and nothing on generation. (The variable asks rocBLAS to prefer hipBLASLt; it can still fall back per shape, so this measures the preference, not a guarantee that every operation used it.) Small, possibly config-dependent, and now it has numbers attached.
- The rocWMMA flash-attention build. rocWMMA lets the attention step use the GPU's matrix cores on ROCm, and older build guides for this hardware turn it on for a big speedup. I expected a win. On this box, this SDK (TheRock 7.13), at 32k it does the opposite: prompt processing collapses by 77 to 79% and generation drops 18 to 27% for both models, with the variant built from the same source commit and differing only in the logged rocWMMA option. The direction matches current guidance rather than contradicting it: the Strix Halo Wiki says the rocWMMA path should not be used on this hardware as of ROCm 7.0.2+, because it gets slower as context grows. My run puts a size on how much. On a gfx1151 box at this depth it is not a tuning tip, it is a trap.
What I didn't measure, and why
- Mesa 26.1. A commenter measured +11 to +27% prompt processing on a Strix Point going from Mesa 25.3.5 to 26.1.4, which would make it a top-tier lever. My Fedora 43 box is on 25.3.6, and there is currently no clean 26.x package for Fedora 43, only rolling git COPRs I won't put under a controlled benchmark. The test happens when a proper package lands; until then the gain is reported on neighboring hardware and remains unverified on mine.
- Power limits and system profiles. In one published LLM workload, raising the power limit from 85W to 120W added about 2%, which that author attributes to memory bandwidth becoming the limiting factor. I cite rather than re-measure these, and will fold them into a setup-pack update.
- IOMMU. My box runs amd_iommu=on with passthrough; the current guides recommend turning it off entirely, and I haven't isolated the difference on this machine. It's on the part-two list.
What I actually run now
What all of this changed in my own setup: flash attention on for the MoE on Vulkan, long-context dense work moved to the ROCm backend with flash attention on, and MTP at draft depth 2 for chat-shaped work. I don't re-bench after every update, nobody does. But when a model suddenly feels slower, the first thing I check now is the build number and the flash-attention mode in the bench JSON, before I blame the model or the hardware. When my numbers disagree with someone else's, those two columns are where I look first too.
The comment thread on the last post shaped half of this one, and the same offer stands: if you measure one of these levers on your own box, especially the ones that didn't reproduce for me, reply with the numbers and the two identity columns. The next update to this ranking will be built the same way this one was.
Discussion