Part two of the tuning series. The kernel parameter I had quietly dismissed turns out to be the biggest system-level lever on this box: +34 to +38% dense prompt processing. The rest of the week's list mostly measured out to nothing, one setting reproducibly crashes the GPU, and one verdict flipped when I re-checked it under the new configuration, which is exactly why I re-checked.
Part one ranked the tuning levers I could measure on a pinned llama.cpp build (9870) and found the biggest one in a flag: flash attention at depth, worth between +42% and -51% depending on model and backend. The comment thread and two current guides then handed me the list for part two. Same box, same build, same two models (Qwen3.6-35B-A3B MoE and Qwen3.6-27B dense, both Q5), and every number comes from the benchmark's own JSON, with the build and the flags logged next to it.
A reader's fix, and what testing it revealed
A reader forked llama.cpp with a 30-line patch for the Vulkan flash-attention path and suggested it might address the dense cliff from part one. I tested it the careful way: one build from the base commit, one from the base plus his patch, in separate directories, with the patched binary labeled as its own build. Nothing moved, on any cell, and the control run with flash attention off stayed put too.
The empty result still answers a question. His patch rescues a case where the fast Coopmat1 attention path falls back to a slow scalar shader when it runs out of shared memory. On my box that fallback never triggers, so my dense cliff is the Coopmat1 path itself scaling badly at deep context, and no fallback repair will fix it. His fix still makes sense for the case he actually hit, a collapse at empty context under Mesa 26.1 that I cannot reproduce on 25.3.6 (tracked in issue #25207), and my A/B data is attached there. So we were looking at two different problems that happened to look alike.
The big one: turning the IOMMU off
The IOMMU is the unit that translates device memory addresses, mainly so VMs and passthrough can be isolated safely. Both current guides for this hardware recommend turning it off for inference. My box ran the other common setting, translation on with passthrough mode, and part one listed the difference as not measured. Now it is measured: one controlled comparison with a reboot in the middle, same session, same binary, and the active configuration recorded at every boot.
| Cell | on + passthrough | IOMMU off | Change |
|---|---|---|---|
| MoE prompt processing, empty | 1007 t/s | 1074 t/s | +6.7% |
| MoE prompt processing, 32k | 658 t/s | 683 t/s | +3.8% |
| Dense prompt processing, empty | 255 t/s | 351 t/s | +37.5% |
| Dense prompt processing, 32k | 78 t/s | 105 t/s | +34.1% |
| Generation, all cells | +0.6 to +1.4% |
The guides were right and I was the outlier. For dense prompt processing this is the largest system-level lever I have measured on this machine, and it barely touches generation, which fits the mechanism, because prompt processing is the phase that pushes the most memory traffic through that translation layer.
Three footnotes before you copy this. First, this was measured A-then-B across one reboot, not a counterbalanced A-B-A; the deep dense cell drifts day to day, and against the harshest comparison the gain is still +25.6%, so the size class holds even if the exact digit wobbles. Second, a kernel wrinkle: on kernel 7.0.11, amd_iommu=on is not even a recognized option value. The IOMMU was actually active through firmware detection, and iommu=pt only supplied the passthrough policy. In other words, my old setting did less than its name suggested, and it still cost a third of dense prompt processing. Third, scope: this is for a dedicated inference box. If your machine runs VMs, VFIO, or anything that needs device isolation, leave the IOMMU alone.
I switched my box permanently, re-measured a fresh reference under the new configuration, and everything below is verified against it.
The cliff survives the configuration change
The obvious follow-up question: was part one's dense flash-attention cliff just an IOMMU artifact? It was not. Under the new configuration the matrix looks the same in relative terms: the MoE gains 41.9% prompt processing at 32k with flash attention on (41.3% before), and the dense model loses 46.6% (42.9% before). Every absolute number rose, but the ratios stayed where they were. So part one's rules carry over unchanged: flash attention on for the MoE, and for dense models at depth either switch to ROCm or weigh your workload mix.
The rest of the list, measured
- mmap on versus off. An external guide reports around +22% prompt processing from disabling mmap. I ran the comparison on a warm box, both models, both depths, and everything lands within about one percent, apart from one volatile deep-dense cell at -2.3%. No throughput reason to touch the flag on a warm box. Cold starts and memory pressure are a different question I have not measured yet.
- GGML_VK_PREFER_HOST_MEMORY plus RADV_PERFTEST=nogttspill. A setup guide for this hardware recommends the pair, with the sensible caveat to benchmark rather than copy. Under my old configuration it measured as harmless, within noise everywhere. Under the new one it is still neutral for the MoE and for shallow dense work, but it costs 6.0% on deep dense prompt processing. The only thing that changed between those two verdicts is a kernel parameter. I left the pair disabled.
- ubatch. Raising the micro-batch from 512 does nothing for the MoE anywhere. For the dense model, 1024 is a small consistent win at depth: +2.1% under the old configuration, +2.8% under the new one. I keep 512 as the default and use 1024 in a dedicated deep-context dense preset.
- ubatch 2048 with the dense model crashes the GPU. Reproducibly: two out of two attempts end in vk::DeviceLostError, the kernel logs a compute ring timeout, writes a device coredump, and recovers through a queue reset. The MoE runs fine at the same setting. Reported upstream with both stack traces and the kernel evidence, in an existing issue that already tracks this crash class on this hardware; my contribution is a deterministic llama-bench reproduction. A tuning value that can take down the device is worth knowing about either way.
Why every verdict got re-measured
Changing a kernel parameter changes the machine you measured on, and a recommendation is only worth copying if it holds on the machine readers will actually configure. So after the switch I re-ran every lever above that could plausibly interact with the change. That extra half hour is how I found out that the host-memory variables, harmless before, now cost six percent. If you take one habit away from this series, take this one: after changing your base configuration, re-check your old conclusions before you repeat them.
What I run now
The settings this series has produced so far, as of this post:
- Kernel command line: amd_iommu=off. Worth +34 to +38% dense prompt processing on this box. For dedicated inference machines only; keep the IOMMU if you run VMs or passthrough.
- Flash attention: on for the MoE everywhere. For dense models at depth, either ROCm with the flag on, or Vulkan with it off past the crossover (part one's rules).
- Micro-batch: 512 as default, 1024 as a preset for deep-context dense work, 2048 never. The last one reproducibly crashes the device with my dense model; issue linked above.
- Host-memory environment variables: none. Harmless under my old configuration, a 6% penalty on deep dense prompt processing under the new one.
- mmap: left at the tool's default. Measurably irrelevant on a warm box. The full configuration behind these, the kernel parameters, memory windows, service layout, and the warnings that come with them, lives in the Setup Pack. I keep it updated as this series moves, and this post's settings are already in it.
The part-three list already has entries: the closed MMQ kernel patch under the new configuration, mmap under cold-start and memory pressure, Mesa 26.x once Fedora ships a controlled path, with issue #25207 as the reason to test flash attention first when it lands, and a reader fork claiming large quantized-KV-cache gains on this hardware, which I'll verify on a 128GB box at depths beyond the author's reach.
The comment threads built half of this post, and the offer from part one stands: measure any of these on your own box and reply with the numbers, with build_commit and the flash-attention mode from the benchmark JSON attached. The next part will be built the same way.
Discussion