One llama.cpp Update Made My MoE Model 38% Faster and My Dense Model 45% Slower

Between June 14 and July 3, llama.cpp on my machine went from build 9637 to build 9870. Same box, same models, same flags; only the build changed. If you quote tokens per second without a build number, you're quoting a moving target.

I run a Strix Halo box (128GB unified memory) as my daily local-inference machine, and every few weeks I re-benchmark it against the current llama.cpp master before I move my production services to a new build. This post is the latest of those re-runs: three builds side by side on the same hardware. It also covers two things that broke silently along the way, which ended up making the point better than the numbers do.

The three-way numbers

The three builds, all measured on the same box in the same session: build 9637 (June 14), build 9791 (June 25), and build 9870 (July 3, current master at time of measurement). Vulkan backend on RADV (Mesa 25.3.6), Fedora 43. The two models are the ones I actually run daily: Qwen3.6-35B-A3B, a mixture-of-experts model, and Qwen3.6-27B, a dense model, both in Q5 quants.

Two terms, briefly, for anyone landing cold: prompt processing is how fast the model reads your input, generation is how fast it writes the answer. The @32k rows are measured with 32,000 tokens of context already in place, which is where long documents and coding agents live.

The same three weeks, with speculative decoding on (the 35B has a draft head that predicts tokens ahead; when its guesses get accepted, decode speeds up):

Reading it top to bottom: the big movements happened in the June 25 build and they persisted. That build brought a unified-memory fix that lifted MoE prompt processing at depth by more than a third, and it brought a dense regression at the same depth that's still there two builds later. Correction, July 10: I originally pinned the lift on the unified-memory work that landed in that window. A reader pushed back with good evidence: the PR author says it's off the pp/tg hot path, and the Vulkan maintainer's own Strix Halo runs of that stack show low single digits. So the honest status is that something in those roughly 150 commits moved these numbers, and I'm bisecting to find out which. I'll update here when I have the answer. Since then the changes are small: a touch faster on fresh context, a touch slower at 64k, and the speculative acceptance rate has drifted down from 0.57 to 0.51. The swing is also backend-specific: over the same three weeks, the ROCm build of the same models moved by low single digits in either direction, so the wild ride above is a Vulkan story.

Now put yourself in two different chairs. If I ran dense models at deep context, "just update llama.cpp" would have cut my prompt speed nearly in half, and nothing in the update notes would have warned me. Since I mostly run the MoE, the same update was the best free speedup of my quarter. If you run dense models at deep context on this hardware, staying on the older build for that work is a reasonable move for now. And I haven't seen this regression filed upstream, so if you can reproduce it on other RDNA 3.5 machines, I'd like to hear about it.

One note on why I trust these deltas: each re-run measures the older builds live again instead of reusing stored numbers. The re-measured cells landed within noise of the previous day's run (three decode cells came back 55.15, 62.93 and 48.48 t/s against 55.11, 62.12 and 47.89 the day before). That run-to-run gap is small, so the big swings in the table really did come from the builds.

The build broke my measuring stick too

When I first ran the new build through the bench harness, every one of its cells came back as a server failure, all six of them. The two older builds sailed through the identical script minutes earlier.

The server was fine. It had loaded the model in 4.3 seconds and answered its health endpoint with a clean 200. What had actually changed: build 9870 prints a different startup line. My harness waits for "server is listening" in the log before it starts measuring, and the new build says "llama_server: listening on http://..." instead. So the script waited 300 seconds for a line that never comes, timed out, and wrote FAIL. Six times in a row.

The fix was one grep pattern. What stays with me is the shape of the failure: a routine build update moved my numbers and, on the side, silently broke the instrument that measures them, in a way that looks exactly like "the new build is broken." If I'd trusted the harness output instead of reading the server log, this post would be reporting a regression that doesn't exist.

Workarounds expire without notice

Last winter I needed an override flag for a 235B model whose native context window is only 40K tokens. llama.cpp hard-caps any request above a model's native limit, so I overrode the advertised limit to get my larger context through. It worked and I noted it down as "required".

While writing this post I went into my production services to finally remove that override. There was nothing to remove. I had only ever applied it ad hoc to that one model, the model itself left my lineup months ago, and my memory of my own configuration was two builds out of date.

I also re-tested the edge case itself on the current build, to check whether the override would even still be needed: a model with a 32K native window, asked for 64K of context. The hard cap is still there. Build 9870 logs "the slot context (65536) exceeds the training context of the model (32768) - capping" and clamps the slot to 32K, exactly like last winter. So my override wasn't obsolete because llama.cpp changed its behavior. It was obsolete because nothing I currently run asks for more context than its model natively supports.

The decay here wasn't in the code at all. The behavior I worked around is still in place, the model that triggered it is gone, and my mental model of my own setup had quietly fallen behind both.

You may not even know which build you're on

This applies beyond people who compile their own. A lot of local-AI tooling ships its own llama.cpp underneath. Lemonade, AMD's local inference server for this hardware, bundles its own build, and when I compared it against my self-compiled llama.cpp earlier this summer, the differences traced back to the bundled binary, not to anything the wrapper adds on top. Two people can both say "I run llama.cpp on Strix Halo" and be weeks of builds apart without either of them knowing it. When your numbers disagree with a stranger's on the same hardware, the missing column in the comparison is usually the build.

What I actually do now

The routine that has survived contact with all of the above:

• Pin the build, and write it next to every number you save or post. A tokens-per-second figure without a build number has a shelf life of a few weeks.
• After updating, re-bench at your own operating point: your models, your context depth, your backend. The MoE and dense rows above moved in opposite directions in the same build, so a global "the update makes it faster" doesn't exist.
• Keep the previous binary around. Being able to A/B yesterday's build against today's turns "it feels slower" into a number in ten minutes.
• Re-test your workarounds once in a while. They expire without notice, and a stale one either hides current behavior or, like mine, quietly stops existing.

Doing this by hand every few weeks is exactly the kind of chore that stops happening the week you get busy. So I've started building a small tool for my own setup: it watches for new llama.cpp builds, re-runs my bench matrix, and shows me the delta before I move production onto the new build. A changelog for my own numbers, basically. It's not ready to share. But if you'd run something like that on your box, hit reply and tell me which backend you're on. If enough people want it, I'll clean it up and put it out.