Seven current models on the bench, five speculative paths that could actually be measured, one build, one evening. Speculative decoding runs from tripling a model's speed to making it five times slower, the sweet spots spread from depth 3 to depth 5, and the gain depends as much on your workload as on your hardware. Also: DeepSeek V4's sidecar drafter works on Vulkan, which I have not seen measured anywhere else yet.
Speculative decoding is the trick where a small, cheap predictor drafts a few tokens ahead and the big model only verifies them. A verified draft token costs a fraction of a generated one, so if enough drafts survive verification, generation gets faster. It is designed to preserve the target model's output; this run measured speed and acceptance, not output equivalence, and there is an open llama.cpp report about greedy-decoding deviations with quantized targets, so I am not claiming the outputs are bit-identical. llama.cpp calls the built-in variant MTP, multi-token prediction. Back in May I swept it properly, 260 runs on Qwen3.6, found the sweet spots, set my production preset accordingly, and have not re-measured it since.
Two things broke that assumption this month. Qwen 3.8 27B arrived and gained far more from MTP than my daily MoE ever did. And a commenter in the day-one thread posted a depth-5 result that beat my depth-2 recommendation by a wide margin. My cold measurement had actually put depth 5 ahead, but with a spread wide enough that I wrote it off as too noisy to recommend; his warm one said it was the best tested setting on the model. He was right, and the proper answer was to inventory everything and sweep every viable speculative path: every current-generation model on this box, every draft depth from 1 to 5, warm, seeded, on the production build.
Three mechanisms, not one
"MTP" on this fleet is three different mechanisms, and knowing which one your model uses decides which flags you need and what to download.
Integrated draft tensors. The Qwen family and Ling carry a small prediction head inside the main GGUF, visible in the metadata as nextn_predict_layers with matching tensors. Nothing extra to download; --spec-type draft-mtp turns it on, and --spec-draft-n-max sets the depth.
A separate draft model. Gemma 4 has no draft tensors in the main file. Instead there is a second, small GGUF that runs alongside as the drafter: same --spec-type draft-mtp, plus --spec-draft-model pointing at the draft file. Two files, two model loads, same idea.
A sidecar drafter. DeepSeek V4 Flash drafts through DSpark, a block drafter that works from the target model's hidden states and ships as its own GGUF, wired up with --spec-type draft-dspark and --spec-draft-model. llama.cpp gained both paths for V4 Flash in the same pull request, merged on August 2: integrated MTP and the DSpark sidecar. On the maintainer's own benchmark DSpark came out ahead, 1.83x against 1.53x. My quantized copy settles which path I could test anyway: it advertises the integrated mechanism in its metadata while containing zero of the tensors, which cost me a confused half hour before I concluded the declaration is stale, or at least useless as evidence of an integrated head. The 10.5 GiB sidecar from the official Unsloth repo is the piece that actually drafts. The numbers I have seen for it so far come from CUDA cards: one writeup cites 1.83x from the pull request's own benchmarks, plus a user on the same thread reporting 39.95 to 79.93 t/s on hardware the writeup does not name. I have not seen DSpark numbers for Vulkan or this hardware class anywhere yet, so the rows below may be the first.
Not every model has a drafting path. The capability scan covered all eleven models on this box, the seven current candidates plus four older ones I keep around. It found no integrated MTP head and no compatible external draft for Qwen3.5-122B, Hy3, Laguna, the older Coder-30B, and Muse-Glimmer-30B, which I pulled fresh in mid-August because it is all over my feeds right now: it runs fine (9.8 t/s decode at this size), but there was nothing compatible to test in this MTP-focused sweep. And my Gemma 4 31B sat out for an honest reason: the available draft file comes from a different finetune than my local copy, and a drafter tuned against different weights would make the acceptance and speed comparison meaningless. For a clean benchmark, draft and target have to match.
The results
All numbers are from one evening on the production build: 10362, commit 5d01b978, built from the Nathanw1014 strix-halo-vulkan branch plus the BailingMoE3 patch and a local loader fix, which is what Ling needs to load at all. That patch has since been merged upstream, on 17 August. Everything ran on Vulkan through llama-server, with greedy sampling, a fixed seed, and prompt caching off. Each server start got one unmeasured warm-up request before that configuration's measured ones. Baseline is the same model with speculative decoding off. Every gain in this post is decode; drafting does not target prompt processing, and I make no prefill claims here, because the server-reported prefill numbers under drafting were not uniformly flat and I do not fully trust them (more on that below).
All rows use f16 KV cache; weight quants differ per model and are in the table. Depth 5 is the upper bound of this sweep, not proof of a peak beyond it. For DSpark it is also the drafter's trained block size, and llama.cpp caps it there.
| Model | Weights | Mechanism | Baseline (t/s) | Best depth | Code decode (t/s) | Code gain | Prose gain | Acceptance |
|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 27B | UD-Q5_K_XL | integrated | 10.5 | 5 | 31.6 | +201% | +67% | 75% |
| DeepSeek V4 Flash | IQ2_XXS | DSpark sidecar | 17.8 | 5 | 35.1 | +97% | +4% | 74% |
| Gemma 4 26B A4B | Q4_K_M | separate draft | 62.3 | 4 | 111.9 | +80% | +24% | 75% |
| Qwen 3.6 35B-A3B | UD-Q5_K_XL | integrated | 56.7 | 3 | 92.0 | +62% | +14% | 85% |
| Ling 3.0 Flash | Q4_K_M | integrated | 44.9 | 4 (least bad) | 9.0 | -80% | -82% (1 valid run, unstable) | 48% |
Three things in that table changed how I run this box.
The dense model tripled. Qwen 3.8 27B goes from 10.5 to 31.6 t/s on code. My reading, not something this run proves: a dense 27B reads every weight for every token, so a verified draft token saves an expensive full pass, while the draft head itself is tiny. My 35B MoE gains the least of the winners for the mirror-image reason: its full pass only activates 3B parameters and was cheap to begin with. Across the whole table, drafting saves roughly in proportion to what a full pass costs.
DSpark turns DeepSeek V4 into a usable coding model. 17.8 to 35.1 t/s on a 284B-parameter model, on a mini PC. The sidecar loaded on the first try, drafted through all five depths without a single error, and the 10.5 GiB it occupies next to an 80.8 GiB model still fits this box with room for context.
And Ling got five times slower, then hung twice. Ling uses the same integrated mechanism as Qwen, the tensors are in the file, and the initial probe request completed fine. Then the sweep ran: 45 t/s without drafting, 9 with it at the least-bad depth. Acceptance collapses to 48 percent, and the box hung twice for real in different cells, once at depth 5 after about a hundred tokens and once during the prose run, each hang hard enough to need a kill, each followed by a connection error on the next attempt. The depth curve is also upside down compared to every other model: depth 1 is the worst at 5.4 t/s and deeper drafting makes it less bad, which smells like a fixed per-draft-call cost that dwarfs everything else on this path. The architecture explanation fits the pattern in the rest of the table: Ling's linear-attention layers may make its full pass cheap enough that there is little to save, leaving the drafting overhead to dominate. But a five-times slowdown plus hangs goes beyond expensive drafting: something in this path is broken on this stack. I have not filed it yet: when I ran this, Ling only loaded through an unmerged upstream pull request, and that landed on 17 August, so it is worth re-checking against a plain master build before anyone files anything. Do not run Ling with MTP today.
No single setting fits every model
The sweeps are the part I was actually after. Acceptance falls with depth on every model I swept: on Qwen 3.8 from 97 percent at depth 1 to 75 percent at depth 5. What differs is how deep the gain keeps growing before falling acceptance eats it. The dense 27B keeps gaining through the highest tested depth, 5. The 35B MoE peaks at 3 and declines after. Gemma peaks at 4. If I had run the whole fleet at my old production setting of 2, depth 2 would have sat 22 to 24 percent below the best tested setting on the two biggest winners and about 5 percent below on the other two, and if I had stuck with my own reading of the cold 14 August measurement, I would still be recommending depth 2 for a model whose best tested setting is 5. The scatter I saw on 14 August did not reappear under the warm, seeded protocol. I cannot pin that on the warm-up alone: the 14 August run was build 10283 at 16k context, this one is 10362 at 128k, and while the baseline matches to within a hundredth of a token per second, depth 2 went from 19.3 to 24.1 t/s between them. Something in the drafting path changed with the build as well, and I did not run a controlled comparison to separate the two. One axis this sweep deliberately held fixed: the draft probability threshold, spec-draft-p-min, which the upstream speculative-decoding discussion treats as a lever of its own. Depth times threshold is a two-dimensional sweep and this was one evening, so the threshold dimension is still on the list.
The workload column matters just as much. Every winner gains far less on prose than on code: the drafter guesses code continuations much better than free text, and DeepSeek is the extreme case, +97 percent on code and +4 on prose. If your box writes code, speculative decoding is the biggest single lever I have measured on this hardware. If it writes essays, expect much less, and for DeepSeek specifically, skip it.
One measurement note for anyone repeating this: the server's reported prompt-processing rate on Qwen 3.8 jumps implausibly above depth 2 in my logs, and on the 35B it reads 8 to 11 percent below baseline under drafting. Neither affected decode numbers or the depth choice, and the raw JSONs keep all of it visible, but treat server-side prefill readings under speculative decoding with suspicion.
What changed on my box
The presets are switched already: the always-on 35B runs depth 3 now (worth about 5.6 percent of code decode over its old depth 2; I did not measure prose at 2 against 3, so that side is an assumption), Qwen 3.8 runs depth 5, Gemma runs depth 4, DeepSeek V4 has the DSpark sidecar at depth 5, and Ling stays without MTP. One caveat on that: my production Qwen 3.8 is a Q6 finetune of the same base, not the exact Q5 file I swept, so its depth 5 was carried over rather than measured on that artifact. The full rows, with build hash, mechanism, depth curves and raw JSON paths, are on the model bench page, and the maintained settings stay in the Setup Pack.
For the KV-cache side of this fleet, including the two models where quantizing the cache backfires, see the previous post. The pattern there matches the pattern here: on this box, the newest attention designs are where the standard advice has now failed twice.
If you run any of these models on your own hardware, two numbers would genuinely help: your sweet-spot depth for the same model, and your acceptance rate on your real workload rather than a benchmark prompt. Post them with the build commit and the mechanism you used. My May sweep was right in May. Three months of build changes moved the one sweet spot I can compare directly, the 35B went from depth 2 to 3, and the rest of this fleet was never in the May series at all. One commenter's number was enough to catch my stale depth-2 assumption. When a model suddenly feels slower or faster than you remember, check the draft depth against the build you are actually running, not against the number you wrote down three months ago, mine included.
Discussion