Speculative decoding on a Radeon GPU with llama.cpp

What a draft does

The draft is a smaller same-family GGUF that runs ahead of the target. llama.cpp proposes draft tokens, then the target checks them in one forward pass. Accepted tokens are free decode; rejected tokens fall back to the target. Acceptance is a measured rate on your box, not a number we invent. A bad draft wastes VRAM and can lose. Size both files plus KV before you launch.

House context

the house default is Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL, llama.cpp Vulkan, one R9700, 262144 context, f16 KV. That launch has no -md. Hermes’s Flash runs on Strata UD-Q6_K_XL with no MTP. Do not turn on a 2-bit / Q2-class MTP head to “make Flash faster.”

Daily-driver recipe: qwen36-35b.llamacpp-vulkan.262k · Flash: /models/flash · Strata (no MTP): /backends/strata.

llama.cpp flags

Cited from the llama.cpp server README. Vulkan first; HIP after rocminfo lists the gfx — Vulkan vs ROCm.

• -md / --model-draft / --spec-draft-model

  Path to the draft GGUF. Official llama.cpp server README. Unused unless you pass a file. Draft stays Q4 and up on this site.

  llama.cpp server README — --spec-draft-model, -md, --model-draft

• --spec-draft-n-max

  How many tokens the draft may propose (default 3 on the README). Official docs say --draft-max / --draft-n were removed; use --spec-draft-n-max.

  llama.cpp server README — --spec-draft-n-max (replaces removed --draft-max)

• --spec-draft-n-min

  Minimum draft tokens to use (default 0). Official docs say --draft-min was removed; use --spec-draft-n-min.

  llama.cpp server README — --spec-draft-n-min (replaces removed --draft-min)

• -ngld / --n-gpu-layers-draft / --spec-draft-ngl

  How many draft layers sit in VRAM (exact count, auto, or all). Same idea as -ngl, for the draft. Leaving layers on the host is RAM offload — print both shopper RAM lines.

  llama.cpp server README — --spec-draft-ngl, -ngld, --n-gpu-layers-draft

• -devd / --device-draft / --spec-draft-device

  Comma-separated devices for the draft (none = do not offload; default follows --device). We have not recorded a dual-R9700 pin. Do not invent one.

  llama.cpp server README — --spec-draft-device, -devd, --device-draft

# Shape only — draft decode is not measured on the local desktop. Draft stays Q4+.
# Pick a real Q4+ same-family draft from the model card. Do not invent a filename.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -md ~/models/draft-Q4_K_M.gguf \
  --spec-draft-n-max 3 \
  --spec-draft-n-min 0 \
  -ngl 99 \
  -ngld 99 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Shape only. The other knobs (-ngl, threads, batch, -fa, KV, split) live on /tuning.

Measured 35B — no draft

Official Qwen3.6-35B-A3B llama.cpp Vulkan cells from existing run cards. These launches have no -md. Draft-on cells stay unmeasured. UD-Q3_K_XL is a labeled estimate. Full matrix: /benchmarks.

VRAM fit — main + leftover for a draft

Calculator estimate for Qwen3.6-35B-A3B on llama.cpp, f16 KV. Leftover is usable VRAM minus target total (weights + KV + overhead). It is not a measured draft file. A dense same-family 27B Q4_K_M is 15.7 GB of weights on the calculator — usually larger than leftover on one 32 GB card after the house 35B. Pick a real smaller Q4+ draft from the card, then size it. UD-Q3_K_XL is a labeled estimate.

One R9700 · 32K

One R9700 · 262K

KV grows. Leftover shrinks. Size the window on /long-context.

When it helps — and when it does not

Can help

It can help when the draft agrees often: code, boilerplate, and repetitive prose. Same-family drafts only. Both files must fit, or the draft is the thing you are willing to spill. Measure acceptance with the Stack Lab speculative decoding A/B and an amdai-bench card.

Often does not

It often does not help on MoE A3B-class models that already decode fast on one R9700 — the house 35B is that class. Low acceptance burns draft VRAM for no tok/s. Dual-card split of a file that already fits one card is often slower at short context (measured split tax on /compare/one-vs-two-gpu). We do not invent a “draft on GPU 1, target on GPU 2” pin. Flash stays the Strata no-MTP path, not a draft-model try.

One vs two R9700s

Two cards are 64 GB of VRAM (2 × 32 GB). Leftover below is the calculator on a split 35B — not a measured draft-on-GPU-2 pin. Official --device-draft exists; we have not recorded one. Split tax: one vs two GPU · dual / multi-GPU.

System RAM

When -ngl or -ngld leaves layers on the host, the spill is system RAM. Shopper lines stay Minimum system RAM needed and RAM needed at full context. House 35B UD-Q5_K_XL on one 32 GB R9700 does not offload. The 16 GB example below is house 35B Q4_K_M on an RX 7800 XT at 32K selected — estimate unless a measured card overlays the kit. A draft that does not fit leftover VRAM adds more host weight.

FAQ

• Did AMDai measure a draft speed-up on the R9700?

  No. Draft decode and acceptance stay not measured yet on the local desktop — run the Stack Lab speculative decoding A/B. Baseline 35B decode (no draft) is the official local desktop llama.cpp Vulkan sweep in the table. Open the Stack Lab planner, then record an amdai-bench card.

  Stack Lab · amdai-bench · Benchmarks

• Can the draft model be Q3?

  Draft / speculative models stay Q4 and up. UD-Q3_K_XL may appear as a labeled estimate for the main 35B file (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). A Q3 draft is not a house pick. Q1/Q2 drafts stay off, including MTP heads.

  Quants · Q6 / Q5 / Q4 + Q3 estimate · Speed up llama.cpp

• Will a draft speed up the house 35B-A3B?

  Maybe, maybe not. That file already decodes fast on one R9700 because only a 3B-class slice is active. A draft still costs VRAM and KV. We will not invent an accept-rate. Measure it on your box.

  MoE on Radeon · R9700 measured page · Calculator

Sources

• llama.cpp server README (draft flags)
• llama.cpp repository
• llama.cpp build.md
• AMDai — speed up llama.cpp
• AMDai — backends
• AMDai — llama.cpp
• AMDai — Vulkan vs ROCm
• AMDai — Strata (no MTP)
• AMDai — Flash model page
• AMDai — Stack Lab
• AMDai — amdai-bench