Speed up llama.cpp on Radeon

Measured 35B on one R9700

Official llama.cpp Vulkan cells from existing run cards. Empty stays not measured yet. UD-Q3_K_XL is a labeled estimate. Full matrix: /benchmarks.

House launch

Start here, then change one flag. Exact recipe: qwen36-35b.llamacpp-vulkan.262k.

# House daily-driver shape — /recipes/qwen36-35b.llamacpp-vulkan.262k
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

GPU layers (-ngl)

-ngl / --n-gpu-layers

What it does

-ngl is how many transformer layers llama.cpp tries to keep on the GPU. -ngl 99 is the house “offload what fits” pin: leftover layers spill to system RAM. That is the same meaning already on the Flash RAM offload page and the Offload glossary entry.

When it helps

Single card: start at -ngl 99 so the file sits in VRAM when it can. Dual card: still start at -ngl 99, then read --split-mode on Dual / multi-GPU — a second card is a split, not a magic -ngl. Partial -ngl is the 16 GB / 12 GB move when Q4 still spills.

Tradeoff

More layers on GPU usually means faster decode and less DDR5 traffic. Too many layers on a small card OOMs or silently spills. Spill is system RAM — print both RAM lines below. Quality of the GGUF does not change; speed does. Partial -ngl tok/s is not measured yet on the local desktop except where a card already names --fit-target.

# Partial offload shape only — not a measured local desktop pin.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 20 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

House daily driver on one R9700 uses -ngl 99 and 0% RAM offload. Measure a -ngl ladder on your box with Stack Lab planner and an amdai-bench card.

Stack Lab A/B: partial offload / -ngl ladder. Measure it on your box via /stack-lab planner and /amdai-bench.

Flash RAM offload · Offload · Calculator · 35B recipe

CPU threads (--threads)

--threads / -t

What it does

--threads sets how many CPU threads llama.cpp uses for work that still runs on the host — leftover layers, some prefill helpers, and tokenization. llama.cpp documents the flag in --help and the server README. We have not measured a thread sweep on the local desktop.

When it helps

Single card with the file in VRAM: extra host threads rarely move decode. They matter when -ngl leaves layers on DDR5, or during prefill. Dual card: the host still owns the leftover; do not invent a per-GPU thread pin.

Tradeoff

Too few threads can stall prefill when RAM offload is in play. Too many fight the GPU for memory bandwidth. Speed vs core count is not measured yet. Quality is unchanged.

# Thread count is not measured on the local desktop. Copy nproc and measure.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -t $(nproc) \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Measure CPU threads on your box. Empty cells stay not measured yet.

Stack Lab A/B: CPU threads. Measure it on your box via /stack-lab planner and /amdai-bench.

llama.cpp · Flash RAM offload · amdai-bench

Batch threads (--threads-batch)

--threads-batch

What it does

--threads-batch is the host thread count for prompt / batch (prefill) work. llama.cpp lists it next to --threads. Prefill and decode are different jobs — the Prefill vs decode glossary entry already says so. We have not measured a --threads-batch sweep on the local desktop.

When it helps

Single card: try it when prefill feels stuck and rocm-smi / amd-smi shows the GPU idle while the GGUF is on NVMe — check the disk first (glossary). Dual card: same host flag; do not invent a per-GPU batch-thread split.

Tradeoff

More batch threads can help prefill when layers sit in RAM. They can also steal bandwidth from decode. Prefill tok/s vs --threads-batch is not measured yet. Quality is unchanged.

# Prefill thread pin is not measured on the local desktop.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -t $(nproc) \
  --threads-batch $(nproc) \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Measure prefill tuning on your box. Do not copy a prefill figure we did not record.

Stack Lab A/B: prefill tuning. Measure it on your box via /stack-lab planner and /amdai-bench.

Prefill · Prefill vs decode · Long context · llama.cpp

Batch sizes (-b / -ub)

-b / --batch-size and -ub / --ubatch-size

What it does

-b is the logical batch (how much prompt llama.cpp plans for). -ub is the physical micro-batch that actually hits the GPU. llama.cpp documents both. Overnight 35B llama-bench cards record their own bench line — they are not a llama-server -b pin we will invent.

When it helps

Single card: raise batch only after the file fits. Dual card: batch still lives on the process; split-mode decides which GPU sees the layers. Long prompts at 128K–262K are a KV problem first — read Long context. Never set -ub 65–256 on Qwen3.5 or Qwen3.6; keep the default 512 unless an allocate forces a cut below 65.

Tradeoff

Larger batch can raise prefill and VRAM. Smaller batch can fit a tight card and slow prefill. Batch vs tok/s is not measured yet on the local desktop llama-server. Quality is unchanged if the GGUF stays the same.

# Example values from llama.cpp --help — not a local desktop pin.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -b 2048 \
  -ub 512 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Copy from llama.cpp --help, then measure. Empty prefill cells stay not measured yet.

Stack Lab A/B: prefill-ab/1 (amdai-bench ab-prefill). Measure it on your box via /stack-lab planner and /amdai-bench. Shopper page: /prefill.

Prefill · Long context · Benchmarks · llama.cpp

Flash attention (-fa)

-fa / --flash-attn

What it does

Flash attention is a fused attention kernel. Overnight 35B llama-bench cards recorded -fa 1. The Long context page already cites that line and warns that llama-server flags are not on that card — copy -fa only when your build documents it.

When it helps

Single card: keep -fa 1 when the measured 35B recipe’s build supports it. Dual card: the split cards used the same bench family; do not invent a per-GPU -fa off switch. If your Vulkan or HIP build rejects -fa, leave it off and record the card.

Tradeoff

When it works, flash attention usually cuts attention VRAM and can help long context. Quality and tok/s for -fa 0 vs -fa 1 on llama-server are not measured yet. Do not invent a 262K delta.

# -fa 1 is on the overnight llama-bench line. Server pin is not recorded.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -fa 1 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Read Long context before you treat -fa as a server pin. Measure if your build differs.

Stack Lab Flash Attention A/B and Prefill tuning A/B. Measure on your box via /stack-lab planner and /amdai-bench. Shopper page: /flash-attention.

Flash Attention · Long context · Run cards · llama.cpp

KV cache type (f16 vs q8_0)

--cache-type-k / --cache-type-v

What it does

KV cache is the per-token memory that grows with context. House default is f16 — overnight 35B cards recorded -ctk f16 -ctv f16. q8_0 is the smaller KV type the calculator already lists. Quality and tok/s for q8_0 KV are not measured yet (same sentence as Long context).

When it helps

Single card: stay on f16 for the house 35B daily driver. Switch to q8_0 only when 32K–262K KV is the thing that spills — size it on the calculator first. Dual card: KV still grows; split does not delete it. Read Long context for 128K–262K.

Tradeoff

q8_0 KV uses less VRAM and can raise the context you fit. It is a quality and speed unknown on the local desktop. f16 is the measured house default. When KV or weights spill, both RAM lines below apply.

# q8_0 KV is listed on the calculator — not measured on the local desktop.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080

House cards stay f16. q8_0 quality and decode stay not measured yet.

Stack Lab A/B: prefill / context. Measure q8_0 KV on your box via /stack-lab planner and /amdai-bench.

KV cache · Long context · KV types · Calculator

Multi-GPU (--split-mode / --tensor-split)

--split-mode / -sm and --tensor-split / -ts

What it does

--split-mode layer (short -sm layer) is the documented llama.cpp multi-GPU default and the overnight 35B split on the local desktop. --split-mode row and --tensor-split are documented and not measured on the local desktop. Dual / multi-GPU already walks those flags — this page does not retell that guide.

When it helps

Single card: skip these flags. Dual card: start at --split-mode layer only after you decide the file cannot stay on one GPU. One R9700 usually decodes the house 35B faster than the split — that comparison lives on One vs two GPUs.

Tradeoff

A split can load a file that one card cannot. It can also slow decode versus one card that already fits. --tensor-split ratios are not a house pin. Quality of the GGUF is unchanged. Speed of row split is not measured yet.

# Overnight 35B split used -sm layer. Do not invent a -ts pair.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  --split-mode layer \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Read Dual / multi-GPU and One vs two GPUs before you split a file that already fits one card.

Stack Lab A/B: multi-GPU. Measure it on your box via /stack-lab planner and /amdai-bench.

Dual / multi-GPU · One vs split · 35B on two cards

Speculative decoding / draft models

-md / --model-draft (server README)

What it does

Speculative decoding runs a smaller draft model ahead of the target and accepts matching tokens. llama.cpp documents draft models on the server README. AMDai has not measured a draft pair on the local desktop. Draft GGUFs stay Q4 and up — Q3 draft is a labeled estimate and not a house pick. Q1/Q2 drafts stay off, including MTP heads.

When it helps

Single card: only if both files fit, or the draft is the thing you are willing to spill. Dual card: do not invent a “draft on GPU 1, target on GPU 2” pin we did not record. Same-family drafts only. The house ban list still applies.

Tradeoff

A good draft can raise decode. A bad draft wastes VRAM and can lose. Draft tok/s is not measured yet. Quality of accepted tokens depends on the draft — we do not invent an accept-rate. Size both files on the calculator.

# Shape only — draft decode is not measured on the local desktop. Draft stays Q4+.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -md ~/models/draft-Q4_K_M.gguf \
  -ngl 99 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Pick a real Q4+ draft of the same family from the model card. Do not invent a filename or a speed. Measure on your box.

Stack Lab A/B: speculative decoding. Measure it on your box via /stack-lab planner and /amdai-bench.

Speculative decoding · Quants · Calculator · llama.cpp · amdai-bench

GPU power cap

vendor power cap (amd-smi / board tool)

What it does

A power cap limits board watts below AMD’s published TBP. The Power guide already cites TBP and PSU minimums. This site does not invent an amd-smi set-power command or a tok/s-per-watt. amd-smi is the successor to rocm-smi on the cited ROCm 10.1 row.

When it helps

Single card: cap only if the room, PSU, or noise needs it — read /power first. Dual card: two TBP numbers are not a dual-card PSU SKU we invent. Decode vs watts is not measured yet on the local desktop.

Tradeoff

A lower cap can cut heat and fan noise. It can also cut clocks and decode. We do not invent either delta. TBP stays the AMD product-page cite.

# No house watt-cap command. Read /power, then your vendor tool.
# amd-smi is the cited successor to rocm-smi. Do not invent a set flag.
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Measure power and efficiency on your box. Empty watts-per-token cells stay not measured yet.

Stack Lab A/B: power and efficiency. Measure it on your box via /stack-lab planner, /power, and /amdai-bench.

Power and PSU · Hardware · Vulkan vs HIP

Vulkan vs ROCm / HIP

build-vk vs build-hip (not a runtime flag)

What it does

Vulkan llama.cpp is the consumer default. HIP is the ROCm path when rocminfo lists the gfx. Vulkan vs HIP already explains the split. Overnight 35B house cards are llama.cpp Vulkan. HIP decode for that sweep is not measured yet.

When it helps

Single card: start Vulkan. Switch to HIP only after rocminfo lists the gfx and Vulkan is the thing you are A/B-ing. Dual card: do not mix Vulkan and HIP in one process. Strata / vLLM stay on /backends.

Tradeoff

Vulkan is the measured house path. HIP can be faster or angrier depending on the gfx pin — we do not invent that delta. Same GGUF, same quality, different backend.

# HIP shape only — house 35B cards are Vulkan. HIP decode is not measured yet.
# After rocminfo lists gfx1201:
./build-hip/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 4096 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Measure Vulkan vs ROCm on your box. Do not copy a HIP tok/s from this page.

Stack Lab A/B: Vulkan vs ROCm backend. Measure it on your box via /stack-lab planner, /rocm/vulkan-hip, and /amdai-bench.

Vulkan vs ROCm · Vulkan vs HIP · llama.cpp · Backends · First-run setup

System RAM

When -ngl leaves layers on the host, the spill is system RAM. Shopper lines stay Minimum system RAM needed and RAM needed at full context for house 35B Q4_K_M on a 16 GB RX 7800 XT at 32K selected and the model’s full context. Estimate unless a measured R9700 card overlays the kit. The house daily driver on one R9700 at UD-Q5_K_XL does not offload.

Shopper lines stay Minimum system RAM needed and RAM needed at full context.

FAQ

Which knob should I turn first?

Fit the file. House daily driver is Qwen3.6-35B-A3B UD-Q5_K_XL, -ngl 99, f16 KV, llama.cpp Vulkan. If it spills, read Flash RAM offload and the RAM lines on this page. Then measure one knob at a time with amdai-bench. Do not stack five unmeasured flags.

35B recipe · Calculator · amdai-bench

Will Strata or vLLM be faster?

Maybe for a different job. This page is llama.cpp. Strata is the house Flash try. vLLM is a multi-client serve try. Both live on /backends. We do not invent a 35B vLLM or Strata decode here.

Backends · llama.cpp · Strata

Can the draft model be Q3?

UD-Q3_K_XL may appear as a labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). Draft / speculative models on this site stay Q4 and up. Q3 draft is an estimate, not a house pick. Q1/Q2 drafts stay off.

Quants · Q6 / Q5 / Q4 + Q3 estimate

How do I force valid JSON from llama.cpp?

That is constrained decoding, not a speed knob. Official --grammar / --json-schema and response_format live on /constrained-decoding. Grammar decode versus free-form is not measured on the local desktop — run Stack Lab grammar-ab/1.

Constrained decoding · Coding assistant · Stack Lab

Sources

• llama.cpp repository
• llama.cpp server README
• llama-bench README
• AMDai — Vulkan vs HIP
• AMDai — long context flags
• AMDai — dual / multi-GPU
• AMDai — board power and PSU
• AMDai — Flash RAM offload
• AMDai — backends