Faster prompt processing (prefill) on a Radeon GPU with llama.cpp: batch (-b) and micro-batch (-ub)

What -b and -ub do

-b is how much prompt llama.cpp plans for. -ub is the chunk the GPU actually computes. Official llama-server README: -b / --batch-size N is the logical maximum batch size (default 2048; env LLAMA_ARG_BATCH). -ub / --ubatch-size N is the physical maximum batch size (default 512; env LLAMA_ARG_UBATCH). The CLI README documents the same pair. A larger -b with a smaller -ub still processes the prompt in -ub-sized pieces. House recipes do not pin either flag — the binary default applies unless you set one. Do not treat the overnight llama-bench prefill cell as a llama-server -b pin.

How they meet Flash Attention

Flash attention is a fused attention kernel. Official CLI is -fa / --flash-attn on|off|auto (default auto; env LLAMA_ARG_FLASH_ATTN). Overnight house 35B Vulkan llama-bench recorded -fa 1 on the same line as the cited prefill cells. llama.cpp multi-gpu.md requires flash attention for --split-mode tensor and/or a quantized V cache. When FA works it usually cuts attention VRAM, which can leave more room for a larger -ub — that leftover is not measured as a -b / -ub delta. llama-server FA on versus off stays not measured yet, run amdai-bench ab-prefill. Shopper page: /flash-attention.

KV type and leftover VRAM

House default is f16 KV: overnight cards recorded -ctk f16 -ctv f16 (same as --cache-type-k f16 --cache-type-v f16). q8_0 and q4_0 shrink the estimated cache and can free VRAM for a bigger physical micro-batch. llama.cpp multi-gpu.md requires FA for a quantized V cache — turn -fa on before -ctv q8_0 or -ctv q4_0. Quality and tok/s for those KV types stay not measured yet. Overnight peak VRAM on UD-Q5 is 25.97 GB at 4K and 29.34 GB at 256K on a 32 GB R9700. Those cards already have -fa 1 and f16 KV. They are not a -b / -ub A/B.

Context size shrinks headroom

Context size grows the KV cache. House window is 262144. Official UD-Q5 one-GPU VRAM is 25.97 GB at 4K and 29.34 GB at 256K — leftover on a 32 GB R9700 shrinks as -c rises. A larger -ub needs a bigger transient compute buffer. llama.cpp #14836 failed to allocate Vulkan compute pp buffers at -b 4096 -ub 4096. Cut -ub when the log shows that allocate, then measure prefill-ab/1. Do not invent a 262K speedup.

What we have measured

Larger -ub can raise prefill when the GPU has unused VRAM for the compute buffer. Smaller -ub can fit a tight card and slow prompt ingest. Quality is unchanged if the GGUF stays the same. AMDai has not measured llama-server -b / -ub, so every server prefill, VRAM, and RAM delta stays not measured yet, run amdai-bench ab-prefill. Overnight llama-bench prefill cells below are one-client Vulkan cards with -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16 — not a -b / -ub sweep. Do not copy 2761.6 tok/s at 4K as a “bigger batch” figure.

• UD-Q5_K_XL — House file. llama-bench, one R9700, f16 KV, -fa 1. Not a llama-server -b / -ub A/B.
• Q6_K — llama-bench. A second 4K card recorded 2621.7 tok/s. 32K and up partly on DDR5 (--fit-target 1024).
• Q5_K_M — llama-bench, one R9700, f16 KV. Not a llama-server -b / -ub A/B.
• Q4_K_M — llama-bench, one R9700, f16 KV. Recommended floor for house recipes.

Cited flags

Vulkan house path. The first block is the recorded llama-server line — no -b / -ub. The second is the official server-README default pair, not a local desktop pin. The bench line is recorded. Flash does not record -b / -ub. Replace the binary and GGUF paths.

# House daily driver as recorded (no -b / -ub pin).
# llama-server README defaults: -b 2048 (LLAMA_ARG_BATCH), -ub 512 (LLAMA_ARG_UBATCH).
# Those defaults are official docs — not a local desktop pin we will invent.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja --host 127.0.0.1 --port 8080

# Official llama.cpp defaults from the server README — not a house pin.
# Measure prefill-ab/1 (amdai-bench ab-prefill) before you keep a new -b / -ub.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 -b 2048 -ub 512 \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

# Overnight 35B llama-bench recorded this Vulkan line (not llama-server).
# No -b / -ub on the card. Prefill cells are llama-bench pp, not a batch A/B.
./build-vk/bin/llama-bench \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16

# Hermes Flash llama.cpp shape as recorded. -b / -ub not recorded.
# Exact -ot tensor regex and --n-cpu-moe N are not recorded from the rig owner.
# Strata Flash stays on 127.0.0.1:8083 so llama-server can keep :8080.
# Flash prefill: not measured yet, run amdai-bench ab-prefill.
./build-vk/bin/llama-server \
  -m ~/models/UD-Q6_K_XL \
  -ngl 99 \
  -c 262144 \
  --n-cpu-moe not recorded \
  -ot not recorded \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --host 127.0.0.1 --port 8080

Q6 / Q5 / Q4 plus a labeled Q3

Batch size is a compute-buffer knob, not a smaller GGUF. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run. Both RAM lines appear when RAM offload appears (Q6 at 32K and up).

• Q6_K

  Qwen3.6-35B-A3B-Q6_K.gguf

  2623.6 / 1683 / 800.3 / 471.6 tok/s at 4K / 32K / 128K / 256K (llama-bench)

  Peak VRAM: 29.26 GB at 4K (sibling 29.32) · 29.3 GB at 256K

  Minimum system RAM needed: not recorded on the Q6 cells — not measured yet, run amdai-bench ab-prefill

  RAM needed at full context: 32K and up partly on DDR5 (--fit-target 1024); peak kit not recorded

  RAM offload appears at 32K and up. llama-server -b / -ub stays not measured yet.

• UD-Q5_K_XL

  Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf

  2761.6 / 1937.9 / 942.8 / 519.2 tok/s at 4K / 32K / 128K / 256K (llama-bench)

  Peak VRAM: 25.97 GB at 4K · 29.34 GB at 256K

  Minimum system RAM needed: 64 GB kit from the measured 37.32 GB peak at 4K

  RAM needed at full context: measured 42.9 GB peak at 256K (0% RAM offload)

  House file. One-client 102.39 tok/s decode at 4K is llama-bench, not a batch A/B.

• Q4_K_M

  Qwen3.6-35B-A3B-Q4_K_M.gguf

  2733.9 / 1902.1 / 942.1 / 558.5 tok/s at 4K / 32K / 128K / 256K (llama-bench)

  Peak VRAM: 21.43 GB at 4K · 27.03 GB at 256K

  Minimum system RAM needed: peak system RAM not recorded on the Q4 cells

  RAM needed at full context: peak system RAM not recorded on the Q4 cells

  Recommended floor for house recipes. More unused VRAM than UD-Q5 at the same -c.

• UD-Q3_K_XL · labeled estimate

  Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf

  not measured

  Peak VRAM: Publisher listing 16.8 GB (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB))

  Minimum system RAM needed: 32 GB kit (labeled estimate)

  RAM needed at full context: 32 GB kit (labeled estimate)

  Labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). Do not download Q3 to run. Q3 does not buy a prefill tok/s.

VRAM headroom by published card

Unused VRAM after weights + KV is the shopper headroom for a bigger -ub. Official /gpus VRAM is the published card size, not a measured -ub pin. House 32 GB R9700 at 256K UD-Q5 leaves about 2.7 GB after the measured 29.34 GB peak. A 48 GB W7900 publishes more unused VRAM than that 32 GB class. A 24 GB RX 7900 XTX cannot hold the house 256K UD-Q5 peak (29.34 GB). House 4K Q4_K_M is 21.43 GB — that already misses 20 GB and 16 GB cards. Do not invent an -ub that fits a SKU we did not measure.

• Radeon AI PRO R9700 · 32 GB published

  House card. Measured UD-Q5 peak 25.97 GB at 4K and 29.34 GB at 256K — about 2.7 GB leftover at 256K for a bigger -ub experiment.

• Radeon PRO W7900 · 48 GB published

  Published 48 GB. More unused VRAM than the house 32 GB class after the same measured 29.34 GB 256K UD-Q5 peak. -ub still unmeasured.

• Radeon PRO W7800 · 32 GB published

  Published 32 GB — same VRAM class as the house R9700. Do not copy the R9700 leftover as a measured pin on this SKU.

• Radeon RX 7900 XTX · 24 GB published

  Published 24 GB. House 256K UD-Q5 peak 29.34 GB does not fit. House 4K Q4_K_M is 21.43 GB — a little leftover, not a measured -ub.

• Radeon RX 7900 XT · 20 GB published

  Published 20 GB. House 4K Q4_K_M is 21.43 GB — that file does not fit. Do not raise -ub on a card that cannot hold the weights + KV.

• Radeon RX 9070 XT · 16 GB published

  Published 16 GB. House 4K Q4_K_M is 21.43 GB — 35B Q4 and up miss this class. Size a smaller model on the calculator.

Zero to a measured -b / -ub pin

1. 1 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Keep the house Vulkan pin — do not invent -b / -ub

   Vulkan llama-server, Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf, -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja --host 127.0.0.1 --port 8080. That is the recorded house line. Official docs default to -b 2048 -ub 512 if you omit both. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run.

   • Speed up llama.cpp
   • Serve Qwen
   • Qwen3.6-35B

2. 2 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Treat -b as logical and -ub as physical

   Official llama-server README: -b / --batch-size is the logical maximum (default 2048; env LLAMA_ARG_BATCH). -ub / --ubatch-size is the physical maximum (default 512; env LLAMA_ARG_UBATCH). The CLI README documents the same pair. The GPU sees -ub-sized chunks. Copy the docs example only as a starting point, then measure.

   • llama.cpp
   • Always-on server
   • Batch (-b)

3. 3 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Size -fa, KV type, and -c before you raise -ub

   Overnight cards already used -fa 1 and f16 KV. llama.cpp multi-gpu.md requires FA for tensor split and/or a quantized V cache. Larger -c grows KV and shrinks leftover VRAM — house UD-Q5 is 25.97 GB at 4K and 29.34 GB at 256K. llama.cpp #14836 failed compute pp buffers at -b 4096 -ub 4096. Cut -ub when that allocate appears.

   • Flash Attention
   • KV cache
   • Long context

4. 4 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Use leftover VRAM, not a guessed -ub, to pick a card

   Published /gpus VRAM minus measured weights+KV is the only headroom we will quote. A 48 GB W7900 publishes more unused VRAM than the house 32 GB R9700 after the 29.34 GB 256K peak. 24 GB and below miss that 256K UD-Q5 cell. Do not invent an -ub that fits a SKU we did not measure.

   • R9700
   • GPU directory
   • Out of VRAM

5. 5 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Long Hermes, tools, and RAG prompts are prefill-bound

   Decode tok/s is chat feel. Prefill tok/s is prompt ingest. Official UD-Q5 4K is 2761.6 tok/s prefill and 102.39 decode; at 256K the same file is 519.2 / 44.4. Those are llama-bench, not a batch A/B. Flash / Hermes llama.cpp does not record -b / -ub. Flash prefill stays not measured yet, run amdai-bench ab-prefill.

   • RAG
   • Hermes
   • Flash RAM offload

6. 6 · Cited llama.cpp --batch-size / --ubatch-size · llama-server -b / -ub not measured

   Measure Stack Lab prefill-ab/1 — do not invent tok/s

   Stack Lab ships prefill-ab/1 (amdai-bench ab-prefill). Record decode, prefill, peak VRAM per card, and both RAM lines on one GPU and on all GPUs. Packages are not published yet and there is no public download. Open the planner — do not invent a prefill-ab deep link.

   • Stack Lab
   • amdai-bench
   • Benchmarks
   • Run cards
   • How we test

RAM lines (Ubuntu-measured)

Minimum system RAM needed on one R9700 with the house UD-Q5 is a 64 GB kit from the measured 37.32 GB peak at 4K (weights + KV + overhead on GPU RAM plus total system DDR — not a file-cache versus anonymous split). RAM needed at full context uses the measured 42.9 GB peak at 256K. Those are llama-bench cards with 0% RAM offload, one client. They are not a llama-server -b / -ub measurement. Q6_K at 32K and up is partly on DDR5 (--fit-target 1024) — both RAM labels appear there; the peak kit on those cells is not recorded. UD-Q3_K_XL is a labeled estimate. Do not download Q3 to run.

Open in Stack Lab

Pack prefill-ab/1 is the prefill A/B. It is not an allowlisted deep link on this site. Use the planner. Packages are not published yet and there is no public download. Anything unmeasured stays not measured yet, run amdai-bench ab-prefill.

Sources

• llama.cpp — llama-server README (-b / --batch-size, -ub / --ubatch-size, LLAMA_ARG_BATCH / LLAMA_ARG_UBATCH)
• llama.cpp — CLI README (same -b / -ub pair)
• llama.cpp — multi-gpu.md (FA required for tensor split and/or quantized V cache)
• llama.cpp #14836 — failed to allocate compute pp buffers at -b 4096 -ub 4096
• llama.cpp repository
• Qwen3.6-35B-A3B official model card