Flash Attention (-fa) on a Radeon GPU with llama.cpp

What -fa does

Flash attention is a fused attention kernel. Official CLI values are on, off, or auto (default auto). Overnight house llama-bench used the numeric form -fa 1. Copy -fa on or -fa 1 when your Vulkan or HIP build documents it. If the build rejects the flag, leave it off and record the card. llama.cpp multi-gpu.md requires flash attention when using --split-mode tensor and/or a quantized V cache. Tensor mode hard-errors if FA is off. House overnight used --split-mode layer and f16 KV, so that requirement did not apply to the recorded 35B line.

KV type and quantized V

House default is f16 KV: overnight cards recorded -ctk f16 -ctv f16 (same as --cache-type-k f16 --cache-type-v f16). q8_0 and q4_0 shrink the estimated cache. llama.cpp multi-gpu.md requires FA for a quantized V cache — turn -fa on before you try -ctv q8_0 or -ctv q4_0. Quality and tok/s for q8_0 / q4_0 KV stay not measured yet. Do not confuse a KV type with a weight quant. Overnight peak VRAM on UD-Q5 is 25.97 GB at 4K and 29.34 GB at 256K — those cards already have -fa 1.

Vulkan first, HIP second

Vulkan is the house path. The recorded overnight line is llama.cpp Vulkan: -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16. llama.cpp documents the same -fa / --flash-attn and --cache-type-k/v flags on the HIP / ROCm compile. HIP FA on versus off for the house 35B sweep stays not measured on the local desktop. Do not invent a HIP flash-attn delta, a watt figure, or an acceptance rate. If a Vulkan or HIP build rejects -fa, leave it off and record the card.

What we have measured

When it works, flash attention usually cuts attention VRAM and can help long context. AMDai has not measured llama-server FA on versus off, so every server decode, prefill, VRAM, and RAM delta stays not measured yet — run Stack Lab Flash Attention A/B. Stack Lab’s Flash Attention A/B records those cells on one GPU and on all GPUs. The Prefill tuning A/B includes FA among the prefill knobs. Do not copy the overnight 102.39 tok/s 4K UD-Q5 cell as an FA-off figure.

Cited flags

Vulkan house path. The bench line is recorded. The server block is a copy-paste that matches the house recipe — not a measured FA on/off card. Replace the binary and GGUF paths.

# Overnight 35B llama-bench recorded this Vulkan line (not llama-server).
# Official CLI: -fa / --flash-attn [on|off|auto] (default auto).
./build-vk/bin/llama-bench \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16

# llama-server FA on/off is not on the overnight card.
# Official values: on | off | auto (default auto; env LLAMA_ARG_FLASH_ATTN).
# Copy -fa on when your build documents it. House recipes stay Q4 and up.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 -fa on \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

Q6 / Q5 / Q4 plus a labeled Q3

Flash attention is a kernel, not a smaller GGUF. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run.

• Q6_K

  Qwen3.6-35B-A3B-Q6_K.gguf. Overnight one-client llama-bench exists with -fa 1. llama-server FA on/off stays not measured yet.

• UD-Q5_K_XL

  Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf. House file. One-client 102.39 tok/s at 4K is llama-bench with -fa 1, not an FA-off A/B.

• Q4_K_M

  Qwen3.6-35B-A3B-Q4_K_M.gguf. Overnight one-client llama-bench exists with -fa 1. Recommended floor for house recipes. FA on/off still needs a Flash Attention A/B card.

• UD-Q3_K_XL · labeled estimate

  Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf. Labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). Do not download Q3 to run. Q3 does not buy flash attention.

Zero to a measured FA pin

1. 1 · Cited llama.cpp --flash-attn · llama-server FA on/off not measured

   Keep the house Vulkan pin with -fa 1

   Vulkan llama.cpp, Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf, -c 262144, f16 KV, -fa 1 (or -fa on). That matches the overnight llama-bench line. Q6/Q5/Q4 first. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run.

   • Speed up llama.cpp
   • Serve Qwen
   • Qwen3.6-35B

2. 2 · Cited llama.cpp --flash-attn · llama-server FA on/off not measured

   Use on, off, or auto — default is auto

   Official CLI is -fa / --flash-attn [on|off|auto] (default auto; env LLAMA_ARG_FLASH_ATTN). Overnight cards used the numeric form -fa 1. Copy -fa on when the current build documents the enum. If the build rejects the flag, leave it off and record the card.

   • llama.cpp
   • Always-on server
   • Parallel slots
   • Prefill

3. 3 · Cited llama.cpp --flash-attn · llama-server FA on/off not measured

   Turn FA on before a quantized V cache or tensor split

   llama.cpp multi-gpu.md requires flash attention for --split-mode tensor and/or quantized V cache. Tensor mode hard-errors if FA is off. House overnight used layer split and f16 KV. Size q8_0 / q4_0 KV on the calculator first — quality and tok/s stay not measured yet.

   • KV cache
   • Multi-GPU llama.cpp
   • Calculator

4. 4 · Cited llama.cpp --flash-attn · llama-server FA on/off not measured

   Vulkan first; HIP / ROCm is the cited second compile

   The recorded 35B line is Vulkan. llama.cpp documents the same -fa flags on HIP. HIP FA on versus off for the house 35B sweep stays not measured on the local desktop. Do not invent a HIP delta, watts, or an acceptance rate.

   • Vulkan vs ROCm
   • Vulkan vs HIP
   • Long context

5. 5 · Cited llama.cpp --flash-attn · llama-server FA on/off not measured

   Measure Stack Lab Flash Attention A/B — do not invent tok/s

   Stack Lab ships a Flash Attention A/B (on vs off) and a Prefill tuning A/B that includes FA. Those A/Bs record decode, prefill, peak VRAM per card, and total / peak system RAM on one GPU and on all GPUs. Packages are not published yet and there is no public download. Open the planner — do not invent a flash-attention deep link.

   • Stack Lab
   • amdai-bench
   • Benchmarks
   • Run cards
   • How we test

RAM lines (Ubuntu-measured)

Minimum system RAM needed on one R9700 with the house UD-Q5 is a 64 GB kit from the measured 37.32 GB peak at 4K (weights + KV + overhead on GPU RAM plus total system DDR — not a file-cache versus anonymous split). RAM needed at full context uses the measured 42.9 GB peak at 256K. Those are llama-bench cards with -fa 1, one client, 0% RAM offload. They are not an FA-off measurement. UD-Q3_K_XL is a labeled estimate. Do not download Q3 to run.

Open in Stack Lab

Flash Attention A/B and Prefill tuning A/B are the A/Bs. They are not allowlisted deep links on this site. Use the planner. Packages are not published yet and there is no public download.

Sources

• llama.cpp — CLI README (-fa / --flash-attn on|off|auto)
• llama.cpp — multi-gpu.md (FA required for tensor split and/or quantized V cache)
• llama.cpp — llama-server README
• llama.cpp repository
• Qwen3.6-35B-A3B official model card