KV cache on a Radeon GPU: f16 vs q8_0 vs q4_0 and how much VRAM context costs

What the KV cache is

Attention stores a key vector and a value vector for every token it has already seen. The next token reads that store instead of recomputing the whole prompt. More tokens → a bigger store. Official Qwen3.6-35B-A3B card: 40 layers, 10 full-attention (GQA 2×256). Only those 10 layers grow a token-length cache — that count is the public model card, not a tape measure. The calculator growing-KV formula is 2 × kv_layers × kv_heads × head_dim × context × bytes. House numbers: 2 × 10 × 2 × 256 × context × bytes, plus the fixed linear-attention state llama.cpp keeps for the other 30 layers. Bytes come from KV_TYPES on the calculator (f16 = 2, q8_0 = 1.0625, q4_0 = 0.5625). Those KV-only figures are estimates. Peak VRAM on the table is the whole load (weights + KV + overhead) and is measured only for f16 on the overnight UD-Q5 cards.

Glossary · KV cache Open calculator

llama.cpp flags

llama.cpp CLI / server (tools/cli README and tools/server README, checked 2026-10-14): -c / --ctx-size sets the window (default 0 = model card). -ctk / --cache-type-k and -ctv / --cache-type-v set the K and V types (default f16; listed types include f16, q8_0, q4_0). -fa / --flash-attn is on|off|auto (default auto). llama.cpp multi-gpu.md: flash attention is required when using quantized V cache and/or --split-mode tensor. Overnight 35B Vulkan cards recorded -fa 1 and -ctk f16 -ctv f16. That is the house pin. llama-server --parallel / -np sizes a slot cache per concurrent sequence (server default auto). A second slot is a second KV pool — size it on the calculator before you raise -np. House daily driver is one interactive user. Copy: --cache-type-k f16 --cache-type-v f16 -c 262144. q8_0 / q4_0 stay labeled estimates on this site.

Speed up llama.cpp llama.cpp

Why the house box stays f16

the house house box runs f16 KV at the full 262144 window on one R9700 because that is what the overnight llama-bench cards recorded, and the peak (29.34 GB at 256K) still fits 32 GB with 0% RAM offload. Switching to q8_0 or q4_0 would shrink the estimated cache. It would also leave the measured cards. Quality, recall, and tok/s for q8_0 / q4_0 KV are not measured yet. Do not confuse a KV type with a weight quant — q8_0 as cache is not “running Q8 weights.”

No AMDai score ranks f16 vs q8_0 vs q4_0 KV. llama.cpp documents the types. It does not publish a house quality delta. Treat any “quantized KV is fine” claim as not measured yet unless a local desktop card says otherwise.

R9700 measured page Long context

Per-context VRAM on one R9700

House file is Qwen3.6-35B-A3B UD-Q5_K_XL on llama.cpp Vulkan, one R9700. Peak VRAM is measured only for f16 KV. The three KV-only columns are calculator estimates from the official card (10 full-attention layers × 2 heads × 256 dim). q8_0 and q4_0 totals stay not measured yet. The house window is 262144 tokens (printed 256K on the overnight cards).

Run cards Size a window

Fit by card at 262144, f16 KV

Q6 / Q5 / Q4 from the calculator unless a cell says measured. UD-Q3_K_XL is a labeled estimate — no Q3 download or run on the local desktop. When a file does not sit in VRAM, shopper RAM lines stay Minimum system RAM needed and RAM needed at full context.

Both RAM lines

When KV or weights spill off the GPU, the two shopper lines stay Minimum system RAM needed (kit-size estimate with a breakdown — 192 or 256 GB when a 35B file spills) and RAM needed at full context (measured llama.cpp peaks when we have them: 37.32 GB at 4K and 42.9 GB at 256K on UD-Q5, one R9700, f16). Those peaks are Linux GiB printed as GB. Label estimate unless a run card exists.

Stack Lab

No Stack Lab KV A/B pack (f16 vs q8_0) is allowlisted on this site. Do not invent a deep link. Measure a type on your box from /stack-lab planner and /amdai-bench after llama.cpp already serves /v1. Packages are not published yet.

Stack Lab planner amdai-bench

FAQ

• What is the KV cache on llama.cpp?

  Key/value attention memory that grows with the token window. Separate from the GGUF. House default is f16. The calculator estimates 2 × 10 full-attention layers × 2 heads × 256 dim × context × bytes plus a fixed linear-attention state for Qwen3.6-35B-A3B.

  • KV cache
  • Calculator
  • Long context

• Should I switch from f16 KV to q8_0 or q4_0 to fit more context?

  House daily driver stays f16 — overnight 35B cards recorded -ctk f16 -ctv f16, and 29.34 GB at 256K still fits one R9700. Keep f16 KV on ROCm/HIP for long context until llama.cpp PR #31234 lands: on master HIP, q8_0 KV drops decode to the vec kernel (author’s R9700 table: about −40% at 128K vs f16). Vulkan already reads q8_0 in-shader. q8_0 and q4_0 shrink the estimated cache and can free VRAM for a bigger -ub — see /prefill. Quality and tok/s for those types are not measured on the local desktop. llama.cpp multi-gpu.md requires flash attention for a quantized V cache. Do not confuse a KV type with a weight quant.

  • Flash Attention
  • Prefill
  • Quants · KV types
  • Speed up llama.cpp
  • Calculator

• How much VRAM does 128K context need on the house 35B?

  Measured peak VRAM on UD-Q5_K_XL, llama.cpp Vulkan, f16 KV, one R9700 is 28.82 GB at 128K (and 25.97 GB at 4K, 26.7 GB at 32K, 29.34 GB at 256K). The KV-only slice at 128K is a calculator estimate (about 2.50 GB at f16). q8_0 / q4_0 totals are not measured.

  • R9700
  • Run cards
  • Long context

• Does --parallel multiply the KV cache?

  Yes on llama-server. Official --parallel / -np sizes a slot cache per concurrent sequence. Two slots at the same -c is two KV pools. House path is one interactive user. Size a second slot on the calculator before you raise -np. The shopper page is /concurrency-ab. Prompt / prefix cache (cached prefixes, --cache-reuse, slot save/restore) is /prompt-cache.

  • Parallel slots
  • Prompt cache
  • Calculator
  • Tuning flags
  • Connect a harness

Sources

• llama.cpp — CLI README (-c, -ctk/-ctv, -fa, -np)
• llama.cpp — server README (same flags; --parallel slots)
• llama.cpp — multi-gpu.md (flash attention required for quantized V cache)
• Qwen3.6-35B-A3B model card (40 layers, 10 full-attention GQA 2×256, 262144 context)
• Unsloth Qwen3.6-35B-A3B GGUF (UD-Q5_K_XL / UD-Q3_K_XL listings)