Context length is how many tokens of prompt plus chat the model can keep. That memory is the KV cache — it grows as the window fills, so VRAM rises and decode usually falls. On AMDai the measured house path is Qwen3.6-35B-A3B on llama.cpp Vulkan, f16 KV, one R9700. Empty cells stay not measured yet. Q3 is a labeled estimate. llama.cpp stays first.
Measured peak system RAM on the local desktop: 42.9 GB (Linux GiB labeled GB). Measured, pending re-verify after RAM fix.
Breakdown
Offloaded weights ·
0 GiB
Separable embed in RAM ·
0 GiB
KV in RAM at selected ·
0 GiB
KV in RAM at full context ·
0 GiB
Runtime / backend ·
0.85 GiB
OS / desktop baseline ·
8.00 GiB
Headroom ·
16.0 GiB
What context length and the KV cache are
Context length
is the token window. The
KV cache
is the attention memory that grows with that window — separate from the GGUF.
Prefill
reads the prompt in a burst;
decode is the streaming rate people mean by “how fast.”
Offload
is what happens when weights or KV spill to DDR5.
Qwen3.6-35B-A3B on llama.cpp Vulkan, f16 KV. Every tok/s and measured GB
is from official amdai-bench/run@1 cards. A missing cell says
not measured yet.
Shopper RAM lines stay Minimum system RAM needed and
RAM needed at full context.
One R9700 · short
Qwen3.6-35B-A3B one GPU at short. Figures from local desktop run cards.
Q6_K: Q6_K single-GPU 4K table cell is 103.23 tok/s from the 23:55 card (the 4K point in the complete Q6_K single-GPU series). The earlier 23:26 card at 82.62 tok/s stays on /results and is not averaged into the cell.
UD-Q3_K_XL: Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB). Q3 is an option labeled estimate — not a local desktop run.
One R9700 · 32K
Qwen3.6-35B-A3B one GPU at 32K. Figures from local desktop run cards.
UD-Q3_K_XL: Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB). Q3 is an option labeled estimate — not a local desktop run.
Qwen 3.8 Flash where we have cards
Decode from verified Flash run cards. Prefill, VRAM, and peak RAM stay not measured yet unless a card recorded them — open the linked run card.
Empty cells stay not measured yet. Do not map 100K onto 128K.
Shopper RAM lines stay Minimum system RAM needed and
RAM needed at full context on
Flash RAM offload
and the
Flash model page.
Flash decode at 4K / 32K / 128K / 256K from local desktop cards. Rows with no measured cell are omitted.
4K and 128K Flash decode stay not measured yet. Do not map the 100K cards onto 128K.
8K and 100K Q6 two-card decode sit on the DIMM-speed table — those are the Flash cards we have.
f16 vs q8_0 KV
House default is f16 KV (2 bytes). Overnight 35B cards recorded -ctk f16 -ctv f16. q8_0 KV is the smaller type the calculator lists (~1 byte). That trade — less VRAM, unknown quality — is not measured yet on the local desktop. q8_0 as cache is not “running Q8 weights.” Minimum system RAM needed and RAM needed at full context stay the shopper RAM lines when offload or long context appears. The cited shopper page is /kv-cache.
Measured 35B llama.cpp quants on the local desktop: UD-Q5_K_XL, Q5_K_M, Q4_K_M, Q6_K. UD-Q3_K_XL is a labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)) — decode stays not measured yet. Prefer Q5 over Q4 when the file still fits. Q1 and Q2 stay off. Nothing below Q4 runs on our rig.
House daily driver is -c 262144. Size that number on the calculator before you copy it. A 16 GB card that loads at 4K can still OOM at 128K because KV grew.
Overnight 35B llama-bench cards recorded -ctk f16 -ctv f16 (same as --cache-type-k/v f16). That is the AMDai default. q8_0 is the smaller KV type the calculator lists. Quality and tok/s for q8_0 KV are not measured yet.
The same llama-bench line recorded -fa 1 (flash attention). llama-server flags are not on that card — do not invent a server pin. Copy -fa only when your build documents it. The shopper page is /flash-attention. Prefill / batch (-b / -ub) is /prefill. Prompt / prefix cache is /prompt-cache.
Overnight 35B split used -sm layer (--split-mode layer), the documented llama.cpp multi-GPU default. On this 35B, one card usually decodes faster than the split. --split-mode row is documented and not measured on the local desktop. Read Dual / multi-GPU before you split.
llama.cpp Vulkan is the consumer default. HIP when rocminfo lists the gfx. Strata is the house Flash speed path with DDR5 expert offload — see Flash RAM offload. vLLM is a try, not the house long-context path. We do not invent a vLLM 262K tok/s.
Tighter than the 16 GB class. House 35B Q4–Q6 files do not sit in 12 GB on the calculator at long context. Expect RAM offload or a shorter window. Q3 is a labeled estimate. Size the file before you set -c 131072.
House 35B Q4–Q6 files do not sit in 16 GB on the calculator at long context. Expect RAM offload or a shorter window. Q3 is a labeled estimate, not runnable advice. Size the file before you set -c 131072.
More VRAM than the 16 GB class. Q4 may fit at a shorter window; Q5 at 262K still needs the calculator. Speed on this class is not measured yet on the local desktop.
Measured house card. UD-Q5_K_XL, llama.cpp Vulkan, f16 KV, 262K, 0% RAM offload. Official 1-GPU decode: 102.39 tok/s at 4K, 90.52 at 32K, 66.43 at 128K, 44.4 at 256K.
A split of the same 35B often decodes slower than one card at short context. Two copies serve two chats. Flash Q6 at 262K is the measured DDR5-offload job — not a free 2× of the 35B.
The run used the context we printed (262144 for the daily driver). We do not publish a planted-token recall score. We do not publish a planted-token percentage. The daily-driver and Flash recipes mark the long-context gate as pass when the stated 262144 window actually ran — that is not a recall score. Liu et al., 2023 (Lost in the Middle) show many models miss facts in the middle of a long prompt. The Qwen3.6-35B-A3B card lists 262144 native context. Neither source is an AMDai needle-in-haystack number.
FAQ
What is context length, and what is the KV cache?
Context length is the token window of prompt plus conversation. The KV cache is the key/value attention memory that grows with that window. It is separate from the GGUF. House daily driver is 262144 tokens, f16 KV. See the glossary entries.
Why does VRAM rise and decode fall as the context fills?
Each new token adds to the KV cache. Official UD-Q5_K_XL on one R9700 is 102.39 tok/s at 4K and 44.4 tok/s at 256K — same file, f16 KV. Prefill is the burst that reads the prompt; decode is the streaming rate. Quote the run card, not a blend.
Should I switch from f16 KV to q8_0 to fit more context?
House default is f16 KV. Keep f16 on ROCm/HIP for long context until llama.cpp PR #31234 lands — on master HIP, q8_0 KV is slower at depth (author’s R9700 table, cited). Vulkan is already fine with q8_0. q8_0 KV is about one byte per element instead of two, so the cache is smaller. That quality trade is not measured on the local desktop — treat any q8_0 KV speed or recall claim as a labeled estimate. Do not confuse q8_0 KV with Q8 weights.
Q3 is a listed option and a labeled estimate. Nothing on our rig runs below Q4. Do not treat Q3 as runnable advice. Size Q6 / Q5 / Q4 on the calculator first. Q1 and Q2 stay off.
Does a 262K window mean the model still finds a fact from the start of the prompt?
No score on this site answers that. The long-context gate means the stated window actually ran — not a planted-token recall percentage. Liu et al. 2023 (Lost in the Middle) document that many models miss the middle of a long prompt. We have not run that test on the local desktop.