Prompt / prefix cache on a Radeon GPU with llama.cpp

What the cache does

Prompt cache (also called prefix cache) keeps processed prompt tokens so a later request that shares a prefix skips that prefill. Official llama-server README: --cache-prompt / --no-cache-prompt enables it (default enabled; env LLAMA_ARG_CACHE_PROMPT). The completion API field cache_prompt does the same per request (default true): the common prefix is not re-processed, only the suffix. Official docs warn that logits are not guaranteed bit-for-bit identical across batch sizes, so enabling cache_prompt can be nondeterministic. --cache-reuse N is the min chunk size for reuse via KV shifting (default 0 = disabled; env LLAMA_ARG_CACHE_REUSE). It requires prompt caching to be on. n_cache_reuse on the API matches that flag.

--cache-ram and system RAM

Official -cram / --cache-ram N sets the maximum prompt-cache size in MiB (default 8192, -1 no limit, 0 disable; env LLAMA_ARG_CACHE_RAM). llama.cpp PR #16391 is the cited write-up. That ceiling is system RAM for cached prefixes, not VRAM for weights. House overnight UD-Q5 peaks stay 37.32 GB at 4K and 42.9 GB at 256K — those cards do not record --cache-ram. Adding the official 8192 MiB default on top of that process is a labeled estimate until a card measures it. --cache-idle-slots / --no-cache-idle-slots saves idle slots into the prompt cache on a new task and clears them when using unified KV (default enabled; requires cache-ram; env LLAMA_ARG_CACHE_IDLE_SLOTS).

Slots, unified KV, save/restore

Official --slot-save-path PATH enables saving a slot’s KV / prompt cache to disk (default disabled; env LLAMA_ARG_SLOT_SAVE_PATH). POST /slots/{id_slot}?action=save and action=restore take a filename under that directory. -np / --parallel is the slot count. -kvu / --kv-unified shares one KV buffer across sequences (default on when slots are auto). --cache-idle-slots then parks an idle slot into the prompt cache so a unified buffer can take the next task. House daily driver stays one interactive slot. Slot save/restore TTFT stays not yet measured. Shopper page for slots: /concurrency-ab.

When it helps — and when it does not

The cache helps when the prefix is stable: a long Hermes system prompt, a coding-tool preamble, thinking-mode instructions, or RAG with a fixed lead-in. The next turn then prefills only the new user text. It does not help a one-shot prompt, and it does not help when the prefix changes every request (rotating tools, a rewritten system message, or a RAG pack that prepends different chunks). Do not copy the overnight 102.39 tok/s 4K decode cell as a cache-hit TTFT. That number is llama-bench, one client, not a prefix-cache A/B.

Cited flags

Vulkan house path. The first block is the recorded llama-server line — no --cache-reuse, --cache-ram, or --slot-save-path. The second is the official server-README default trio, not a local desktop pin. The third is slot save/restore, also not a pin. Replace the binary and GGUF paths.

# House daily driver as recorded (no --cache-reuse / --cache-ram / --slot-save-path pin).
# llama-server README: --cache-prompt default on; --cache-reuse 0; --cache-ram 8192 MiB.
# Those defaults are official docs — not a local desktop pin we will invent.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja --host 127.0.0.1 --port 8080

# Official llama.cpp defaults from the server README — not a house pin.
# --cache-prompt (default on; env LLAMA_ARG_CACHE_PROMPT).
# --cache-reuse 0 disables KV-shift reuse (env LLAMA_ARG_CACHE_REUSE).
# --cache-ram 8192 MiB (0 disables, -1 unlimited; env LLAMA_ARG_CACHE_RAM).
# Measure prompt-cache-ab/1 (amdai-bench ab-prompt-cache) before you keep a new pin.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --cache-prompt --cache-reuse 0 --cache-ram 8192 \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

# Slot save/restore. Official --slot-save-path default is disabled.
# POST /slots/{id_slot}?action=save and action=restore write under this directory.
# Not a local desktop pin. Slot TTFT stays not yet measured.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --slot-save-path /path/to/slot-cache \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

Q6 / Q5 / Q4 plus a labeled Q3

Prompt cache is a prefix reuse knob, not a smaller GGUF. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run. Both RAM lines appear. Cache-hit TTFT stays not yet measured.

• Q6_K

  Qwen3.6-35B-A3B-Q6_K.gguf

  TTFT: not yet measured

  Minimum system RAM needed: peak system RAM not recorded on a prompt-cache card

  RAM needed at full context: peak system RAM not recorded on a prompt-cache card

  Overnight llama-bench exists. Prompt-cache TTFT stays not yet measured.

• UD-Q5_K_XL

  Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf

  TTFT: not yet measured

  Minimum system RAM needed: 64 GB kit from the measured 37.32 GB peak at 4K (no --cache-ram on that card)

  RAM needed at full context: measured 42.9 GB peak at 256K. Official --cache-ram 8192 MiB is a labeled estimate on top.

  House file. One-client 102.39 tok/s decode at 4K is llama-bench, not a cache-hit TTFT.

• Q4_K_M

  Qwen3.6-35B-A3B-Q4_K_M.gguf

  TTFT: not yet measured

  Minimum system RAM needed: peak system RAM not recorded on a prompt-cache card

  RAM needed at full context: peak system RAM not recorded on a prompt-cache card

  Overnight llama-bench exists. Recommended floor for house recipes. Cache A/B still needs prompt-cache-ab/1.

• UD-Q3_K_XL · labeled estimate

  Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf

  TTFT: not yet measured

  Minimum system RAM needed: 32 GB kit (labeled estimate)

  RAM needed at full context: 32 GB kit (labeled estimate)

  Labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). Do not download Q3 to run. Q3 does not buy a prompt-cache TTFT.

Run it yourself

# Stack Lab pack prompt-cache-ab/1. Not an allowlisted deep link — open the planner.
amdai-bench ab-prompt-cache
amdai-bench ab-prompt-cache-status

# Export a lab-notes-report/1 pack from Stack Lab, then ingest into Lab Notes.
# Each row must be a validating amdai-bench/run@1 card. Do not hand-type tok/s.
npx tsx src/scripts/amdai-bench.ts ingest --dry-run pack.json
npx tsx src/scripts/amdai-bench.ts ingest pack.json
npx tsx src/scripts/amdai-bench.ts ingest pack.json data/runs/stack-lab-import.jsonl
npx tsx src/scripts/amdai-bench.ts validate

Zero to a measured prompt-cache pin

1. 1 · Cited llama.cpp --cache-prompt / --cache-reuse · llama-server TTFT not yet measured

   Keep the house Vulkan pin — do not invent a cache flag

   Vulkan llama-server, Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf, -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja --host 127.0.0.1 --port 8080. Official --cache-prompt is on by default. House recipes do not record --cache-reuse, --cache-ram, or --slot-save-path. House recipes stay Q4 and up. UD-Q3_K_XL is a labeled estimate — do not download it to run.

   • Always-on server
   • Serve Qwen
   • Qwen3.6-35B

2. 2 · Cited llama.cpp --cache-prompt / --cache-reuse · llama-server TTFT not yet measured

   Reuse a stable prefix — skip a changing one

   Official cache_prompt reuses the common prefix and prefills only the suffix. That helps Hermes, a coding tool, thinking instructions, or RAG with a fixed preamble. It does not help a one-shot prompt or a prefix that changes every turn. TTFT with versus without the cache stays not yet measured.

   • Harnesses
   • Thinking mode
   • RAG

3. 3 · Cited llama.cpp --cache-prompt / --cache-reuse · llama-server TTFT not yet measured

   Cite --cache-reuse and --cache-ram before you raise them

   Official --cache-reuse N is the min chunk for KV-shift reuse (default 0 = off). --cache-ram N is the MiB ceiling (default 8192; 0 disables; -1 unlimited). Minimum system RAM needed and RAM needed at full context stay the measured 64 GB kit from 37.32 / 42.9 GB unless a card records a cache-ram peak — extra cache RAM is a labeled estimate.

   • KV cache
   • Long context
   • Calculator

4. 4 · Cited llama.cpp --cache-prompt / --cache-reuse · llama-server TTFT not yet measured

   Size slots before you save them to disk

   Official --slot-save-path enables POST /slots save and restore. -np is the slot count. --kv-unified shares one KV buffer; --cache-idle-slots parks an idle slot into the prompt cache. House path is one interactive user. Slot TTFT stays not yet measured. Do not copy a one-client 35B tok/s as a restore figure.

   • Parallel slots
   • Always-on server
   • KV cache

5. 5 · Cited llama.cpp --cache-prompt / --cache-reuse · llama-server TTFT not yet measured

   Run amdai-bench ab-prompt-cache — do not invent TTFT

   Stack Lab ships prompt-cache-ab/1 (amdai-bench ab-prompt-cache, plus ab-prompt-cache-status). Record TTFT on a cache hit versus a miss, peak VRAM, and both RAM lines. Export a lab-notes-report/1 pack and ingest with npx tsx src/scripts/amdai-bench.ts ingest pack.json. Packages are not published yet. Open the planner — do not invent a prompt-cache deep link.

   • Stack Lab
   • amdai-bench
   • Run cards
   • How we test

RAM lines (Ubuntu-measured)

Minimum system RAM needed on one R9700 with the house UD-Q5 is a 64 GB kit from the measured 37.32 GB peak at 4K. RAM needed at full context uses the measured 42.9 GB peak at 256K. Those are llama-bench cards with 0% RAM offload and no recorded --cache-ram. Official default --cache-ram 8192 MiB can grow system RAM for cached prefixes — labeled estimate unless a card measures the peak. UD-Q3_K_XL is a labeled estimate. Do not download Q3 to run.

Open in Stack Lab

Pack prompt-cache-ab/1 is the prompt / prefix cache A/B. It is not an allowlisted deep link on this site. Use the planner. Packages are not published yet and there is no public download. Anything unmeasured stays not yet measured.

Sources

• llama.cpp — llama-server README (--cache-prompt, --cache-reuse, --cache-ram, --slot-save-path, cache_prompt, POST /slots save/restore)
• llama.cpp PR #16391 — --cache-ram prompt-cache size limit
• llama.cpp repository
• Qwen3.6-35B-A3B official model card