From the test rig · snapshot 2026-10-12
Recommended stacks
Our test rig: Two AMD Radeon AI PRO R9700 (32 GB GDDR6, gfx1201), 256 GB DDR5.Daily driver uses ONE card. The second card is there; do not assume a split.Command lines and UD-Q5_K_XL VRAM stay marked TODO. Decode tok/s for the 35B is filled — seebenchmarks. We do not invent flags.
Consumer house rules
How we pick a stack
- 01Prefer llama.cpp over vLLM for a home box. vLLM is a serve try, not the daily driver.
- 02Prefer Q5 over Q4. Q4_K_M is the floor. Never Q1, Q2, Q3, or IQ1–IQ3 — including draft / MTP heads.
- 03Keep GGUFs on fast NVMe. Do not run a 20-plus-GB file from a spinning disk.
- 04One LLM per GPU unless you deliberately combine (tensor/layer split).
- 05Daily driver on our rig uses 0% system RAM offload. Offload is slow — treat it as a last resort.
Measured on our rig, 2026-10
35B decode, measured
Qwen3.6-35B-A3B on one R9700, llama.cpp, f16 KV. Cells that we still do not have stay not measured. Units match the hardware labels (GB). Full hub: benchmarks.
| Quant | Decode · short | Decode · 32K | Decode · 256K | Prefill · 32K | VRAM |
|---|---|---|---|---|---|
| UD-Q5_K_XL | 100.8 tok/s | ~88 tok/s | ~50 tok/s | not measured | not measured |
| Q5_K_M | 108.3 tok/s | not measured | not measured | not measured | 28.5 GB |
| Q4_K_M | 116.5 tok/s | not measured | not measured | 1814 tok/s | not measured |
- One R9700, llama.cpp, f16 KV. Exact command lines stay TODO.
- UD-Q5_K_XL VRAM is not measured. Q5_K_M VRAM is 28.5 GB.
- Prefill is published only for Q4_K_M at 32K.
One R9700 · llama.cpp Vulkan
Daily driver
Open in calculatorFull native context on one 32 GB card, Q5-family quality, no RAM crawl.
- Model
- Qwen3.6-35B-A3B
- Qwen/Qwen3.6-35B-A3B
- Weights
- unsloth/Qwen3.6-35B-A3B-GGUF
- UD-Q5_K_XL
- Engine
- llama.cpp (Vulkan backend)
- GPUs
- 1 × Radeon AI PRO R9700 (32 GB, gfx1201)
- Context / KV
- 262144 · f16
- RAM offload
- 0% system RAM
What we know / still TODO
- Publisher GGUF listing 26.6 GB (Hugging Face) — file size, not VRAM
- Context 262144 and f16 KV confirmed on our rig
- UD-Q5_K_XL decode: 100.8 tok/s at short context, ~88 tok/s at 32K, ~50 tok/s at 256K. Measured on our rig, 2026-10. One R9700, llama.cpp, f16 KV.
- Q5_K_M sibling: 108.3 tok/s, 28.5 GB VRAM. Measured on our rig, 2026-10. One R9700, llama.cpp, f16 KV.
- Q4_K_M sibling: 116.5 tok/s decode; 1814 tok/s prefill at 32K. Measured on our rig, 2026-10. One R9700, llama.cpp, f16 KV.
- TODO: exact command line / measured VRAM for UD-Q5_K_XL from rig owner.
Plug a harness into llama-server on
127.0.0.1:8080/v1aftercurlworks.Hermes Agentis used on our rig.Connect snippets· All harnesses.Strata · Qwen 3.8 Flash
Fast agent
Open in calculatorFaster agent loop when you already have the daily driver proven.
- Model
- Qwen 3.8 Flash
- Qwen/Qwen3.8-Flash-Next
- Weights
- unsloth/Qwen3.8-Flash-Next-GGUF
- UD-Q6_K_XL
- Engine
- Strata v0.1.42 (llama.cpp Q6 fallback)
- GPUs
- Our test rig (R9700 class)
- Context / KV
- 262144 · f16 (llama.cpp fallback; Strata flags TODO)
- RAM offload
- Publisher file is ~169 GB — expect RAM to hold most of it on one 32 GB card
What we know / still TODO
- ~32.6 tok/s on Strata v0.1.42 vs ~16 tok/s on llama.cpp — Measured on our rig, 2026-10
- Publisher UD-Q6_K_XL folder ~169 GB (Hugging Face, six shards)
- TODO: exact command line / measured VRAM+RAM from rig owner.
Plug a harness into Strata on
127.0.0.1:8083/v1. William’s daily agent isHermes, pointed at this port and at llama-server.Connect snippets· All harnesses.Not the daily driver
vLLM serve try
Open in calculatorContinuous batching when several clients hit /v1. Consumers still start on llama.cpp.
- Model
- Same official Qwen you already trust on llama.cpp
- Pin whatever the weekly card names
- Weights
- HF weights, not GGUF
- Framework default — still never below Q4-class
- Engine
- Community / ROCm vLLM 0.31 try (0.30 previous; AMD 0.27 on ROCm 10.0 is the conservative pin)
- GPUs
- One GPU first
- Context / KV
- Confirm the engine, do not assume 262K · Engine default
- RAM offload
- Do not treat vLLM as a RAM-offload strategy
What we know / still TODO
- AMD-supported combo on the 10.0 matrix remains vLLM 0.27 + PyTorch 2.13 (gfx1201 listed)
- TODO: exact image tag / measured VRAM from rig owner if 0.31 is pinned.
Hardware for these stacks
Check price for the R9700 box
No live prices. As an Amazon Associate I earn from qualifying purchases.
Check price · no live or stale prices
AMD Radeon AI PRO R9700
Check price · no live or stale prices
High-capacity DDR5 kit
Search only — not a specific SKU. Our test rig is 256 GB DDR5; 64 GB is a reasonable one-card minimum.
Check price · no live or stale prices
Fast NVMe SSD
House rule — load GGUFs from NVMe, not a spinning disk. Search only, not a SKU.
Check price · no live or stale prices
1000W-class PSU
AMD’s R9700 minimum recommendation is 750 W. A 1000W-class unit is search guidance for headroom, not a SKU.
As an Amazon Associate I earn from qualifying purchases. Some Newegg links use Rakuten Advertising and some B&H links use B&H’s affiliate program when those IDs are configured. Micro Center and AMD Store links are plain “also available at” links — no tracking. Buttons say “Check price” and never show a live or stale price.Full disclosure.