Consumer Radeon · living notes
Local Qwen on an R9700. Reliably.
AMDai is the independent notebook for home-lab AI on Radeon. Which GGUF, which engine, whether it fits your VRAM — grounded in a two-card test rig, written for people who buy one GPU.
Our test rig is two AMD Radeon AI PRO R9700 cards
(gfx1201, 32 GB each) and 256 GB DDR5.
The daily driver uses one card.
What works this week
Snapshot 2026-10-12 · viable
Daily driver is Qwen3.6-35B-A3B (UD-Q5_K_XL) on one R9700 via llama.cpp Vulkan. Fast agent is Qwen 3.8 Flash on Strata.
Consumer north star: run local Qwen on Radeon, not a data-center brochure. Daily driver on our test rig — one Radeon AI PRO R9700 (32 GB, gfx1201), Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL, llama.cpp Vulkan, 262144 context, f16 KV, 0% system RAM offload. Measured on our rig, 2026-10: UD-Q5_K_XL 102.39 tok/s short / 90.52 at 32K / 66.43 at 128K / 44.4 at 256K; Q5_K_M 109.73 tok/s (24.68 GB VRAM); Q4_K_M 117.69 tok/s decode, 1902.1 tok/s prefill at 32K. Fast agents: Qwen 3.8 Flash Unsloth UD-Q6_K_XL on Strata v0.1.42 at 262K (Measured on our rig, 2026-10: ~32.6 tok/s vs llama.cpp ~16 tok/s); llama.cpp Q6 is the fallback. Prefer llama.cpp over vLLM at home. Prefer Q5 over Q4. Q3 is listed as an estimate or community result. Q1/Q2 stay off. Load GGUFs from fast NVMe. One LLM per GPU unless you deliberately combine. vLLM 0.31 is a serve try, not the daily path. Abliterated weights stay educational, never the default.
- ROCm
- AMD ROCm Core SDK 10.0.0 / current 10.1 matrix line — one RDNA4/gfx120x family package (R9700 = gfx1201; official support confirmed)
- Daily inference
- llama.cpp Vulkan on gfx1201 (HIP build is fine if it configures). Prefer that over official Ollama ROCm and over vLLM for a single-user home box.
- Known good (this week)
- One R9700 + llama.cpp Vulkan + Qwen3.6-35B-A3B UD-Q5_K_XL at 262K f16 KV (44.4 tok/s at 256K; Measured on our rig, 2026-10). Strata + Qwen 3.8 Flash UD-Q6_K_XL for agents. vLLM 0.31 = try, not the house default.
- AMD-supported combo
- ROCm 10.0 AI ecosystem matrix — vLLM 0.27 + PyTorch 2.13; gfx1201 listed. 10.1 matrix lists the same Radeon rows.
- Serving
- Consumer default is llama.cpp. Fast-agent try on our rig is Strata v0.1.42. Cutting-edge vLLM try is community/ROCm 0.31 (0.30 previous; AMD-validated 0.27/0.29 still the conservative pin on the 10.0 matrix).
- Qwen (reliability first)
- Qwen/Qwen3.6-35B-A3B · GGUF unsloth/Qwen3.6-35B-A3B-GGUF (file Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf; publisher listing 26.6 GB) Quant · UD-Q5_K_XL (Q5 family). Prefer Q5 over Q4. Q3_K_M / Q3_K_S / UD-Q3_K_XL / IQ3 are listed as estimates. Our measurements stay Q4 and above. Serve · llama.cpp Vulkan (or HIP gfx1201). Measured on our rig, 2026-10 — UD-Q5_K_XL 102.39 / 90.52 / 44.4 tok/s. Peak VRAM 25.97 GB at 4K / 29.34 GB at 256K (official llama-bench c060ca974).
This block is driven by src/data/what-works.yaml.
Cards: stacks
· calculator
· GPU directory.
Historical notes live under ROCm and the blog.
Latest from the lab
Newest findings
Result · Idea lab
CPU cooler for an R9700 build: Ryzen 9000X has none in the box, and a 360 AIO in the front or bottom of the case feeds warm air to the cards
AMD ships the Ryzen 5 9600X, Ryzen 9 9900X and Ryzen 9 9950X without a cooler; AMD recommends a premium air cooler for the 65 W 9600X and a liquid cooler for the 120 W 9900X and 170 W 9950X for best boost. for a local-AI box with blower R9700s, where the radiator goes matters: the cards pull their air from the case's front and bottom intake, so a 360 radiator should go on top as exhaust, and a case that only takes one in front or bottom is better paired with a large dual-tower air cooler.
Result · Idea lab
Case for two R9700s: a standard 7-slot ATX case is enough on 4-slot-spacing x8/x8 boards, and case fans should follow the GPUs, not the CPU
Every R9700 board-partner card is a 2-slot blower no longer than 271 mm. on the ASRock X870 Taichi Creator and Gigabyte B850 AI TOP, owners report 4-slot spacing between the two CPU x8/x8 slots, so the bottom card ends by expansion slot 7 and leaves a 2-slot gap above it. that means any 7-slot ATX case with at least 271 mm of card length fits two cards; no 8-slot case is needed unless you plan a third card or a riser. blowers push their heat out the back, so the case's job is fresh intake air for the card area.
Result · Idea lab
How much NVMe a Radeon local-AI box needs: 2 TB holds every Q4-Q6 quant of three model families, and Gen5 barely speeds up loading
Every Q4 to Q6 quant of Qwen3.6-35B-A3B (20.6 to 29.7 GiB), Qwen3.8-Flash-Next (103.7 to 157.6 GiB) and GLM-5.3-Flash (186.0 to 271.8 GiB) adds up to about 1,587 GiB once you keep one extra file of room for the next download and 150 GiB for the OS. That fits a 2 TB drive (1,863 GiB) with 10% left free; a fourth Flash-class family such as MiniMax-M3.1-Flash (UD-Q4 232 GiB) needs 4 TB. On llama.cpp's default memory-mapped loader, loading is usually limited by page faults and one CPU core rather than the drive, so a Gen5 SSD helps less than its doubled rating suggests.
From the rig
Tested stacks, measured where we have numbers
Daily driver
Qwen3.6-35B-A3B · UD-Q5_K_XL
llama.cpp Vulkan on one R9700. 262K context, f16 KV, 0% RAM offload. 102.39 tok/s short context, 90.52 at 32K, 66.43 at 128K, 44.4 at 256K — Measured on our rig, 2026-10.
Measured on our rig, 2026-10 102.39 / 90.52 / 44.4 tok/s
Fast agent
Qwen 3.8 Flash · UD-Q6_K_XL
Two-card Q6 at 262K: Strata 32.8 tok/s at 8K/32K, 31.8 at 100K. Prior first-party ~32.6 vs llama.cpp ~16. DDR5 speed is a visible factor.
Measured on our rig, 2026-10 32.8 / 31.8 tok/s
Two cards
35B · two copies beat a split
UD-Q5_K_XL on llama.cpp Vulkan, DDR5-5200. Two single-card copies: about 100 tok/s per request, 172 total at 2 concurrent. A split server tops out at 79.
Measured on our rig, 2026-10 0 errors in 423
First-party
Benchmarks
Overnight 35B llama.cpp matrix from official run cards — UD-Q5 / Q5 / Q4 / Q6 at 4K–256K, one GPU and split. Flash Strata vs llama.cpp stays its own cited block. Empty cells stay not measured.
Measured on our rig, 2026-10 No leaderboard spam
Also in the lab
Compare, jargon, and the rest of the stack
Decisions
Compare 35B
One GPU vs split, and Q6 / Q5 / Q4 plus a labeled Q3 estimate — from official cards.
Words
Glossary
ROCm, GGUF, Q3–Q6, KV cache, run cards — plain English for a home Radeon box.
Comparable
Official run cards
amdai-bench/run@1 cards credited to AMDai on the local desktop. Exact launch command.
The stack
What is ROCm
Plain-English deep dive: the stack, gfx IDs, Vulkan vs HIP, versions.
When it breaks
Troubleshooter
Start from a symptom, or paste a log. Cited ROCm / llama.cpp / Ollama patterns.
Hardware
GPU directory
Radeon cards, gfx IDs, VRAM, official ROCm rows. No invented support claims.
Pictures
Image gen on Radeon
Cited ComfyUI-first SD / SDXL / Flux-class guide. Linux first. Speeds not yet measured by AMDai.
See a picture
Vision on Radeon
Cited llama.cpp mmproj path. Linux first. Speeds not yet measured by AMDai. Q3 is a labeled estimate.
Speech
Speech on Radeon
Cited whisper.cpp Vulkan STT plus Piper TTS. Linux first. Speeds not yet measured by AMDai.
Many users
Serving on 1–2 cards
llama.cpp -np and --split-mode first. Third-party vLLM and Radiance figures labeled. Our lab 35B and Flash.
Context
Long context on Radeon
128K–262K from local desktop 35B llama.cpp cards. f16 KV default. Empty cells stay not measured yet.
Your files
RAG on Radeon
Cited llama.cpp-first local RAG for PDFs and notes. Linux first. Speeds not yet measured by AMDai.
Consumer house rules
How we pick a stack
- 01 Prefer llama.cpp over vLLM for a home box. vLLM is a serve try, not the daily driver.
- 02 Prefer Q5 over Q4. Q3_K_M / Q3_K_S / UD-Q3_K_XL / IQ3 are listed as estimates unless measured. Never Q1, Q2, IQ1, or IQ2 — including draft / MTP heads.
- 03 Keep GGUFs on fast NVMe. Do not run a 20-plus-GB file from a spinning disk.
- 04 One LLM per GPU unless you deliberately combine (tensor/layer split).
- 05 Daily driver on our rig uses 0% system RAM offload. Offload is slow — treat it as a last resort.
Tutorials
Do the work
-
Install ROCm on a home Linux box
ROCm 10.1.0 on a Radeon AI PRO R9700 — official Ubuntu 26.04.1 / 24.04.5 / 22.04.5 rows, ROCm CLI v0.2.0, and a do-not-mix-apt decision tree.
-
Verify GPU and ROCm
Prove the R9700 is visible, that HIP works, and that PyTorch is using the card — before you download a 20 GB model.
-
Run a local Qwen (llama.cpp)
Build llama.cpp first — Vulkan is the daily-driver path on our rig, HIP if rocminfo is clean — then run official Qwen instruct.
-
Serve Qwen on the R9700
Official Qwen instruct on a loopback /v1 — llama-server first, Strata for Flash, vLLM only in Advanced.
Dated notes
ROCm / vLLM snapshots
-
Oct 12, 2026
Consumer stacks — Qwen3.6 daily driver, Flash agent, calculatorNorth star is now the two stacks from our test rig. llama.cpp Vulkan daily driver; Strata for Flash; /calculator and /stacks are first-class pages.
-
Oct 12, 2026
Overnight sweep, 2026-10-1233 official amdai-bench/run@1 cards from local desktop (AMDai) for Qwen3.6-35B-A3B on one R9700 and both cards. Measured, pending re-verify after the DDR5-5200 RAM fix.
-
Oct 12, 2026
ROCm 10.1.0 notes, CLI v0.2.0, error paste, version matrix10.1.0 lists R9700 as gfx1201 and Radeon Ubuntu 26.04.1 / 24.04.5 / 22.04.5. Published ROCm CLI is v0.2.0, not a 1.0.0 tag. New pages for search, errors, backends, quants, RSS.