The model files are not the runtime. On a R9700 you choose a server that speaks HIP or Vulkan, then optionally a harness that speaks HTTP to that server.
Reliability first on llama.cpp, then a faster agent engine if you need it. Prefer Q5 over Q4. Never go below Q4. Load weights from fast NVMe. One LLM per GPU unless you deliberately combine. Abliterated is educational only.
Snapshot 2026-10-12. Open the stack cards or the calculator.
daily driver
Qwen3.6-35B-A3B · UD-Q5_K_XL
Official Qwen/Qwen3.6-35B-A3B (35B total, 3B active, 262144 native context). Unsloth GGUF UD-Q5_K_XL is the file we run every day on ONE Radeon AI PRO R9700 (32 GB, gfx1201) via llama.cpp Vulkan, f16 KV, no system RAM offload.
Serve · llama.cpp Vulkan (HIP gfx1201 is fine if it builds). TODO: exact command line / measured VRAM from rig owner.
32 GB · Publisher listing for Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf is 26.6 GB (Hugging Face). Measured VRAM + RAM on our rig is TODO. Calculator estimate — not a bench.
Weights from unsloth/Qwen3.6-35B-A3B-GGUF. Keep the file on NVMe. See /stacks and /calculator. Do not drop below Q5 unless the calculator says Q5 cannot fit.
fast agent
Qwen 3.8 Flash · UD-Q6_K_XL
Official Qwen3.8-Flash-Next (125B total, 6B active on the card). Unsloth UD-Q6_K_XL served by Strata v0.1.42 at 262K on our test rig. Measured ~32.6 tok/s versus llama.cpp ~16 tok/s on that same box. llama.cpp Q6 is the fallback if you are not on Strata.
Serve · Strata v0.1.42 first; llama.cpp Q6 fallback. TODO: exact command line / measured VRAM+RAM from rig owner.
32 GB · Publisher UD-Q6_K_XL folder lists ~169 GB (Hugging Face, six shards). That will not sit in 32 GB VRAM — expect system RAM to hold most of the file. Measured split / VRAM / RAM on our rig is TODO. Offload is slow.
Weights from unsloth/Qwen3.8-Flash-Next-GGUF. Fast NVMe is not optional at this file size. Still never below Q4. Open /calculator with the Flash preset before you buy a second card.
optional educational
Huihui-Qwen3.8-27B-abliterated
Educational refusal-removal study of Qwen3.8-27B. Not a default. Not a Stack Lab profile. Prove the official daily driver first.
Serve · llama.cpp via a published GGUF — confirm the exact file. Not the reliability path.
32 GB · 27B-class. Confirm the GGUF size + KV. Do not invent a fit.
See /abliterated. Pin a revision. Experimental publisher framing.
Consumer house rules
How we pick a stack
- 01Prefer llama.cpp over vLLM for a home box. vLLM is a serve try, not the daily driver.
- 02Prefer Q5 over Q4. Q4_K_M is the floor. Never Q1, Q2, Q3, or IQ1–IQ3 — including draft / MTP heads.
- 03Keep GGUFs on fast NVMe. Do not run a 20-plus-GB file from a spinning disk.
- 04One LLM per GPU unless you deliberately combine (tensor/layer split).
- 05Daily driver on our rig uses 0% system RAM offload. Offload is slow — treat it as a last resort.
Verify this week — do not trust a frozen pin
From src/data/qwen-recommendations.yaml. A stack that passed last month still has to pass these on your box.
- 1. Completes. Prompt: “Reply with the single word READY and nothing else.” Expect: Short completion, GPU used.
- 2. Template. Prompt: “A simple multi-turn follow-up (ask a fact, then "shorter").” Expect: No leaked special tokens, no infinite think loop.
- 3. Qwen3 think switch. Prompt: “Same fact question, then a /no_think follow-up if the template supports it.” Expect: Thinking block only when enabled; no hang when disabled.
- 4. Tools if needed. Prompt: “A one-tool dummy call through your harness.” Expect: Tool JSON your server actually executes.
tok/s is a sanity check (CPU crawl vs GPU), not a spec sheet. If VRAM never moves, you failed before the model did.
Quick compare
| Stack | Best at | Pain on Radeon | Pick when |
|---|---|---|---|
| llama.cpp | One user, GGUF, tight control | You compile HIP (gfx1201) or use Vulkan |
First Qwen, debugging, agents that just need /v1 |
| Ollama | Pull + API with almost no flags | Bundled ROCm can miss gfx1201; Windows is worse |
You want convenience and rocm-smi proves GPU use |
| vLLM | Continuous batching, OpenAI server, spec decode | Version/wheel hell; multi-GPU RCCL | Several clients or a harness that needs a real server |
| HF TGI | Production-shaped HF serving | Official ROCm image is tested on Instinct, not promised on Radeon | You already live in TGI and will validate on this card |
| PyTorch + HF | Fine-tune, custom pipelines | Need the gfx1201 ROCm wheel | Training loops, not your daily chat |
llama.cpp
C/C++ GGUF runner. Backends that matter here: HIP/ROCm (-DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201) and Vulkan. Vulkan is a first-class on-ramp, not a consolation prize — see the blog note.
Use it for the first local Qwen. llama-server gives harnesses an OpenAI-shaped loopback.
Ollama
Same job as llama.cpp for many people, with model tags and a daemon. On Linux, treat GPU attach as unproven until rocm-smi moves. If it sits on CPU, upgrade, read the backend log, or go back to llama.cpp.
vLLM
Throughput-oriented serving. Two different “official” stories:
- AMD’s ROCm 10.0 docs (dated snapshot): vLLM 0.27 with PyTorch 2.13, often via
rocm/vllm:…images. That is the supported combo and this week’s reliability pin. - Community ROCm Docker (22 Sep 2026):
docker pull vllm/vllm-openai-rocm:v0.30.0. Previous try-tagv0.29.0/ wheelshttps://wheels.vllm.ai/rocm/0.29.0/rocm723.pip install vllmwith no extra index is CUDA.
Single-GPU Docker, pinned, is the honest “production-viable” home path. Two consumer cards is a different project (RCCL, topology). Serve Qwen stays on one GPU.
Hugging Face TGI
TGI on AMD publishes a ROCm Docker image and says it is supported and tested on Instinct MI210 / MI250 / MI300. Radeon may work; it is not the tested row. AWQ load is called out as unsupported on that ROCm build. Prefer vLLM or llama.cpp unless TGI is already your shop standard.
PyTorch / Transformers
Install from AMD’s PyTorch for ROCm with a gfx1201 (or documented gfx12) selector. Confirm torch.version.hip before from_pretrained. Fine-tune on Radeon is possible and still a worse cookbook than CUDA — say that out loud before you plan a LoRA weekend.
Workload honesty
| Setup | Honest take |
|---|---|
| llama.cpp or Ollama, one card | Least pain. First chat, first harness. |
| vLLM, one GPU, Docker/wheel pinned | Production-viable OpenAI server (batching, spec decode). |
| Two+ consumer / PRO cards | RCCL / atomics tax. Measure before you buy speed. |
| Fine-tune on Radeon | Possible. Still a thinner cookbook than CUDA. |
Hardware limits: Hardware & power.