Serve Qwen on the R9700

Official Qwen instruct on a loopback /v1 — llama-server first, Strata for Flash, vLLM only in Advanced.

Reliability first: Qwen3.6-35B-A3B UD-Q5_K_XL, one GPU, a loopback /v1 you can curl. Harnesses and abliterated weights come after. Stack Lab calls this qwen-reliable.

Reliability first on llama.cpp, then a faster agent engine if you need it. Prefer Q5 over Q4. Never go below Q4. Load weights from fast NVMe. One LLM per GPU unless you deliberately combine. Abliterated is educational only.

Snapshot 2026-10-12. Open the stack cards or the calculator.

daily driver

Qwen3.6-35B-A3B · UD-Q5_K_XL

Official Qwen/Qwen3.6-35B-A3B (35B total, 3B active, 262144 native context). Unsloth GGUF UD-Q5_K_XL is the file we run every day on ONE Radeon AI PRO R9700 (32 GB, gfx1201) via llama.cpp Vulkan, f16 KV, no system RAM offload.

Serve · llama.cpp Vulkan (HIP gfx1201 is fine if it builds). TODO: exact command line / measured VRAM from rig owner.

32 GB · Publisher listing for Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf is 26.6 GB (Hugging Face). Measured VRAM + RAM on our rig is TODO. Calculator estimate — not a bench.

Weights from unsloth/Qwen3.6-35B-A3B-GGUF. Keep the file on NVMe. See /stacks and /calculator. Do not drop below Q5 unless the calculator says Q5 cannot fit.

fast agent

Qwen 3.8 Flash · UD-Q6_K_XL

Official Qwen3.8-Flash-Next (125B total, 6B active on the card). Unsloth UD-Q6_K_XL served by Strata v0.1.42 at 262K on our test rig. Measured ~32.6 tok/s versus llama.cpp ~16 tok/s on that same box. llama.cpp Q6 is the fallback if you are not on Strata.

Serve · Strata v0.1.42 first; llama.cpp Q6 fallback. TODO: exact command line / measured VRAM+RAM from rig owner.

32 GB · Publisher UD-Q6_K_XL folder lists ~169 GB (Hugging Face, six shards). That will not sit in 32 GB VRAM — expect system RAM to hold most of the file. Measured split / VRAM / RAM on our rig is TODO. Offload is slow.

Weights from unsloth/Qwen3.8-Flash-Next-GGUF. Fast NVMe is not optional at this file size. Still never below Q4. Open /calculator with the Flash preset before you buy a second card.

optional educational

Huihui-Qwen3.8-27B-abliterated

Educational refusal-removal study of Qwen3.8-27B. Not a default. Not a Stack Lab profile. Prove the official daily driver first.

Serve · llama.cpp via a published GGUF — confirm the exact file. Not the reliability path.

32 GB · 27B-class. Confirm the GGUF size + KV. Do not invent a fit.

See /abliterated. Pin a revision. Experimental publisher framing.

Verify this week — do not trust a frozen pin

From src/data/qwen-recommendations.yaml. A stack that passed last month still has to pass these on your box.

1. Completes. Prompt: “Reply with the single word READY and nothing else.” Expect: Short completion, GPU used.
2. Template. Prompt: “A simple multi-turn follow-up (ask a fact, then "shorter").” Expect: No leaked special tokens, no infinite think loop.
3. Qwen3 think switch. Prompt: “Same fact question, then a /no_think follow-up if the template supports it.” Expect: Thinking block only when enabled; no hang when disabled.
4. Tools if needed. Prompt: “A one-tool dummy call through your harness.” Expect: Tool JSON your server actually executes.

tok/s is a sanity check (CPU crawl vs GPU), not a spec sheet. If VRAM never moves, you failed before the model did.

llama.cpp first. You already built it in Run a local Qwen. vLLM is not the next step — it is Advanced. Backend map: chooser.

Path A — llama-server (least pain)

Daily driver: Unsloth UD-Q5_K_XL (Q5 family — do not drop to Q3 to “make it fit”). Vulkan binary (build-vk) is what we run; HIP (build) is fine if that is the tree you compiled.

TODO: exact command line / measured VRAM from rig owner.

./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

--jinja matters on current Qwen templates.

watch -n1 rocm-smi
curl -s http://127.0.0.1:8080/v1/models
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen","messages":[{"role":"user","content":"Reply with the single word READY and nothing else."}],"max_tokens":8}'

If VRAM never moves, stop. Point Hermes or OpenClaw at http://127.0.0.1:8080/v1 only after this returns READY.

Path B — Strata (fast agent on our rig)

When the daily driver is boring and you want the Flash stack: Qwen 3.8 Flash Unsloth UD-Q6_K_XL on Strata v0.1.42 at 262K. Measured on our test rig ~32.6 tok/s versus llama.cpp ~16 tok/s. llama.cpp Q6 is the fallback.

Publisher UD-Q6_K_XL listing is ~169 GB — the calculator will show RAM overflow on one 32 GB card. TODO: exact flags / measured VRAM+RAM from rig owner.

Cards: stacks.

Advanced — vLLM (not the consumer default)

Two pins. Do not mix them in one venv. The weekly card is the current recommendation.

Reliability (qwen-reliable) is llama.cpp, not vLLM. AMD-aligned vLLM 0.27 / ROCm 10 remains the conservative supported combo (docs snapshot: vLLM 0.27 + PyTorch 2.13). Copy the current image tag from AMD’s vLLM page if you serve HF weights.

Community / ROCm Docker — vLLM 0.31 try

Engine try-tag only. Previous try-tag v0.30.0 is still valid if 0.31 misbehaves. Do not replace the daily-driver GGUF path with this.

# Community ROCm image — NOT default PyPI
# TODO: confirm the current 0.31-class tag on Docker Hub / the weekly card
docker pull vllm/vllm-openai-rocm:v0.30.0
# previous: vllm/vllm-openai-rocm:v0.29.0
# previous wheels: pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.29.0/rocm723

# Image tags change. Confirm against AMD or vLLM / Docker Hub.
docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add video --ipc=host --network host \
  -v "$HOME/models:/models" \
  vllm/vllm-openai-rocm:v0.30.0 \
  --model /models/YOUR-QWEN-HF \
  --host 127.0.0.1 --port 8000

ROCm CLI v0.2.0 documents rocm install vllm for 10.1 wheels — that is an install claim, not a first-party bench. Version matrix.

curl -s http://127.0.0.1:8000/v1/models

OpenClaw’s vLLM provider wants http://127.0.0.1:8000/v1 and VLLM_API_KEY set (dummy is fine if the server does not auth).

Golden checks

Short READY completion.
A second turn that stays in-template.
rocm-smi during both.
Then: harness tools. Then, if you still want it, a second weight (coder / abliterated).

If thinking never ends, stay on official instruct with thinking off, or fix --jinja. Do not “fix” it by grabbing a random uncensored GGUF.