Path

Getting started

One end-to-end path for a home AMD box — pick a GPU, install ROCm, run the daily-driver Qwen, optionally add a harness.

Updated Oct 12, 2026

Do this in order. Skip a layer and you will debug the wrong one.

1. Pick the GPU

  • 32 GB local models: AMD Radeon AI PRO R9700 (gfx1201). Official name — not an RX 9700.
  • 16 GB is enough: Radeon RX 9070 XT (also gfx1201 on current matrices). Same generation, half the VRAM.
  • Already own a 7900 XTX: still a practical home card (gfx1100). Different ROCm family package.

Read the GPU directory and Hardware before you buy a PSU. Estimate a fit on the calculator.

2. Install ROCm on Linux

Practical Ubuntu path (as of the weekly card):

  1. Use an Ubuntu LTS named on AMD’s current matrix. ROCm 10.1.0 Radeon rows: 26.04.1, 24.04.5, 22.04.5.
  2. Add AMD’s 10.1 repo — or use ROCm CLI v0.2.0 (rocm install sdk). Do not mix those with Ubuntu archive rocm.
  3. Install one gfx family (gfx1201 on the R9700 — confirm with apt-cache search). Prefix: ls /opt/rocm (10.0 was often /opt/rocm/core-10.0).
  4. Do not apt install rocm from Canonical’s archive.

Walkthrough: Install ROCm. Prove it: Verify GPU and ROCm.

Before you rebuild the world: check firmware / MES after a kernel upgrade, and try Vulkan llama.cpp if HIP configure is a mess.

3. Pick inference software

Consumer default on this site:

  1. llama.cpp (Vulkan, or HIP gfx1201) — daily driver. Prefer that over official Ollama ROCm and over vLLM for a single-user box.
  2. Strata v0.1.42 — fast-agent try on our rig for Qwen 3.8 Flash.
  3. vLLM — only when you need a multi-client server. Try this week: community / ROCm 0.31. Conservative AMD pin on the 10.0 matrix is still 0.27. Default pip install vllm is CUDA.

Compare them on inference engines and AI software. Run the daily driver: Local LLM tutorial then Serve Qwen. Cards: Recommended stacks.

Consumer house rules

How we pick a stack

  1. 01Prefer llama.cpp over vLLM for a home box. vLLM is a serve try, not the daily driver.
  2. 02Prefer Q5 over Q4. Q4_K_M is the floor. Never Q1, Q2, Q3, or IQ1–IQ3 — including draft / MTP heads.
  3. 03Keep GGUFs on fast NVMe. Do not run a 20-plus-GB file from a spinning disk.
  4. 04One LLM per GPU unless you deliberately combine (tensor/layer split).
  5. 05Daily driver on our rig uses 0% system RAM offload. Offload is slow — treat it as a last resort.

4. Optional harness

A harness is the agent loop above the model. It does not replace ROCm.

  • Hermes Agent — used on our rig (wtpai1), pointed at Strata :8083 and llama-server.
  • OpenClaw — self-hosted gateway into Discord/Telegram/etc.
  • Chat UIs, IDE agents, and ComfyUI on ROCm: harnesses catalog.

curl /v1 first. Then point the harness at 127.0.0.1. Snippets: connect.

5. Qwen

Default: Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL on one R9700, llama.cpp Vulkan, 262144 context, f16 KV, no RAM offload. Measured on our rig, 2026-10: 100.8 tok/s short context, ~88 at 32K, ~50 at 256K. Fast agents: Qwen 3.8 Flash UD-Q6_K_XL on Strata. Abliterated weights come after that pipe is boring.

TODO: exact command line / UD-Q5_K_XL VRAM from rig owner.

Reliability first on llama.cpp, then a faster agent engine if you need it. Prefer Q5 over Q4. Never go below Q4. Load weights from fast NVMe. One LLM per GPU unless you deliberately combine. Abliterated is educational only.

Snapshot 2026-10-12. Open the stack cards or the calculator.

daily driver

Qwen3.6-35B-A3B · UD-Q5_K_XL

Official Qwen/Qwen3.6-35B-A3B (35B total, 3B active, 262144 native context). Unsloth GGUF UD-Q5_K_XL is the file we run every day on ONE Radeon AI PRO R9700 (32 GB, gfx1201) via llama.cpp Vulkan, f16 KV, no system RAM offload.

Serve · llama.cpp Vulkan (HIP gfx1201 is fine if it builds). TODO: exact command line / measured VRAM from rig owner.

32 GB · Publisher listing for Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf is 26.6 GB (Hugging Face). Measured VRAM + RAM on our rig is TODO. Calculator estimate — not a bench.

Weights from unsloth/Qwen3.6-35B-A3B-GGUF. Keep the file on NVMe. See /stacks and /calculator. Do not drop below Q5 unless the calculator says Q5 cannot fit.

fast agent

Qwen 3.8 Flash · UD-Q6_K_XL

Official Qwen3.8-Flash-Next (125B total, 6B active on the card). Unsloth UD-Q6_K_XL served by Strata v0.1.42 at 262K on our test rig. Measured ~32.6 tok/s versus llama.cpp ~16 tok/s on that same box. llama.cpp Q6 is the fallback if you are not on Strata.

Serve · Strata v0.1.42 first; llama.cpp Q6 fallback. TODO: exact command line / measured VRAM+RAM from rig owner.

32 GB · Publisher UD-Q6_K_XL folder lists ~169 GB (Hugging Face, six shards). That will not sit in 32 GB VRAM — expect system RAM to hold most of the file. Measured split / VRAM / RAM on our rig is TODO. Offload is slow.

Weights from unsloth/Qwen3.8-Flash-Next-GGUF. Fast NVMe is not optional at this file size. Still never below Q4. Open /calculator with the Flash preset before you buy a second card.

optional educational

Huihui-Qwen3.8-27B-abliterated

Educational refusal-removal study of Qwen3.8-27B. Not a default. Not a Stack Lab profile. Prove the official daily driver first.

Serve · llama.cpp via a published GGUF — confirm the exact file. Not the reliability path.

32 GB · 27B-class. Confirm the GGUF size + KV. Do not invent a fit.

See /abliterated. Pin a revision. Experimental publisher framing.

Verify this week — do not trust a frozen pin

From src/data/qwen-recommendations.yaml. A stack that passed last month still has to pass these on your box.

  1. 1. Completes. Prompt: “Reply with the single word READY and nothing else.” Expect: Short completion, GPU used.
  2. 2. Template. Prompt: “A simple multi-turn follow-up (ask a fact, then "shorter").” Expect: No leaked special tokens, no infinite think loop.
  3. 3. Qwen3 think switch. Prompt: “Same fact question, then a /no_think follow-up if the template supports it.” Expect: Thinking block only when enabled; no hang when disabled.
  4. 4. Tools if needed. Prompt: “A one-tool dummy call through your harness.” Expect: Tool JSON your server actually executes.

tok/s is a sanity check (CPU crawl vs GPU), not a spec sheet. If VRAM never moves, you failed before the model did.

Stop conditions

  • rocminfo has no gfx1201 → do not download 20 GB.
  • rocm-smi idle while you “generate” → you are on CPU.
  • File is on a spinning disk → move it to NVMe before you tune flags.
  • Harness is slow → check the backend, not the card’s reputation.
esc

Type to filter. Enter opens the first match. Full-text search lives on /search.