Inference engines compared

The model files are not the runtime. On a Radeon you pick anengine that speaks Vulkan or HIP, then optionally aharness that POSTs to/v1. Consumer default is llama.cpp. As of 2026-10-12.

Consumer default is llama.cpp. Vulkan is what we run for the 35B daily driver.
First-party 35B (one R9700, llama.cpp, f16 KV). Measured on our rig, 2026-10: UD-Q5_K_XL 100.8 tok/s short / ~88 at 32K / ~50 at 256K; Q5_K_M 108.3 tok/s (28.5 GB VRAM); Q4_K_M 116.5 tok/s decode, 1814 tok/s prefill at 32K.
Never go below Q4 (no Q1/Q2/Q3 or IQ1–IQ3, including draft/MTP).
curl /v1 before you debug a harness. Do not let a desktop app download a CUDA llama.cpp on a Radeon.

llama.cpp deep dive Benchmarks Calculator What is ROCm Paste a log

wtpai1

Used on our rig

llama.cppUsed on our rig
Consumer default — GGUF, Vulkan or HIP, one user
StrataUsed on our rig
Fast-agent try on our rig — Flash Q6

Which one should I use?

The diagram needs JavaScript. The list under it is the same advice and reads without it.

Default is llama.cpp

flowchart TD
  start["Need tokens on a Radeon tonight?"] --> llama["llama.cpp\nVulkan 35B: 100.8 / ~88 / ~50 tok/s\nMeasured on our rig, 2026-10"]
  llama --> proven{"Daily driver already\nworks on /v1?"}
  proven -->|"No"| stay["Stay on llama.cpp\ncurl /v1, then a harness"]
  proven -->|"Yes, want Flash agent"| strata["Strata v0.1.42\n~32.6 tok/s vs llama.cpp ~16\nfirst-party Flash Q6"]
  proven -->|"Several clients / batching"| vllm["vLLM ROCm\n0.31 try — not default PyPI"]
  proven -->|"Want a GUI"| gui{"GPU meter moves?"}
  gui -->|"Yes"| lm["LM Studio or Ollama\nstill a black box"]
  gui -->|"No"| stay
  vllm --> pin["Pin a ROCm image/wheel\nAMD 10.0 combo is still 0.27"]
  strata --> hermes["Point Hermes at :8083"]

If the diagram does not draw, the layers still read top to bottom in the list on this page.

1. Need tokens on one card tonight → llama.cpp (Vulkan if HIP CMake sulks).
2. Daily driver already works and you want the Flash agent → Strata v0.1.42.
3. Several clients / continuous batching → vLLM ROCm, advanced. Not default PyPI.
4. Want a GUI → LM Studio only if a GPU meter moves. Else back to llama.cpp.
5. Ollama → only if the backend log and a GPU meter agree. gfx1201 is still cited pain.
6. SGLang, ExLlama/TabbyAPI, MLC-LLM, KTransformers, raw PyTorch → specialized. Not the first home box.

Measured on our rig, 2026-10

35B on llama.cpp, measured

Qwen3.6-35B-A3B on one R9700, llama.cpp, f16 KV. Cells that we still do not have stay not measured. Units match the hardware labels (GB). Full hub: benchmarks.

Quant	Decode · short	Decode · 32K	Decode · 256K	Prefill · 32K	VRAM
UD-Q5_K_XL	100.8 tok/s	~88 tok/s	~50 tok/s	not measured	not measured
Q5_K_M	108.3 tok/s	not measured	not measured	not measured	28.5 GB
Q4_K_M	116.5 tok/s	not measured	not measured	1814 tok/s	not measured

One R9700, llama.cpp, f16 KV. Exact command lines stay TODO.
UD-Q5_K_XL VRAM is not measured. Q5_K_M VRAM is 28.5 GB.
Prefill is published only for Q4_K_M at 32K.

Comparison

Cells are paraphrases of cited docs plus this site’s house rules. “Not documented” and TODO stay labeled. First-party speed is the 35B llama.cpp set (Measured on our rig, 2026-10) plus the Flash Q6 pair.

Engine	Radeon / ROCm	Vulkan	Quants	Multi-GPU	CPU / RAM offload	MoE	Practical context	Concurrency	API	Setup	Best for	Worst for
llama.cpp Start hereOn our rig	First-class if you build it. HIP needs a gfx1201 userspace. Vulkan needs a working ICD (RADV is the usual Linux path).	Yes — GGML_VULKAN. Daily driver on our rig.	GGUF Q4–Q8 and UD dynamic. Floor is Q4. Never Q1/Q2/Q3 or IQ1–IQ3.	Possible (split / tensor). We run the daily driver on ONE R9700. Do not assume a split.	-ngl / --n-gpu-layers. Daily driver is 0% system RAM offload.	MoE GGUF (Qwen3.6-35B-A3B is A3B). Measured on our rig, 2026-10 — one R9700, llama.cpp, f16 KV: UD-Q5_K_XL 100.8 tok/s short / ~88 at 32K / ~50 at 256K; Q5_K_M 108.3 tok/s (28.5 GB VRAM); Q4_K_M 116.5 tok/s decode, 1814 tok/s prefill at 32K.	Our daily driver is 262144, f16 KV — confirmed on the rig; ~50 tok/s decode at 256K (Measured on our rig, 2026-10). Exact flags TODO.	One interactive user is the honest home path. llama-server can take more clients; it is not vLLM continuous batching.	OpenAI /v1 (chat/completions, models). llama.cpp also documents Anthropic /v1/messages and OpenAI /v1/responses.	Medium — you compile. Vulkan CMake is the on-ramp.	First Qwen, debugging, a harness that just needs /v1.	A shop that wants continuous batching on day one.
Strata Measured tryOn our rig	HIP download on the v0.1.42 notes. Official notes name RX 7900 / 9070 / AI PRO R9700. gfx12 prompt-path switches default on for gfx1200/gfx1201.	Not documented as a Vulkan engine. Do not treat it as “Vulkan won.”	Publisher Flash file we run is Unsloth UD-Q6_K_XL. Floor is still Q4. Do not use their IQ2/IQ3 examples as a house quant.	Layer split is documented. 0.1.42 notes a two-R9700 checkpoint-save fix. We do not assume a split on the daily driver.	Expert cache / RAM / SSD tiers are the point of the product. Publisher UD-Q6_K_XL folder ~169 GB — expect RAM to hold most of it on one 32 GB card.	First-class. Flash-Next is a large MoE; routing and expert cache are the architecture.	We ran 262144. Exact --max-context / KV flags TODO.	Docs default to one request at a time; --parallel N is documented. Not measured here.	OpenAI /v1 plus Anthropic Messages. Responses support is documented upstream; we have not measured Codex against Strata (TODO).	Medium-hard — follow their setup.sh / update.sh. Exact flags for our rig stay TODO.	Qwen 3.8 Flash UD-Q6_K_XL at 262K when you want the faster of the two stacks we ran.	Being the first install; a box with no proven llama.cpp /v1.
vLLM Advanced / experimental	vLLM’s GPU page lists Radeon RX 9000 (gfx1200/1201) among supported devices. That is not the same as AMD’s 10.0 supported-combo pin (0.27 + PyTorch 2.13). Treat 0.31 as a try on this site.	No. HIP/ROCm only.	Engine/framework quants (FP8, AWQ, GPTQ, …) plus HF weights — still never below Q4-class. GGUF is not the house path here.	Yes, RCCL. Two consumer cards is a different project. Start on one GPU.	Not a RAM-offload strategy. Do not treat vLLM as llama.cpp -ngl.	Supported on some pins (AITER MoE paths). Completeness on gfx1201 Qwen MoE is not measured here.	Confirm the engine. Do not assume 262K.	This is the reason to be here. Continuous batching is the product.	OpenAI /v1 (vllm serve). Default port 8000.	Hard — wheel/image hell. Default pip install vllm is CUDA.	Several clients or a harness that needs a real server, after llama.cpp already works.	Tonight’s first Qwen.
Ollama Use with eyes open	Official gpu.mdx lists gfx1201 (example GPU RX 9070 XT). Cited pain remains: #12908 / #11555 hipBLASLt TensileLibrary_lazy_gfx1201.dat in the wrong directory; Windows ROCm lagged Linux (6.2 vs 7.2 in #14391). Listing gfx1201 ≠ “works on my R9700 tonight.”	Optional / emerging in their roadmap comments. Prefer a Vulkan llama.cpp you built if ROCm attach fails.	Their library tags. Floor is still Q4. Do not pull a Q2 “to make it fit.”	Not the house path. One card first.	Their scheduler. If VRAM never moves, you are on CPU.	Whatever the bundled runner implements. Not measured here.	Tag-dependent. Do not assume 262K.	Daemon can take several clients. Not vLLM batching.	Native /api plus /v1 on :11434. Do not append /v1 if you want some tool-call paths (OpenClaw notes).	Easy install, hard debug when gfx1201 tensiles miss.	Pull-and-run after rocm-smi proves GPU use.	First gfx1201 box; Windows ROCm.
LM Studio Use with eyes open	AMD’s LM Studio playbook says the app offers Vulkan and AMD ROCm runtimes. Older 0.2.31 “extension pack” scripts are version-specific — do not run them on a current build. gfx1201 completeness of the bundled ROCm engine is not documented as our daily driver.	Yes — the honest cross-vendor path. Prefer it over a mystery ROCm blob until the meter moves.	GGUF in the app library. Floor is Q4.	Not documented as our path.	In-app GPU offload slider. We have no measured VRAM.	Depends on the bundled llama.cpp. Not measured.	Set in the app. Do not assume 262K.	Local server can take clients. Not vLLM.	OpenAI-compatible on :1234, including Responses (LM Studio documents Codex that way).	Easy GUI, opaque engine.	A GUI when you accept a black box and will watch a GPU meter.	Matching this site’s compile flags or benches.
SGLang Advanced / experimental	AMD lists R9700 / RX 9070 XT on the ROCm 10.0 SGLang page. Known issue: default AITER attention may fail on Radeon PRO W7900, AI PRO R9700, and RX 9070 XT. Workaround: SGLANG_USE_AITER=0 / --attention-backend triton. Some MoE models are called out as possibly broken on Radeon.	No.	SGLang/AMD matrix (FP8, AWQ, GPTQ, …). GGUF is documented as not on the AMD quant short-list. Floor is still Q4-class.	Documented for Instinct-class serving. Not measured on two R9700.	Not llama.cpp -ngl. Host/HF cache only as documented.	Some MoE models (GPT-OSS-20B, MiniMax-M2.7) are listed as possibly failing on Radeon. Do not assume Qwen MoE works.	Engine default. Do not assume 262K.	Serving engine — batching is the point. Not measured here.	OpenAI-compatible server (SGLang runtime).	Hard — Docker + AITER off + untested on this rig.	You already run SGLang elsewhere and will validate on this card.	Tonight’s Qwen chat.
ExLlama / TabbyAPI Use with eyes open	Getting-Started documents pip install -U '.[amd]' against ROCm 6.1 (older text says 6.0 / 5.6). That is not ROCm 10.1 / gfx1201. ExLlamaV2 has a community PR to fix gfx1201 compile — do not treat it as merged or as a house path. FA2 on ROCm has been discussed as CDNA-oriented.	No.	EXL3 (recommended on current TabbyAPI), FP16/BF16; EXL2 on the checkpoint branch. Still never below Q4-class.	Documented on NVIDIA. Not documented for two R9700.	ExLlama cache / GPU split. Not measured.	Not documented as our Qwen MoE path.	Model/config dependent. Not measured.	Paged attention / continuous batching documented for Ampere+. ROCm completeness not documented.	OpenAI-compatible (TabbyAPI).	Hard on Radeon — custom wheels, old ROCm extra, untested here.	You already have EXL3 files and an NVIDIA box. On Radeon, curiosity only.	Matching this site’s GGUF daily driver.
MLC-LLM Use with eyes open	Official GPU page says MLC LLM only supports ROCm 6.1/6.2. Nightly wheels are mlc-llm-nightly-rocm61 / rocm62. That is not the 10.0/10.1 userspace on this site. Vulkan is documented as available in Linux packages.	Yes — documented. A more honest AMD on-ramp here than their ROCm 6.2 wheel on a 10.1 box.	MLC’s own compiled quants. Floor is still Q4-class. Do not chase IQ2 artifacts.	Not documented as our path.	Compile-time device placement. Not measured.	Not documented against Qwen3.6-35B-A3B here.	Artifact-dependent. Not measured.	Not a continuous-batching server.	Python / REST depending on the build. Not our :8080 convention.	Hard — compile a model, mismatch ROCm.	You already compile MLC artifacts. On this rig, curiosity + Vulkan, not ROCm 6.2.	The weekly Qwen GGUF.
KTransformers / ik_llama.cpp Use with eyes open	KT ROCm.md is beta, developed on EPYC + Radeon 7900 XTX (gfx1100), PyTorch ROCm 6.2.4 in the doc. Marlin is unsupported. Multi-GPU HIP graph capture has reported crashes. gfx1201 is not a listed test row. ik_llama.cpp: ROCm/Vulkan are explicitly unmaintained; issues will not be resolved unless you do the work.	ik_llama.cpp can compile Vulkan; maintainers say it is not the focus and to use upstream llama.cpp on AMD.	GGUF. Their docs mention IQ2/IQ3-class kernels — this site still refuses anything below Q4, including draft/MTP.	KT multi-GPU HIP has reported segfaults. Not our path.	CPU/RAM is the product. That fights this site’s “0% RAM offload” daily driver.	Yes — that is why people show up. Not a replacement for our 35B Vulkan Q5.	Not measured.	Local chat / one user.	KT local_chat; ik is llama.cpp-shaped. Not our harness convention.	Hard, and off-mission.	A 7900 XTX + lots of RAM experiment you will not confuse with the house stack.	The R9700 daily driver.
PyTorch / Transformers Fine second path	Install from AMD’s PyTorch-for-ROCm page with a gfx1201 selector. Confirm torch.version.hip before from_pretrained. Fine-tune on Radeon is possible and still a thinner cookbook than CUDA. ROCm 10.0 KI — training GPU resets; try TORCH_BLAS_PREFER_HIPBLASLT=0.	No (this is HIP). Use llama.cpp Vulkan for chat.	bitsandbytes / AWQ / GPTQ / native dtypes — still never below Q4-class for anything we would serve.	DDP / device_map. RCCL tax. Not the house chat path.	device_map="auto" / CPU offload — a research tool, not the daily driver.	If the HF model implements it. Not our Flash stack.	Whatever you code. Easy to OOM. Not 262K-as-a-flag.	One script.	None unless you wrap it. Prefer llama-server or vLLM for /v1.	Medium if the wheel matches; hard if it does not.	Fine-tune, custom pipelines, proving torch.cuda (HIP) sees gfx1201.	Replacing llama.cpp for chat.

Deep dives

Fit a quant: quantsand the calculator. Vulkan vs HIP: ROCm guide. Errors: paste a log. Plug a client in: harnesses.

Inference engines compared

Used on our rig

Which one should I use?

35B on llama.cpp, measured

Comparison

Deep dives

llama.cpp

Strata

vLLM

Ollama

LM Studio

SGLang

ExLlama / TabbyAPI

MLC-LLM

KTransformers / ik_llama.cpp

PyTorch / Transformers