What it is
Fast, memory-efficient LLM serving (RadixAttention, a structured generation front). AMD publishes ROCm Docker tags and lists the Radeon AI PRO R9700 (gfx1201) on that page.
Architecture in plain English
Python serving process, attention backends (AITER on Instinct, Triton workaround on Radeon). OpenAI-shaped HTTP. Not a GGUF daily driver.
What it does well
Serving / structured generation when you already live in SGLang.
What it does badly
First home box; default AITER on Radeon.
ROCm / Radeon
AMD lists R9700 / RX 9070 XT on the ROCm 10.0 SGLang page. Known issue: default AITER attention may fail on Radeon PRO W7900, AI PRO R9700, and RX 9070 XT. Workaround: SGLANG_USE_AITER=0 / --attention-backend triton. Some MoE models are called out as possibly broken on Radeon.
Vulkan. No.
Supported quants
SGLang/AMD matrix (FP8, AWQ, GPTQ, …). GGUF is documented as not on the AMD quant short-list. Floor is still Q4-class.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# AMD ROCm 10.0 image from the AI-ecosystem page (Radeon: disable AITER).
docker pull rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0
docker run -it --rm \
--device /dev/kfd --device /dev/dri \
--network=host --ipc=host --group-add=video \
-e SGLANG_USE_AITER=false \
-e SGLANG_ROCM_FUSED_DECODE_MLA=false \
rocm/sgl-dev:v0.5.15.post1-ubuntu24.04-py3.14-rocm10.0.0 \
bash
Run
# Launch command for our Qwen GGUF is not documented on that page.
# Use their HF/serving examples inside the container. Not measured here.
Tuning flags
export SGLANG_USE_AITER=0 and SGLANG_USE_AITER_AR=0, or --attention-backend triton, if AITER faults on the R9700.
When to choose it
Only if SGLang is already your shop standard.
After llama.cpp, if you need SGLang’s serving features and will disable AITER.