What it is
Throughput-oriented Python serving engine: PagedAttention, continuous batching, an OpenAI-compatible server. On AMD it is a ROCm build, not the default PyPI wheel.
Architecture in plain English
GPU workers, a scheduler, and an HTTP front (default :8000/v1). Multi-GPU uses RCCL. Quant and MoE paths depend on the ROCm / AITER kernels in that pin — do not assume every CUDA feature landed.
What it does well
Several clients, continuous batching, a harness that wants a real server.
What it does badly
A first home box; mixing AMD’s 0.27 pin with community 0.31 in one venv.
ROCm / Radeon
vLLM’s GPU page lists Radeon RX 9000 (gfx1200/1201) among supported devices. That is not the same as AMD’s 10.0 supported-combo pin (0.27 + PyTorch 2.13). Treat 0.31 as a try on this site.
Vulkan. No. HIP/ROCm only.
Supported quants
Engine/framework quants (FP8, AWQ, GPTQ, …) plus HF weights — still never below Q4-class. GGUF is not the house path here.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# Community / ROCm 0.31 try — confirm the live extra-index before you pin.
# pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.31.0/rocm723
# Docker try-tag (previous try-tag v0.30.0 if 0.31 misbehaves):
# docker pull vllm/vllm-openai-rocm:v0.31.0
# AMD 10.0 conservative pin remains 0.27 + PyTorch 2.13 (often rocm/vllm images).
# Default `pip install vllm` with no extra index is CUDA. Do not do that.
Run
# TODO: exact image tag / measured VRAM if 0.31 is pinned on this rig.
# Shape only:
# vllm serve Qwen/Qwen3.6-35B-A3B --host 127.0.0.1 --port 8000
curl -s http://127.0.0.1:8000/v1/models
Tuning flags
One GPU first. Pin the ROCm extra-index or image. Confirm hip/rocm in the version string. After a kernel upgrade, check firmware before you rebuild the wheel.
When to choose it
Batching, not the daily driver.
You already have llama.cpp working and you need batching.