Serving many users on two Radeon cards: what works in 2026

The 30-second answer

llama.cpp handles concurrent requests well on one GPU. A third-party Strix Halo shootout (Qwen3.6-35B-A3B UD-Q4_K_M) saw aggregate tok/s climb from 58.8 at 1 slot to 189.7 at 16 on Vulkan, and from 48.6 to 207.8 on a Lemonade ROCm build. Per-token latency rose with load: Vulkan p95 inter-token latency went from 16.2 ms to 81.9 ms. That is third-party, not an AMDai R9700 card. What llama.cpp lacks is fast multi-GPU serving. Puget’s third-party 2× R9700 layer-split run (Qwen3.6-27B Q4_K_M, --parallel 8) flattened at about 56 tok/s from 4 to 8 users. The same article’s vLLM TP=2 stock build went from 15.88 tok/s at 1 user to 156.21 at 8. Try vLLM or Radiance when a model does not fit one card and you need tensor-parallel throughput on bare metal — not as the daily driver.

Building a two-card box? See the two-R9700 parts list (board with x8/x8 slots, 1200 W PSU, 96 GB DDR5), each part linked to its Newegg search.

Two-card parts list

Building a two-card box? Board with x8/x8 slots, 1200 W PSU, 96 GB DDR5. Each row opens the Newegg search for that part. The GPU row goes to the R9700 buy notes — we do not print a GPU price here.

Our lab measurements

These two cells are AMDai first-party numbers. They are one-client or Flash-pair cards, not a 16-user serving table.

• 102.39 tok/s

  Our lab · 35B UD-Q5_K_XL · one R9700 · 4K · llama.cpp Vulkan. Our lab.

  AMDai overnight 35B llama.cpp card — one R9700 UD-Q5_K_XL at 4K · checked 2026-10-28 · Measured on our rig, 2026-10

• 32.6 vs 16 tok/s

  Our lab · Flash Q6 at 262K · Strata vs llama.cpp. Our lab.

  AMDai first-party Flash Q6 pair — Strata vs llama.cpp · checked 2026-10-28 · Measured on our rig, 2026-10

llama.cpp first: flags that work today

Start with one llama-server. Official -np / --parallel sets slot count (default -1 = auto; env LLAMA_ARG_N_PARALLEL). Continuous batching (-cb) is on unless you pass --no-cont-batching. GET /health prints total_slots. Usable context per client is total ctx / np unless you raise -c or set --kv-unified-per-slot. House daily driver stays -np 1 until several clients share the unit. Shopper flags for two cards are --split-mode layer (default, best batch throughput) and --split-mode tensor (experimental, aims at lower single-stream latency). Official docs: pipeline-parallel maximizes batch throughput; tensor-parallel minimizes latency. Tensor mode needs flash attention, an unquantized KV cache, and no --fit. Only CUDA is optimized. ROCm works through HIP. Vulkan tensor split is still an open PR. Tensor mode is not available on Windows (no P2P).

Stay on llama.cpp when the GGUF fits one 32 GB card, when you want the widest model list, or when gfx1201 has to stay boring. Raise -np on one process when Open WebUI, Hermes, and a phone share one unit. Pin one llama-server per card behind a router when two copies of a model that already fits beat a split — that is the measured /qwen36-dual layout. Try vLLM TP=2 on bare metal with NCCL_PROTO=Simple when you need continuous batching plus tensor parallel on a dense ~27B that does not fit one card, and you can live with FP8 (GGUF is an experimental plugin). Try Radiance when you want an Apache-2.0 C++/HIP server written for two 32 GB R9700s, you can convert off GGUF, and the architecture is on its curated list. Do not start a private vLLM gfx1201 fork — the deprecated vllm-radiance tree is the cautionary tale.

# One interactive user on one R9700 (house daily driver).
# llama-server README: -np / --parallel N (default -1 = auto, four slots).
# Default idle-slot save copies empty slots on each new task (slow path on Radeon).
# Keep -np 1 for a single-user lane so those empty slots do not stall the stream.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --jinja \
  -np 1 --host 127.0.0.1 --port 8080

# Shared server on one GPU: size per-slot context first.
# Official --kv-unified-per-slot N is the context limit per slot.
./build-vk/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 --kv-unified-per-slot 32768 -np 4 \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

# Two cards, pipeline split (documented default).
# llama.cpp multi-gpu.md: layer maximizes batch throughput.
./build-hip/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 32768 --split-mode layer --parallel 8 \
  --cache-type-k f16 --cache-type-v f16 --jinja \
  --host 127.0.0.1 --port 8080

# Two cards, experimental tensor split. CUDA is the optimized path.
# Requires flash attention, unquantized KV, and no --fit.
./build-hip/bin/llama-server \
  -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 -c 32768 --split-mode tensor -fa on \
  --cache-type-k f16 --cache-type-v f16 --parallel 4 --jinja \
  --host 127.0.0.1 --port 8080

Two ways to split a model across two GPUs

Pipeline (layer) split parks some layers on GPU 0 and the rest on GPU 1. One user’s decode still waits on both cards in sequence, so a single stream does not get faster. Extra users can fill the idle card, which is why layer split is the throughput layout. Tensor (row / experimental tensor) split cuts each layer across both cards and all-reduces activations. That can cut single-stream latency when the interconnect and all-reduce are healthy. On this desktop the cards sit on PCIe 5.0 x8/x8. Puget’s third-party P2P bandwidth on 2× R9700 was 27.6 GB/s. RCCL hangs under KVM/VFIO in that article and works on bare metal. A GPU behind the chipset (no PCIe atomics) breaks RCCL.

Engine status board

Last checked 2026-10-28. Cells paraphrase cited docs plus house rules. Third-party speed cells stay on the figures list below.

Cited third-party figures

Every number below is third-party. Each cell links to its source and the date we checked it. None of these are our lab.

vLLM TP=2 on R9700

vLLM’s docs list RX 9000 / gfx1200–1201. AMD’s ROCm 10.1 matrix lists the R9700 with vLLM 0.29.0. AMD ships gfx1201 wheels. Puget’s third-party 2× R9700 table (Qwen3.6-27B, 500 in / 200 out) at concurrency 1 / 4 / 8: stock v0.20.2 FP8 TP=2 was 15.88 / 69.34 / 156.21 tok/s; a tuned “vllm-radiance” build 32.0 / 110.6 / 198.8; tuned plus MTP 62.8 / 186.3 / 320.2. Wendell’s third-party 4× R9700 notes: Llama-3.1-8B BF16 38.1 tok/s at TP1 and 65.1 at TP2; Qwen3.6-35B-A3B-FP8 17.2 at TP2 and 15.6 at TP4. That last pair is far below our lab one-card 35B Q5. FP8 MoE on gfx12 landed through Triton. An R9700-tuned config PR is still open. AWQ/GPTQ on AMD is patchy. GGUF moved to vllm-gguf-plugin; the docs call it experimental and under-optimized. The vllm-radiance Docker fork is deprecated.

Strata batching

Strata’s HIP backend lists the R9700 as maintainer-tested. Multi-GPU is layer split — explicitly not tensor parallelism. Pipeline windows took a third-party 2× R9700 decode from 103 to 125 tok/s. Batching is opt-in since 0.1.39 (issue #465: “not planned” on 2026-10-02, shipped 2026-10-04). Up to 8 slots per window; --batch-groups pipelines slot groups across cards. Third-party owner figures: 4× R9700 and 8 clients, 166 vs 86 tok/s; on 2 GPUs, 109 vs 78; 16 slots on 4× R9700, 324 tok/s. --batch-mtp on 2× R9700: 80.2 to 62.7 tok/s. Showcase packs often sit below Q4, so they are not a house recipe. Batch windows have no sampling penalties.

Radiance

Radiance is an Apache-2.0 C++/HIP inference server. Bundled kernels target RDNA4 (gfx1200/1201). Serve flags are written for two 32 GB R9700s. APIs: OpenAI, Anthropic, and llama-server-compatible. Features: --tp 2/3/4/8, up to 32 sequences per step, FP8 / int8 / int4 / MXFP4 KV and weights, MTP/DFlash, and a lossy compressed TP wire (--tp-wire wht6). Published third-party 2× R9700 numbers: Qwen3.8-27B MXFP4 134–140 tok/s at 1 stream and 550–555 at 16; TranslateGemma 27B MXFP4 56 tok/s per stream and 205 at four. The guide notes MoE aggregate decode roughly doubles at batch 8 because routed-expert reads are per token. It does not serve GGUF directly; Q4_K / Q6_K / Q8_0 import through rad-convert for supported architectures. Model support is a curated list. The Codeberg repo was created 2026-09-25.

Other engines

Status paraphrases cited docs. We do not invent a tok/s for rows without a linked figure.

Known pitfalls on R9700

RCCL 7.2.x has no gfx1201 tuning entry, so the LL protocol can deadlock. The third-party workaround is NCCL_PROTO=Simple. The ROCm/rccl Navi4 tuning PR is still open. vLLM issue #40980 (TP=2 deadlock on dual R9700) is still open as of 2026-10-17; a maintainer wrote there is no official AITER support for gfx1201. A GDN Triton miscompile hangs TP≥2 startup for Qwen3.6 and Qwen3-Next. shm_broadcast hangs appear in vLLM ≥0.21.0 on that Level1Techs thread. A GPU behind the chipset fails RCCL. VMs hang RCCL; bare metal is the cited path. Windows has no tensor-mode P2P. Do not copy a one-client 35B tok/s onto an 8-user cell.

What to run, in order

1. 1 · Cited multi-user serving · llama.cpp first

   Serve several users on one card with -np

   Vulkan llama-server, Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf, f16 KV, loopback :8080. Start at -np 1. Raise -np only when several clients stream at once, and size --kv-unified-per-slot first. Slot tok/s on one R9700 stay not measured here — see /concurrency-ab. Q6/Q5/Q4 first. UD-Q3_K_XL is a labeled estimate.

   • Parallel slots
   • Always-on server
   • llama.cpp

2. 2 · Cited multi-user serving · llama.cpp first

   On two cards, prefer copies or layer split

   If the GGUF fits one 32 GB card, two independent llama-server processes beat a split for aggregate tok/s. That is the measured /qwen36-dual soak. --split-mode layer is the documented default when one file must span both cards. It does not speed up one user.

   • 35B on two cards
   • Dual / multi-GPU
   • Multi-GPU llama.cpp

3. 3 · Cited multi-user serving · llama.cpp first

   Treat --split-mode tensor as an experiment

   PR #19378 merged 2026-04-09. CUDA is the optimized path. HIP works and was slower than layer split on the author’s mixed AMD setups. Requires -fa, f16 KV, and no --fit. Third-party 2× R9700 ROCm tg128: Qwen3-27B Q8_0 19.50 → 28.30 → 30.43 with a P2P AllReduce hack. Windows has no tensor mode.

   • Flash Attention
   • KV cache
   • Vulkan vs ROCm

4. 4 · Cited multi-user serving · llama.cpp first

   Try vLLM TP=2 only on bare metal

   Worth a try when you need continuous batching plus tensor parallel on a dense file that does not fit one card, and you can run FP8. Set NCCL_PROTO=Simple. Expect open RCCL, AITER, and Triton issues. GGUF is an experimental plugin. The vllm-radiance fork is deprecated.

   • Backends
   • ROCm problems
   • AI software

5. 5 · Cited multi-user serving · llama.cpp first

   Try Radiance for an R9700-native TP server

   Apache-2.0 C++/HIP, flags written for two 32 GB R9700s, up to 32 sequences, --tp 2. Convert GGUF with rad-convert. Model list is curated. Published third-party 27B MXFP4 numbers are far above llama.cpp layer split — they are not our lab.

   • Backends
   • R9700
   • How we test

6. 6 · Cited multi-user serving · llama.cpp first

   Do not invent a 16-user tok/s

   Our lab 35B and Flash cells are one-client or first-party Flash pairs. Multi-user tables on this page are third-party and linked. The coming dual-Radeon serving kit ships a harness for 1/2/4/8/16 users, TTFT, inter-token latency, and total tok/s on Q4–Q6 only. No repository is linked yet.

   • Benchmarks
   • Run cards
   • How we test
   • Stack Lab

Open project: dual-Radeon serving kit (coming soon)

The dual-Radeon serving kit is an open project, coming soon. Milestone 1 is a thin OpenAI-compatible router in front of one llama-server per card, plus an open benchmark harness. The router will use least-outstanding-tokens balancing with prefix and session affinity so prompt caches stay warm, health checks, and /metrics passthrough. Optional spill will route a model that does not fit one card to a TP or layer-split instance. The harness will drive 1, 2, 4, 8, and 16 concurrent users and record TTFT, inter-token latency, and total tok/s. Weight files stay Q4 through Q6. Engines in the same harness: llama-server on ROCm and Vulkan (1 card, -sm layer, -sm tensor, and 2 replicas plus the router), Strata when a Q4+ pack exists, vLLM ROCm FP8 TP=2 with NCCL_PROTO=Simple, and Radiance FP8 or 4-bit. No public repository is linked yet.

Sources

Every claim on this page links here. Dates are when we checked the URL, plus the source’s own date when it has one.

• S-19378 — llama.cpp PR #19378 — --split-mode tensor (experimental) — third-party · checked 2026-10-28 · merged 2026-04-09
• S-MGPU — llama.cpp multi-gpu.md — layer / row / tensor — third-party · checked 2026-10-28 · master, read 2026-10-28
• S-27825 — llama.cpp PR #27825 — HIP internal AllReduce — third-party · checked 2026-10-28 · merged 2026-09-15
• S-25197 — llama.cpp Discussion #25197 — 2× R9700 ROCm tensor-split numbers — third-party · checked 2026-10-28 · 2026-07-01
• S-B9788 — llama.cpp b9788 — SYCL tensor-parallel all-reduce template — third-party · checked 2026-10-28 · 2026-06-25
• S-25051 — llama.cpp PR #25051 — Vulkan tensor-parallel all-reduce (open) — third-party · checked 2026-10-28 · opened 2026-06-26, updated 2026-10-27
• S-25356 — llama.cpp issue #25356 — Vulkan MoE batch cliff, closed — third-party · checked 2026-10-28 · 2026-07-06, closed 2026-10-15
• S-CSV — Strix Halo server shootout CSV — llama-server -np scaling — third-party · checked 2026-10-28 · measured 2026-05-05
• S-REDDIT — r/ROCm — dual 7900 XTX chipset RCCL and Windows tensor-mode notes — third-party · checked 2026-10-28 · 2026, read 2026-10-28
• S-PUGET — Puget Systems — dual R9700 AI inference (vLLM vs llama.cpp) — third-party · checked 2026-10-28 · published 2026-08-18, modified 2026-09-10
• S-L1T — Level1Techs — R9700 Linux TP=2 and TP=4 guide — third-party · checked 2026-10-28 · 2026-05-22
• S-RCCL2166 — ROCm/rccl PR #2166 — Navi4 default tuning (open) — third-party · checked 2026-10-28 · open as of 2026-10-28
• S-40980 — vLLM issue #40980 — TP=2 deadlock on dual R9700 — third-party · checked 2026-10-28 · open; comments through 2026-10-17
• S-46190 — vLLM PR #46190 — GDN Triton miscompile hangs TP≥2 startup — third-party · checked 2026-10-28 · 2026, read 2026-10-28
• S-VLLM-INSTALL — vLLM docs — RX 9000 / gfx1200–1201 GPU install — third-party · checked 2026-10-28 · read 2026-10-28
• S-AMD-VLLM — AMD ROCm docs — vLLM on Radeon, gfx1201 wheels — third-party · checked 2026-10-28 · read 2026-10-28
• S-ROCM-MATRIX — ROCm 10.1 compatibility matrix — R9700, vLLM 0.29.0, SGLang 0.5.18 — third-party · checked 2026-10-28 · read 2026-10-28
• S-38086 — vLLM PR #38086 — Triton FP8 MoE for gfx12 — third-party · checked 2026-10-28 · merged 2026-04-02
• S-36659 — vLLM PR #36659 — R9700 FP8 MoE tuned config (open) — third-party · checked 2026-10-28 · open as of 2026-10-28
• S-GGUF — vLLM GGUF plugin — experimental, under-optimized — third-party · checked 2026-10-28 · read 2026-10-28
• S-VLLM-RAD — vllm-radiance — deprecated R9700 vLLM fork — third-party · checked 2026-10-28 · read 2026-10-28
• S-RAD — Radiance engine — Apache-2.0 C++/HIP README — third-party · checked 2026-10-28 · repo created 2026-09-25, updated 2026-10-28
• S-RAD-GUIDE — Radiance GUIDE.md — 2× R9700 serve flags and published numbers — third-party · checked 2026-10-28 · read 2026-10-28
• S-STRATA-HIP — Strata AMD_HIP.md — R9700 maintainer-tested HIP backend — third-party · checked 2026-10-28 · read 2026-10-28
• S-STRATA-MGPU — Strata MULTI_GPU.md — pipeline windows, not tensor parallelism — third-party · checked 2026-10-28 · read 2026-10-28
• S-STRATA-BATCH — Strata BATCHING.md — opt-in slots and batch-groups — third-party · checked 2026-10-28 · read 2026-10-28
• S-STRATA-465 — Strata issue #465 — batching shipped in 0.1.39 — third-party · checked 2026-10-28 · 2026-10-02 / 2026-10-04
• S-AMD-SGL — AMD ROCm docs — SGLang; disable AITER on Radeon — third-party · checked 2026-10-28 · read 2026-10-28
• S-SGL30599 — SGLang issue #30599 — consumer Radeon / Instinct-only ROCm path — third-party · checked 2026-10-28 · opened 2026-07-09, updated 2026-10-28
• S-SGL-FORK — Community SGLang RDNA4 fork — 2× R9700 TP=2 notes — third-party · checked 2026-10-28 · sweep 2026-10-19
• S-MLC — MLC-LLM CLI — tensor_parallel_shards — third-party · checked 2026-10-28 · read 2026-10-28
• S-EXL3 — ExLlamaV3 releases — preliminary ROCm (v1.6.0+) — third-party · checked 2026-10-28 · v1.6.0 2026-10-07 … v1.6.5 2026-10-25
• S-LEMON — Lemonade issues — gfx1201 vllm-rocm reports — third-party · checked 2026-10-28 · read 2026-10-28
• S-OLLAMA — Ollama FAQ — OLLAMA_NUM_PARALLEL and multi-GPU placement — third-party · checked 2026-10-28 · read 2026-10-28
• S-LLAMA-SERVER — llama.cpp llama-server README — -np / --parallel — third-party · checked 2026-10-28 · master, read 2026-10-28
• S-LAB-35B — AMDai overnight 35B llama.cpp card — one R9700 UD-Q5_K_XL at 4K — our lab · checked 2026-10-28 · Measured on our rig, 2026-10
• S-LAB-FLASH — AMDai first-party Flash Q6 pair — Strata vs llama.cpp — our lab · checked 2026-10-28 · Measured on our rig, 2026-10