Dual / multi-GPU

Decision

When a second card helps

A second 32 GB Radeon AI PRO R9700 is not free 2× decode. Models that already fit one card run fastest on one card. Buy the second card for a file that does not fit, or to run two models at once.

Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL at 262K is the house daily driver on ONE R9700. It stays in 32 GB VRAM. Official llama.cpp Vulkan overnight cards: one-card decode is faster than a split of the same file. A split pays a topology tax; it is not “two cards, twice the chat.”

A second card is for models that do not fit one 32 GB card. The measured house example is Qwen 3.8 Flash UD-Q6_K_XL on two R9700s (Strata two-card points on /flash-daily). That file is a ~169 GB publisher listing — expect RAM / expert offload even with two cards. Do not invent a Flash split tok/s that is not on those cards.

The other honest two-card job is two models side by side: Flash on one GPU plus 35B on the other. House rule: one LLM per GPU unless you deliberately combine. That is two llama-server (or Strata + llama-server) processes, each pinned to one device — not tensor parallel.

Two copies of the same 35B (one per card, least-connections balancer) serve two requests. That is concurrency, not a faster single chat. Measured soak lives on /qwen36-dual.

Q6_K single-GPU 4K table cell is 103.23 tok/s from the 23:55 card (the 4K point in the complete Q6_K single-GPU series). The earlier 23:26 card at 82.62 tok/s stays on /results and is not averaged into the cell.

Two copies · per request

100 tok/s

About 100 tok/s · one copy per card

Total · 2 concurrent

172 tok/s

4–8 concurrent ~177 tok/s

Soak

0 errors

423 requests · 20.6 minutes

Full soak table: 35B on two cards. Matrix: one vs split.

Engine · default

llama.cpp first

llama.cpp is the consumer default. Overnight split cards recorded -sm layer. Pin one process per card when you want two models, not a split.

• Layer split. --split-mode layer (short -sm layer) splits layers across the visible GPUs. That is the overnight 35B split on the local desktop. It is the documented llama.cpp default for multi-GPU. It is slower on this 35B than one card.
• Row split. --split-mode row splits tensor rows (a tensor-parallel-shaped layout). llama.cpp documents the flag. We have not measured row split tok/s on the local desktop — treat any row-split speed as an estimate, not a house number.
• --tensor-split. --tensor-split / -ts sets the fraction of the model on each GPU (comma-separated). Do not invent a local desktop ratio. Copy the flag from llama.cpp --help after --list-devices; measure. Overnight cards name -sm layer, not a -ts pair.
• --main-gpu. --main-gpu / -mg picks the GPU for scratch and small tensors, or the only GPU with --split-mode none. Use it when one process should own one card.
• Vulkan devices. Vulkan: run llama-cli --list-devices and pick the R9700 (RADV GFX1201), not the iGPU. Set GGML_VK_VISIBLE_DEVICES (and VK_ICD_FILENAMES if the ICD is messy) only after that list shows the confusion. One process, one index, when you pin a model to a card.
• HIP / ROCm devices. ROCm / HIP: rocminfo first. HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES select the HIP device list. PyTorch-on-ROCm still also reads CUDA_VISIBLE_DEVICES. Same rule: one model process per card unless you split on purpose.
• Pin one model per card. Two models side by side: two servers, two visible-device pins, two ports (house convention: llama-server :8080, Strata :8083). Do not point both at every GPU.

Docs: ggml-org/llama.cpp. Install path: llama.cpp · run a local Qwen · verify ROCm.

Engine · secondary

vLLM tensor parallel

vLLM tensor parallel is a secondary option, not the house path. Multi-GPU uses RCCL. Two consumer cards is a different project — start on one GPU.

The 50–61 tok/s band on /qwen36-dual is one R9700, INT4-W4A16, community/first-party image stilldeadcode/vllm-radiance:0.5.8. That is not tensor parallel. vLLM is unused on the engine catalog for this rig.

Tensor-parallel decode on two R9700s is not measured here. Cite vLLM’s GPU install and AMD’s vLLM page; do not invent a TP tok/s.

vLLM GPU install · vLLM on AMD ROCm · vLLM on this site.

Board · power · air

Hardware

Consumer boards are not Instinct servers. Matching CPU-direct lanes, slot spacing, and AMD’s published watts matter more than a second SKU sticker.

PCIe lanes and bifurcation

AMD’s AM5 chipset table lists CPU graphics as 1×16 or 2×8 (PCIe 5.0 on X870E and X670E; PCIe 4.0 on B650). That is the official CPU-direct split, not a chipset x4 leftover. ROCm on Radeon multi-GPU wants identical width on the slots the GPUs use (x16+x16 or x8+x8). A mixed x16+x8 pair is listed as not recommended. Keep GPUs on CPU-connected slots; read the board map.

Dual x8/x8 motherboards · BIOS notes · AMD AM5 chipset table · ROCm on Radeon mGPU.

Slot spacing and cooling

The R9700 product page lists a double-slot card with active cooling. Many AI PRO cards are compact blowers that exhaust out the bracket — leave the adjacent slot empty, keep intake clear. Open-air benches lie about case recirculation. Partner coolers change acoustics; look at the actual AIB board. We do not invent a noise dB.

Longer notes: hardware.

PSU sizing

AMD publishes 300 W TBP and a 750 W minimum PSU recommendation per R9700 (product page). That is electrical, not a street price. Two 300 W TBP cards plus 750 W minimum per card is not a dual-card PSU SKU we will invent. Check the unit you buy; use a native 12V-2x6 cable if that is what the card expects.

AMD R9700 product page — 300 W TBP, 750 W minimum, double slot, PCIe 5.0 x16. No prices on this page.

Q6 / Q5 / Q4 · Q3 estimate

Quants

House 35B compare is Q6_K, UD-Q5_K_XL, Q5_K_M, and Q4_K_M from overnight cards. UD-Q3_K_XL is a labeled estimate from the Unsloth listing. Q1/Q2 stay off. Q3 stays on the table as that labeled estimate.

UD-Q3_K_XL is not a local desktop run. Cite: Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB). Flash Q3 file rows on /flash-daily are the same rule — estimate unless a card is measured.

• UD-Q5_K_XL — measured on the overnight 35B cards.
• Q5_K_M — measured on the overnight 35B cards.
• Q4_K_M — measured on the overnight 35B cards.
• Q6_K — measured on the overnight 35B cards.
• UD-Q3_K_XL — estimate. Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB).

Side-by-side: Q6 / Q5 / Q4 + Q3 estimate · quant explainer.

When weights miss VRAM

RAM offload

The 35B daily driver is 0% system-RAM offload on one 32 GB card. RAM offload shows up when the file does not fit — Flash Q6 is the measured house example. The calculator always prints two lines: “Minimum system RAM needed” (the load at the context you picked) and “RAM needed at full context” (262144 for the house 35B and Flash).

Change context on the calculator. Launch shape: Flash RAM offload. The 35B daily driver does not offload weights — those RAM peaks are OS / runtime on the 256 GB kit.

Planned kit · not measured

Desktop 2

the house Desktop 2 is planned as dual Radeon AI PRO R9700 + 192 GB DDR5. 192 GB is a real DIMM kit label on the calculator. That box is not measured. Do not treat 192 GB as a local desktop result — the measured kit is 256 GB DDR5-5200 (about 249 GB usable, Linux GiB labeled GB). No Desktop 2 tok/s exists on this site.

192 GB is on the calculator DIMM list. Until that box records an amdai-bench/run@1 card, every Desktop 2 speed stays not measured.

Keep reading

More pages

• R9700 measured page

  32 GB 35B llama.cpp sheet — one GPU vs split, cited cards.

• 35B on two cards

  Split vs two copies of UD-Q5_K_XL. Soak and concurrency.

• One vs two GPU

  Same 35B, one R9700 versus a split. Decode winner per row.

• Flash daily driver

  Qwen 3.8 Flash two-card Q6 points and Q3–Q6 RAM offload.

• Flash RAM offload

  Experts on DDR5. Minimum system RAM and full-context lines.

• Backends

  llama.cpp first. vLLM multi-GPU is RCCL — a different project.

• Serving many users

  Several users on 1–2 cards. llama.cpp -np and --split-mode first.

• llama.cpp

  Vulkan daily driver. --list-devices before visible-device pins.

• First-run setup

  Ubuntu 24.04+ to first token, then this guide if you seat two cards.

• Multi-GPU llama.cpp how-to

  Hands-on device pins, --split-mode layer, --tensor-split, and two-server combo.

• Calculator

  Size a GGUF against 32 GB or 64 GB. Client-side, no prices.

• Budget

  Starter through two-R9700 builds. No invented prices.

• Dual x8/x8 motherboards

  Cited AM5 boards that actually split CPU lanes x8/x8. Maker manuals only.

• BIOS

  Above 4G, ReBAR/SAM, CSM, x8/x8 vs x16/x4 — cited chipset notes.

• Hardware

  300 W TBP, 750 W min PSU, double slot, cooling notes.

• Power and PSU

  Cited TBP per GPU. Dual R9700 is not a PSU SKU we invent.

• Glossary

  Dual-GPU, offload, run cards — plain English.

• Benchmarks

  33-card local desktop matrix. Every cell links a run card.

• Run cards

  Official amdai-bench/run@1. Exact launch command.

Citations

• llama.cpp repository
• AMD Radeon AI PRO R9700 product page
• AMD Socket AM5 Chipset
• mGPU setup and configuration (ROCm on Radeon)
• vLLM GPU installation
• vLLM on AMD ROCm
• Qwen3.6-35B-A3B GGUF (Unsloth)