Run llama.cpp across two (or more) Radeon GPUs

Two patterns

Zero to both cards

1. 1 · Cited how-to; 35B split measured on the local desktop

   Ubuntu 24.04+ first — see every GPU before you split

   Bare-metal Ubuntu 24.04+ is the AMDai measured ROCm path. Confirm both discrete cards before you invent a pin. Vulkan: llama-cli --list-devices and pick RADV GFX1201 / the R9700, not the iGPU. HIP: rocminfo first and confirm /dev/kfd and /dev/dri. Windows Vulkan is a cited second path at /setup/windows — we do not have Windows multi-GPU tok/s. AMD’s mGPU page: Microsoft does not currently support mGPU setup in WSL — /setup/wsl stays a single-GPU preview. Fedora and Arch have their own first-run pages; we do not invent a split tok/s on those.

   • Ubuntu setup (measured)
   • Verify ROCm
   • Windows setup
   • WSL2
   • R9700

2. 2 · Cited how-to; 35B split measured on the local desktop

   Vulkan — list devices, then --device or GGML_VK_VISIBLE_DEVICES

   Official llama.cpp multi-GPU page: run llama-cli --list-devices first, then restrict with --device / -dev (comma-separated names, or none). --device works for any backend. CUDA_VISIBLE_DEVICES is the CUDA-only env that page names. This site’s Vulkan pin after that list is GGML_VK_VISIBLE_DEVICES (and VK_ICD_FILENAMES if the ICD is messy) — set it only after the list shows the confusion. Overnight 35B split named Vulkan1. Do not copy Vulkan1 onto a box you have not listed. One process, one index, when you pin a model to a card.

   • llama.cpp multi-GPU docs
   • llama.cpp
   • Vulkan vs HIP
   • Dual / multi-GPU (why)

3. 3 · Cited how-to; 35B split measured on the local desktop

   ROCm / HIP — rocminfo, then HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES

   Official llama.cpp --device still applies on a HIP build. AMD’s GPU isolation page documents HIP_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES (and CUDA_VISIBLE_DEVICES, which PyTorch-on-ROCm still also reads). rocminfo first. Same rule as Vulkan: one model process per card unless you split on purpose. AMD’s mGPU page: PCIe slots the GPUs use must have identical width (x16+x16 or x8+x8); mixed x16+x8 is not recommended; keep GPUs on CPU-connected slots. RCCL for the HIP backend is a compile-time -DGGML_HIP_RCCL=ON — official docs say it is disabled by default. We do not invent a house RCCL pin.

   • AMD GPU isolation
   • AMD mGPU setup
   • Verify ROCm
   • ROCm versions

4. 4 · Cited how-to; 35B split measured on the local desktop

   --split-mode, --tensor-split, --main-gpu, and KV

   Official flags (same for llama-cli and llama-server): --split-mode / -sm none | layer (default) | tensor. Current docs mark row deprecated (older tensor-parallel-shaped path; superseded by tensor). --tensor-split / -ts is comma-separated proportions in --device order; official example llama-cli -m model.gguf -ts 3,1 puts 75 % on GPU 0 and 25 % on GPU 1 (same as 75,25). Omit -ts and layer splits proportional to memory. Do not invent a local desktop ratio — overnight named -sm layer, not a -ts pair. --main-gpu / -mg (default 0) is the single GPU in --split-mode none. Layer mode: KV for layer l lives on the GPU that owns layer l. Tensor mode also splits KV; official docs require non-quantized KV (f16 / bf16 / f32) and flash-attn on. House cards used f16 KV. --n-gpu-layers / -ngl 999 or all to push layers to the GPUs. --fit is on by default except in tensor mode.

   • llama.cpp flag table
   • One vs split
   • 35B on two cards
   • Speed up llama.cpp

5. 5 · Cited how-to; 35B split measured on the local desktop

   Pattern A — one llama-server, layer split

   Cited shape: llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -sm layer -ngl 99 -ctk f16 -ctv f16 --host 127.0.0.1 --port 8080. Keep the GGUF on fast NVMe, not /mnt/bulk. Bind 127.0.0.1. Overnight recorded llama-bench with -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16 — those are bench flags, not a llama-server promise. A 35B that already fits one 32 GB card is faster on one card. Split the file that does not fit. systemd-style unit is an example only, not a measured local desktop service.

   • Serve Qwen
   • Run a local Qwen
   • 35B on two cards
   • Flash daily

6. 6 · Cited how-to; 35B split measured on the local desktop

   Pattern B — two servers, one card each (combo)

   Pin each process. Vulkan example: GGML_VK_VISIBLE_DEVICES=0 llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --split-mode none --main-gpu 0 -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 127.0.0.1 --port 8080 and GGML_VK_VISIBLE_DEVICES=1 llama-server -m /path/to/nvme/other.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8081. HIP example: HIP_VISIBLE_DEVICES=0 llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8080 and HIP_VISIBLE_DEVICES=1 llama-server -m /path/to/nvme/other.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8081. After an env pin the process only sees one GPU, so --main-gpu 0 is that visible device. Or pass --device names from llama-cli --list-devices and skip the env. House combo is Flash (often Strata on 127.0.0.1:8083) plus 35B llama-server on 127.0.0.1:8080. A second llama-server copy uses 127.0.0.1:8081 — same port /setup/docker already uses for a second container. Two copies of the same 35B are concurrency, measured on /qwen36-dual, not 2× one chat. Combo-preset tok/s is not measured — use amdai-bench / Stack Lab.

   • Dual / multi-GPU (why)
   • Flash daily
   • 35B on two cards
   • Docker / Podman
   • Connect a harness

7. 7 · Cited how-to; 35B split measured on the local desktop

   VRAM fit — existing 35B and Flash numbers only

   Q6 / Q5 / Q4 first. Q3 stays a labeled estimate. Nothing below Q3. House 35B UD-Q5 already fits one 32 GB R9700 (publisher 26.6 GB; measured peak VRAM 25.97 GB at 4K, 29.34 GB at 256K). Buy the second card for a file that does not fit, or to run two models. Flash UD-Q6_K_XL is the measured house two-card job — publisher folder ~169 GB; expect RAM / expert offload even with two 32 GB cards. Flash Q5 / Q4 listings live on /flash-daily. UD-Q3_K_XL listing is 16.8 GB — Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB). Size both cards on the calculator before you seat the second SKU.

   • Calculator
   • One vs split
   • Q6 / Q5 / Q4
   • Flash daily
   • Flash RAM offload

8. 8 · Cited how-to; 35B split measured on the local desktop

   System RAM — both shopper labels wherever offload appears

   Minimum system RAM needed is the load at the context you picked (a kit estimate rounded to a real size unless a card measured the peak). RAM needed at full context is the 262144 line for the house 35B and Flash. 35B daily driver is 0% system-RAM offload on one 32 GB card — measured peaks 37.32 GB at 4K and 42.9 GB at 256K. Flash Q6 two-card still offloads experts; those RAM lines are calculator estimates unless a card is measured. Desktop 2 is planned as dual R9700 + 192 GB DDR5 — not measured. local desktop kit is 256 GB. GB here is Linux GiB printed as GB.

   • Calculator
   • Flash RAM offload
   • Long context
   • Dual / multi-GPU

9. 9 · Cited how-to; 35B split measured on the local desktop

   Speed — only numbers already on this site

   35B llama.cpp Vulkan UD-Q5_K_XL at 4K: one R9700 102.39 tok/s, two-card layer split 82.78 tok/s. Full matrix: /compare/one-vs-two-gpu and /benchmarks. Flash Q6 two-card Strata: 32.8 tok/s at 8K and 32K, 31.8 at 100K. Same Flash file on llama.cpp DDR5-5200: 18.5 / 18.2 / 17.9 tok/s. We do not have a measured combo-preset tok/s, a row-split tok/s, a tensor-split tok/s, or a Desktop 2 number. Do not guess — record a card with amdai-bench or open Stack Lab (packages are not published yet).

   • One vs split
   • Benchmarks
   • Flash daily
   • amdai-bench
   • Stack Lab
   • Wanted

10. 10 · Cited how-to; 35B split measured on the local desktop

    Hardware — PCIe split, PSU, airflow, mixing generations

    AMD AM5 chipset table: CPU graphics is 1×16 or 2×8 (PCIe 5.0 on X870E and X670E; PCIe 4.0 on B650). AMD’s mGPU page wants identical width on the slots the GPUs use (x16+x16 or x8+x8); mixed x16+x8 is not recommended; use CPU-connected slots, not chipset leftovers. R9700 product page: 300 W TBP and a 750 W minimum PSU recommendation per card. Two 300 W cards plus 750 W minimum per card is not a dual-card PSU SKU we will invent — check the unit you buy; /power and /budget keep the existing /go/ links, no prices here. Double-slot active cooling; leave the adjacent slot empty. Mixing RDNA 2 / 3 / 4 is not measured on the local desktop. Identical PCIe width still applies. HSA_OVERRIDE_GFX_VERSION is unofficial — do not spray it on gfx1201. BIOS notes (Above 4G, ReBAR/SAM, CSM) stay on /bios.

    • Hardware
    • Power and PSU
    • Budget
    • BIOS
    • R9700
    • RX 7900 XTX

11. 11 · Cited how-to; 35B split measured on the local desktop

    When the second card never lights up

    Paste the log in /troubleshoot/errors. Common multi-GPU misses: the process grabbed the iGPU (list devices, then pin); --n-gpu-layers is 0; visible-device env hid every GPU; --split-mode tensor on a MoE file (official: LLAMA_SPLIT_MODE_TENSOR not implemented — use layer); OOM because --fit is off in tensor mode (lower -c first); split slower than one card (measured 35B tax, not a broken install); HIP no binary for the gfx; unofficial HSA override. AMD mGPU: IOMMU hangs, chipset-slot HIP errors, and WSL mGPU unsupported. Start from /troubleshoot.

    • Error paste
    • Troubleshoot
    • ROCm problems
    • HIP no binary
    • FAQ

Cited commands

Official list, then a layer split or two pinned servers. Overnight 35B cards recorded -dev Vulkan1 -sm layer -ngl 99 -fitt 1024 -fa 1 -ctk f16 -ctv f16 on llama-bench. Paths are placeholders. Dual-card systemd units below are examples, not measured local desktop services. The one-GPU always-on unit lives on /setup/server.

llama-cli --list-devices

llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -sm layer -ngl 99 -ctk f16 -ctv f16 --host 127.0.0.1 --port 8080

llama-cli -m model.gguf -ts 3,1

GGML_VK_VISIBLE_DEVICES=0 llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --split-mode none --main-gpu 0 -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 127.0.0.1 --port 8080

GGML_VK_VISIBLE_DEVICES=1 llama-server -m /path/to/nvme/other.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8081

HIP_VISIBLE_DEVICES=0 llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8080

HIP_VISIBLE_DEVICES=1 llama-server -m /path/to/nvme/other.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8081

# Example only — not a measured local desktop unit.
# /etc/systemd/system/llama-split.service
[Unit]
Description=llama-server layer split across visible Radeon GPUs
After=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -sm layer -ngl 99 -ctk f16 -ctv f16 --host 127.0.0.1 --port 8080
Restart=on-failure

[Install]
WantedBy=multi-user.target

# Example only — not a measured local desktop unit.
# /etc/systemd/system/llama-35b.service
[Unit]
Description=llama-server 35B pinned to one Radeon
After=network-online.target

[Service]
Type=simple
Environment=GGML_VK_VISIBLE_DEVICES=0
ExecStart=/usr/local/bin/llama-server -m /path/to/nvme/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --split-mode none --main-gpu 0 -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 127.0.0.1 --port 8080
Restart=on-failure

[Install]
WantedBy=multi-user.target

# Example only — not a measured local desktop unit.
# /etc/systemd/system/llama-second.service
[Unit]
Description=llama-server second model pinned to the other Radeon
After=network-online.target

[Service]
Type=simple
Environment=GGML_VK_VISIBLE_DEVICES=1
ExecStart=/usr/local/bin/llama-server -m /path/to/nvme/other.gguf --split-mode none --main-gpu 0 -ngl 99 --host 127.0.0.1 --port 8081
Restart=on-failure

[Install]
WantedBy=multi-user.target

35B quants on two cards

Q6 / Q5 / Q4 from the overnight 35B table. Q3 stays a labeled estimate. Nothing below Q3. Flash fit — the file that actually needs two 32 GB cards — stays on /flash-daily.

RAM lines

Minimum system RAM needed is the load at the context you picked. RAM needed at full context is the 262144 line. House 35B peaks below are Ubuntu llama.cpp on one R9700. Flash two-card offload is an estimate unless a card is measured.

Troubleshooting

• Process attached to the iGPU

  llama-cli --list-devices and pick the discrete RADV / gfx name. Set GGML_VK_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, or ROCR_VISIBLE_DEVICES only after that list. Same row as /troubleshoot/errors (iGPU vs dGPU).

• LLAMA_SPLIT_MODE_TENSOR not implemented

  Official llama.cpp docs: tensor mode is experimental and missing for many MoE architectures. House 35B is MoE. Use --split-mode layer. We have not measured tensor on the local desktop.

• Split is slower than one card

  Measured 35B UD-Q5 4K: 102.39 tok/s on one R9700 versus 82.78 on a layer split. That is the topology tax on /compare/one-vs-two-gpu, not a missing flag.

• OOM at startup or during prefill

  Official tensor-mode note: --fit is off, so lower --ctx-size first, then llama-server --parallel, then -ngl. Layer mode still OOMs if the file plus KV does not fit the split. Calculator first.

Open in Stack Lab

Stack Lab remains Ubuntu llama.cpp-first. There is no public combo or dual-GPU package and no public download. Doctor still checks the box you already have. New speeds belong on amdai-bench.

Sources

• llama.cpp — Using multiple GPUs
• llama.cpp repository
• AMD — mGPU setup and configuration (ROCm on Radeon)
• AMD — GPU isolation environment variables
• AMD Socket AM5 Chipset
• AMD Radeon AI PRO R9700 product page
• Qwen3.6-35B-A3B GGUF (Unsloth)