Run llama.cpp on Radeon in Docker or Podman

Two tracks

Zero to first token

1. 1 · Cited, not measured on the local desktop

   Host first — Ubuntu is still the measured path

   A container does not replace a working host GPU stack. Ubuntu 24.04+ on bare metal is the measured AMDai path. The host login still needs render and video so /dev/kfd and /dev/dri exist. Fedora and Arch have their own cited guides; they are not on the ROCm 10.1 OSID row. Windows is not this page. Not measured on the local desktop.

   • Ubuntu setup (measured)
   • Fedora setup
   • Arch setup
   • Windows setup
   • R9700 measured page

2. 2 · Cited, not measured on the local desktop

   Pick the official llama.cpp image

   First token: docker pull ghcr.io/ggml-org/llama.cpp:server-vulkan. HIP second: docker pull ghcr.io/ggml-org/llama.cpp:server-rocm. Those tag names are from llama.cpp docker.md — we do not invent a digest or a dated pin. full-* and light-* siblings exist on the same page if you want llama-cli. CUDA tags are the wrong family on a Radeon. Not measured on the local desktop.

   • llama.cpp
   • Backends
   • Vulkan vs HIP

3. 3 · Cited, not measured on the local desktop

   Pass /dev/dri — and /dev/kfd for ROCm

   Vulkan: docker run --device /dev/dri --group-add video --group-add render. ROCm / HIP: add --device /dev/kfd. AMD’s ROCm 10.1 container page documents both nodes and optional --security-opt seccomp=unconfined (memory mapping; they mark it optional). Our error matcher already cites --device /dev/kfd --device /dev/dri --group-add video. Passing all of /dev/dri exposes every GPU; AMD shows renderD128 / renderD129 when you want to restrict. Not measured on the local desktop.

   • Error paste
   • Verify ROCm
   • What is ROCm

4. 4 · Cited, not measured on the local desktop

   Podman rootless, groups, and Fedora SELinux

   podman run accepts the same --device paths. Official podman-run notes: in rootless mode a device is bind-mounted, and if access is only via a group you need --group-add keep-groups. On SELinux (Fedora), device bind-mounts can get permission denied until sudo setsebool -P container_use_devices=true. Volume labels :z (shared) and :Z (private) are for the model mount, not for /dev. Link the Fedora guide for distro packages; this page is the container layer. Not measured on the local desktop.

   • Fedora setup
   • Arch setup
   • Troubleshoot

5. 5 · Cited, not measured on the local desktop

   RDNA 2 gfx1031 — HSA_OVERRIDE stays unofficial

   RX 6700 XT is gfx1031. That is not the official gfx1030 W6800 row. Inside a ROCm container the same unofficial HSA_OVERRIDE_GFX_VERSION=10.3.0 community note applies — it is not a recommendation. Prefer server-vulkan. Do not set an override on gfx1201. Not measured on the local desktop.

   • gfx1031 HSA_OVERRIDE
   • RX 6700 XT sheet
   • Vulkan vs HIP

6. 6 · Cited, not measured on the local desktop

   Mount the GGUF from NVMe, not a spinning disk

   docker.md uses -v /path/to/models:/models. Point that host path at the fast NVMe the house rule already names — not a bulk HDD. The file inside the container is /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf for the daily-driver shape. On Fedora/Podman add :Z (this container) or :z (shared) on that volume. Staging is not RAM offload. Not measured on the local desktop.

   • 35B recipe
   • Hardware
   • Glossary

7. 7 · Cited, not measured on the local desktop

   Pick Q6, Q5, or Q4 — Q3 is a labeled estimate

   Serve the same official-instruct GGUF the Ubuntu recipes name. Start at Q6, Q5, or Q4 and check VRAM + KV on the calculator. A cited Q3 file is a labeled estimate unless a container run card exists — none does in this repo. Q1/Q2 stay off. Tok/s on /benchmarks and /results are Ubuntu llama-bench on the local desktop, not a Docker or Podman promise.

   • Calculator
   • Quants
   • Qwen compare
   • 35B recipe

8. 8 · Cited, not measured on the local desktop

   Fit VRAM and the two RAM lines

   The calculator prints Minimum system RAM needed (the load at the context you picked) and RAM needed at full context (262144 for the house 35B). Overnight UD-Q5 one-GPU Ubuntu peaks are 37.32 GB at 4K and 42.9 GB at 256K — measured on the local desktop, not inside a container. When a file spills to system RAM, round Minimum system RAM needed to a real kit (192 or 256 GB) with a breakdown and keep it a labeled estimate until measured. Flash offload notes live on Flash RAM offload. Not measured on the local desktop.

   • Calculator
   • Flash RAM offload
   • Benchmarks

9. 9 · Cited, not measured on the local desktop

   First-run command (Ubuntu-measured flags, container not measured)

   House daily-driver flags from the 35B recipe / tuning guide, wrapped in the official server-vulkan image: docker run --rm --device /dev/dri --group-add video --group-add render -v /path/to/nvme/models:/models:ro -p 127.0.0.1:8080:8080 ghcr.io/ggml-org/llama.cpp:server-vulkan -m /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 0.0.0.0 --port 8080. docker.md publishes --host 0.0.0.0 inside the container; the host publish is 127.0.0.1:8080 so the OpenAI-shaped /v1 stays on loopback. HIP swap: add --device /dev/kfd --security-opt seccomp=unconfined and ghcr.io/ggml-org/llama.cpp:server-rocm. Confirm tokens, then curl -s http://127.0.0.1:8080/v1/models. That curl shape is already on this site. Do not copy Ubuntu tok/s onto the container. Not measured on the local desktop.

   • 35B recipe
   • 4K run card
   • Tuning flags
   • Harnesses

10. 10 · Cited, not measured on the local desktop

    Compose, two cards, and a loopback check

    AMD’s ROCm 10.1 container page shows a Compose devices list of /dev/kfd and /dev/dri. The Vulkan first file on this page uses /dev/dri only. Publish 127.0.0.1:8080:8080. Two cards: AMD documents passing renderD128 and renderD129 to restrict GPUs. House preference is two copies, not an automatic split — see Dual / multi-GPU. Two containers, one render node each, different host ports, is the two-copy shape; we do not invent --tensor-split values or a container tok/s. Check readiness with curl -s http://127.0.0.1:8080/v1/models. A Compose healthcheck that hits that URL is the same check, not a measured SLA. Not measured on the local desktop.

    • Dual / multi-GPU
    • 35B on two cards
    • Connect a harness

11. 11 · Cited, not measured on the local desktop

    Keep /v1 on localhost, then pick a harness

    Bind the published port to 127.0.0.1. Do not publish 0.0.0.0:8080 unless you mean to share the LAN. Harnesses that run in a second container need host.docker.internal, not 127.0.0.1 — that note is already on the harness pages. llama.cpp stays first. Not measured on the local desktop.

    • Harnesses
    • Connect a harness
    • Backends
    • llama.cpp

12. 12 · Cited, not measured on the local desktop

    When it breaks

    Permission denied on /dev/kfd: host groups, missing --device, or rootless Podman without keep-groups / container_use_devices. No Vulkan device: host mesa-vulkan-drivers / vulkaninfo first, then pass /dev/dri. Wrong gfx: rocminfo inside the ROCm image; do not invent an override on gfx1201. Start from /troubleshoot and paste a log in the error matcher. Cited rows live on /rocm-problems. Stack Lab remains the Ubuntu companion — packages are not published yet and there is no container package. Not measured on the local desktop.

    • Troubleshoot
    • Error paste
    • ROCm problems
    • FAQ
    • Glossary
    • Stack Lab

Compose examples

Cited shape only. Replace the NVMe host path. HIP adds /dev/kfd and optional seccomp=unconfined from AMD’s ROCm 10.1 container page. Not measured on the local desktop.

One card, Vulkan

# Cited shape — not measured on the local desktop.
# Replace /path/to/nvme/models with the NVMe directory that holds the GGUF.
# Image tag from llama.cpp docker.md. Host publish stays on 127.0.0.1.
services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    devices:
      - /dev/dri
    group_add:
      - video
      - render
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - /path/to/nvme/models:/models:ro
    command:
      - -m
      - /models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf
      - -ngl
      - "99"
      - -c
      - "262144"
      - --cache-type-k
      - f16
      - --cache-type-v
      - f16
      - --host
      - 0.0.0.0
      - --port
      - "8080"

Two copies

# Two copies — one render node per container. Not measured on the local desktop.
# Confirm ls -l /dev/dri on the host. AMD documents renderD128 / renderD129.
# Do not invent --tensor-split or a 2× tok/s. See /multi-gpu.
services:
  llama-a:
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    devices:
      - /dev/dri/renderD128
    group_add:
      - video
      - render
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - /path/to/nvme/models:/models:ro
    command: ["-m", "/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf", "-ngl", "99", "-c", "262144", "--cache-type-k", "f16", "--cache-type-v", "f16", "--host", "0.0.0.0", "--port", "8080"]
  llama-b:
    image: ghcr.io/ggml-org/llama.cpp:server-vulkan
    devices:
      - /dev/dri/renderD129
    group_add:
      - video
      - render
    ports:
      - "127.0.0.1:8081:8080"
    volumes:
      - /path/to/nvme/models:/models:ro
    command: ["-m", "/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf", "-ngl", "99", "-c", "262144", "--cache-type-k", "f16", "--cache-type-v", "f16", "--host", "0.0.0.0", "--port", "8080"]

RAM lines (Ubuntu-measured)

Use the calculator before you download a second quant. The two labels stay Minimum system RAM needed and RAM needed at full context. The house 35B peaks below are Ubuntu, not Docker or Podman.

Open in Stack Lab

Stack Lab remains Ubuntu-first. The button is the allowlisted scheme. There is no container package and no public download.

Sources

• llama.cpp — docker.md (official GHCR images)
• llama.cpp — llama-server README (OpenAI-shaped /v1)
• AMD — Run ROCm Docker containers (ROCm 10.1.0)
• AMD — Linux prerequisites (render and video groups)
• Docker — docker run --device
• Docker — publish a port on a host IP
• Docker — --security-opt (seccomp)
• Podman — podman-run --device, keep-groups, SELinux :z / :Z