llama.cpp

Docs checked 2026-10-12. Do not invent a release pin.

Consumer default — GGUF, Vulkan or HIP, one user

What it is

C/C++ GGUF runner from ggml-org. You compile a backend (Vulkan or HIP), load a quantised file, and optionally serve an OpenAI-shaped /v1 on llama-server (default 127.0.0.1:8080).

Architecture in plain English

ggml graphs: weights stay on the device you chose, KV cache is sized by -c and --cache-type-k/v. llama-server is the same process as the CLI, plus HTTP. It is not a Python serving stack.

What it does well

First Qwen, debugging, a loopback /v1 for a harness, tight control of context and KV type.

What it does badly

Many concurrent clients, continuous batching, “just pip install.”

ROCm / Radeon

First-class if you build it. HIP needs a gfx1201 userspace. Vulkan needs a working ICD (RADV is the usual Linux path).

Vulkan. Yes — GGML_VULKAN. Daily driver on our rig.

Supported quants

GGUF Q4–Q8 and UD dynamic. Floor is Q4. Never Q1/Q2/Q3 or IQ1–IQ3.

House floor is still Q4. Prefer Q5. See quants and thecalculator.

Install on an R9700

# Vulkan (what we run for the 35B)
sudo apt install -y build-essential cmake git libvulkan-dev
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build-vk -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-vk -j"$(nproc)"
./build-vk/bin/llama-cli --list-devices
# Pick RADV GFX1201 / the R9700, not the iGPU.

# HIP sibling (when rocminfo is clean)
cmake -B build-hip -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201 -DCMAKE_BUILD_TYPE=Release
cmake --build build-hip -j"$(nproc)"
# Prefix may be /opt/rocm/core-10.0 — ls first.

Run

# TODO: exact daily-driver command line / measured VRAM from the rig owner.
./build-vk/bin/llama-cli \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  -p "Say hello from a Radeon AI PRO R9700." \
  -n 64
# Server (harnesses curl this first):
./build-vk/bin/llama-server -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --host 127.0.0.1 --port 8080

Tuning flags

-ngl 99 means “offload what fits.” Daily driver uses 0% RAM offload. If layers spill, open /calculator before you drop below Q5. GGML_VK_VISIBLE_DEVICES / VK_ICD_FILENAMES only after --list-devices. Missing gfx1201 HIP binaries show up as hipErrorNoBinaryForGpu.

When to choose it

Tokens on one card tonight, or a /v1 for Hermes. 35B daily driver lives here (Vulkan). Measured on our rig, 2026-10: UD-Q5_K_XL 100.8 / ~88 / ~50 tok/s.

Start here. Always.