Run a local Qwen (llama.cpp)

Goal: official Qwen instruct generating on the R9700, with rocm-smi (or amd-smi) showing GPU use. llama.cpp first. Ollama is optional and still rocky on gfx1201. vLLM lives in the advanced section of Serve Qwen, not here.

Do this in order

1. ROCm installed and verified if you want HIP later
2. Build llama.cpp — Vulkan first (what we run), HIP if you already have a clean /opt/rocm
3. Run one official instruct GGUF with -ngl 99
4. Confirm VRAM moves
5. Optionally expose llama-server on 127.0.0.1:8080 — then attach a harness

sudo apt install -y build-essential cmake git libvulkan-dev

Fedora/RHEL: same idea, different package names. Backend pick: chooser.

Path A — Vulkan (daily driver on our rig)

This is not a consolation prize. See Vulkan is an on-ramp.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build-vk -DGGML_VULKAN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build-vk -j"$(nproc)"
./build-vk/bin/llama-cli --list-devices

Pick the RADV GFX1201 (or AMDVLK) device that is the R9700, not the iGPU. Set GGML_VK_VISIBLE_DEVICES / VK_ICD_FILENAMES only after --list-devices shows the confusion.

Official GGUF

Daily-driver file: unsloth/Qwen3.6-35B-A3B-GGUF · Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf (publisher listing 26.6 GB). Pin a revision.

mkdir -p ~/models
# huggingface-cli download unsloth/Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf

Run all layers on the GPU

TODO: exact command line / measured VRAM from rig owner.

./build-vk/bin/llama-cli \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  -p "Say hello from a Radeon AI PRO R9700." \
  -n 64

-ngl 99 means “offload what fits.” The daily driver on our rig uses 0% system RAM offload. If layers spill to RAM, stop and open the calculator before you drop below Q5.

watch -n1 rocm-smi
# ROCm 10.1 also points people at amd-smi — either meter moving is the point

VRAM should rise. If it stays empty, you are on CPU — stop. Paste the log.

llama-server (/v1)

# TODO: exact command line / measured VRAM from rig owner
./build-vk/bin/llama-server \
  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \
  -ngl 99 \
  -c 262144 \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

Clients use http://127.0.0.1:8080/v1. Do not bind 0.0.0.0 unless you mean to. More serve detail: Serve Qwen.

Path B — HIP (ROCm)

When rocminfo already lists gfx1201 and you want the HIP tree:

cmake -B build \
  -DGGML_HIP=ON \
  -DAMDGPU_TARGETS=gfx1201 \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"

CMake names moved (GGML_HIP now; older posts say LLAMA_HIPBLAS). If configure fails, read the repo’s HIP docs for the commit you cloned. hipErrorNoBinaryForGpu means this binary was not built for gfx1201 — troubleshooter.

export HIP_PATH=/opt/rocm
# or /opt/rocm/core-10.0 — ls first
export PATH="/opt/rocm/bin:$PATH"
./build/bin/llama-cli --list-devices

You want the Radeon AI PRO R9700, not only an iGPU. Same GGUF flags as Path A.

Path C — Ollama (optional)

Official Ollama ROCm is still rocky on gfx1201. Linux Docker looked for TensileLibrary_lazy_gfx1201.dat in the wrong directory (#12908). Windows R9700 ROCm filtered the device to CPU; Vulkan worked (#14686).

ollama --version
ollama run <a-small-official-tag>
rocm-smi
ollama ps

If VRAM never moves, go back to llama.cpp. Do not set random HSA_OVERRIDE_* on a card current ROCm already lists.

What “it works” looks like

• Tokens stream at interactive speed (you will know CPU)
• rocm-smi / amd-smi VRAM moves; daily driver should not be crawling through system RAM
• A second prompt still works

Then: raise -c, or serve /v1 for a harness. Compare HIP vs Vulkan on the same GGUF if you like — notes, not leaderboards. Abliterated weights come after this pipe is boring.