When it breaks

Troubleshoot

Start from what you see, in plain English. Each symptom has quick checks, ranked causes, and a fix. Commands are the ones this site already cites. We do not invent error strings or tok/s. As of 2026-10-13.

Have a log? Paste it. Longer order-of-operations: Troubleshooting. First-run: Ubuntu setup.

Paste a log First-run setup ROCm problems

Open in Stack Lab

If the app is not installed, packages are not published yet. There is no public download URL. Read the Stack Lab install notes . If Stack Lab is already running on this machine, open the local app .

15 symptoms in 6 groups. Share a filter with ?q=.

GPU not detected

rocminfo or the Vulkan device list shows no agent, or the wrong gfx.

rocminfo shows no GPU

rocminfo lists only the CPU, or there is no GPU agent. Do not download a 20 GB GGUF until a discrete agent appears.

Quick checks

  1. lspci

    Good output. An AMD VGA or 3D controller line. No AMD line means power, slot, or the card never initialized. 

  2. lsmod | grep amdgpu

    Good output. The amdgpu module is listed. 

  3. dmesg | grep -i amdgpu

    Good output. The card initialized. MODE1 / MES lines belong on the hang flow, not here. 

  4. rocminfo

    Good output. A GPU agent with a marketing name and a gfx ID. CPU-only output is a miss.

Likely causes

  1. The card never initialized (cable, slot, or firmware).
  2. The amdgpu module is missing (Secure Boot / DKMS / distro module).
  3. Userspace is the Ubuntu archive rocm tree, or mixed with AMD’s repo.
  4. The process cannot open /dev/kfd or /dev/dri (groups).

The fix

Work top down. lspci first, then the module, then rocminfo. New groups need a full logout. Do not sudo inference to dodge permissions. Ubuntu 24.04+ is the measured path on this site.

ROCm 10.1.0 install — post-install device access

Wrong gfx or hipErrorNoBinaryForGpu

rocminfo or llama-cli --list-devices shows gfx1100 when you expected gfx1201, or HIP logs hipErrorNoBinaryForGpu / “No kernel image is available.”

Quick checks

  1. rocminfo | grep -E 'Marketing Name|gfx'

    Good output. R9700 GPU agent is gfx1201. gfx1100 is a different card (RX 7900 XTX class), not a misspelling. 

  2. llama-cli --list-devices

    Good output. The discrete RADV / HIP device, not the RAPHAEL iGPU.

Likely causes

  1. You are looking at a different GPU or the iGPU.
  2. The HIP wheel, container, or llama.cpp HIP build was compiled for another AMDGPU_TARGETS list.
  3. Mixed ROCm libraries shadow the gfx you actually have.
  4. An unofficial HSA_OVERRIDE_GFX_VERSION is papering over a missing kernel image.

The fix

Confirm the gfx on the discrete agent. Rebuild or reinstall with that target (-DAMDGPU_TARGETS=gfx1201 / -DGGML_HIP=ON for llama.cpp). Do not set a random HSA_OVERRIDE_GFX_VERSION on a card current ROCm already lists. If HIP still has no binary, use llama.cpp Vulkan tonight.

HIP error codes — hipErrorNoBinaryForGpu (209) · Ollama

Install problems

ROCm userspace vs kernel, groups, Secure Boot / DKMS, mixed apt repos.

Not in render or video / Permission denied on /dev/kfd

Userspace is installed but the process cannot open /dev/kfd or /dev/dri. Logs say Permission denied, no GPU found, or you are not a member of render/video.

Quick checks

  1. sudo usermod -aG render,video "$USER"

    Good output. The command exits 0. Groups do not apply until a full logout or reboot. 

  2. groups | grep -E 'render|video'

    Good output. Both render and video appear in the current session after logout. 

  3. ls -l /dev/kfd /dev/dri

    Good output. The device nodes exist. Containers also need those nodes and matching GIDs.

Likely causes

  1. The user was never added to render and video.
  2. You opened a new shell without leaving the session.
  3. A container is missing --device /dev/kfd --device /dev/dri --group-add video.

The fix

Add the groups, then log out completely. Do not sudo inference to dodge permissions. Official Linux installs put the user in render and video.

ROCm 10.1.0 install — post-install device access

Distro apt ROCm mixed with AMD’s repo

gfx1201 “has 2 ISAs but can only support a single ISA,” or rocminfo never lists the card after you installed both Ubuntu archive rocm and AMD’s repo.

Quick checks

  1. which rocminfo rocm-smi

    Good output. The binaries resolve. Then ldd $(which rocminfo) should prefer /opt/rocm, not archive HIP. 

  2. rocminfo | grep -E 'Marketing Name|gfx'

    Good output. One GPU ISA that matches the card. A shadowed 5.7-era or 7.1-era archive tree will not list gfx1201 cleanly.

Likely causes

  1. Canonical/Ubuntu rocm or libamdhip64-dev is loaded instead of AMD’s 10.x install.
  2. You installed every amdrocm-core-sdk family package — TheRock wants one gfx family.
  3. Kernel vs userspace drifted after apt upgrade.

The fix

Pick one source. Purge archive rocm, libamdhip64-dev, and libhsa-runtime* before adding AMD’s 10.0/10.1 repo. Re-read ldconfig so /opt/rocm wins. Ubuntu 24.04+ is the measured path on this site. If a stack died after apt upgrade, bisect kernel vs ROCm userspace before you reinstall.

ROCm

Secure Boot blocks the amdgpu DKMS module

lsmod | grep amdgpu is empty. rocminfo sees only the CPU. dmesg mentions Secure Boot, Lockdown, module verification failed, or DKMS key rejected.

Quick checks

  1. lsmod | grep amdgpu

    Good output. The module is listed. Empty output with Secure Boot on is the classic miss. 

  2. rocminfo

    Good output. A GPU agent. CPU-only here usually means the module never loaded.

Likely causes

  1. An unsigned DKMS amdgpu will not load when Secure Boot is on.
  2. Distro module vs DKMS mismatch after a kernel upgrade.

The fix

Enroll a MOK as your distro documents, disable Secure Boot, or keep a signed distro module and install userspace only (--no-dkms when that is the method you chose). Do not install a third driver to “fix” a black screen.

ROCm 10.1.0 install methods (DKMS / distro module)

Model will not load

Out of VRAM, context too big, wrong quant, or system RAM too small for offload.

Out of VRAM at long context

Load fails with out of memory, ggml_gallocr, HIP/CUDA out of memory, or a KV-cache allocate. Short context works; long context does not.

Quick checks

  1. rocm-smi --showmeminfo vram --showuse

    Good output. VRAM rises when you load. An empty table or idle VRAM at “OOM” means you are not looking at this card.

Likely causes

  1. Context grows the KV cache. This site’s estimate is 2 × kv_layers × kv_heads × head_dim × context × bytes.
  2. Hybrid Qwen cards only grow KV on full-attention layers. A 262K f16 KV on Qwen3.6-35B-A3B is a large add on top of the publisher GGUF.
  3. Partial offload plus a long context overflows the card you actually picked.

The fix

Open /out-of-vram, then the calculator. Prefer a smaller context or q8_0/q4_0 KV before you drop the weight quant. House recipes start at Q4. Q3 is a labeled estimate on the calculator — do not invent a Q3 to “make it fit.” Daily driver on our rig is 262K f16 KV with 0% RAM offload on one 32 GB card.

Quant + Hardware Calculator (KV formula; estimates only)

System RAM too small for offload

The GGUF is larger than VRAM. The process dies, swaps, or never finishes mapping weights. Flash-class files usually need DDR5 expert offload even with two 32 GB cards.

Quick checks

  1. rocm-smi --showmeminfo vram --showuse

    Good output. VRAM is in use if any layers landed on the GPU. Idle VRAM with a huge RSS is offload or CPU.

Likely causes

  1. llama.cpp -ngl 99 means “offload what fits”; leftover layers spill to DDR5.
  2. Flash UD-Q6_K_XL is a ~169 GB publisher listing — expect RAM / expert offload.
  3. The 35B daily driver on our rig uses 0% RAM offload. Do not treat that as a Flash recipe.

The fix

Size the offload on the calculator and the Flash RAM offload page. Copy the launch shape. Exact -ot regex and --n-cpu-moe N from the rig owner are not recorded — do not invent flag values. Q3 Flash rows are estimates unless a card is measured.

Flash RAM offload — kits and launch shape · Calculator — estimates only

Wrong quant for the card

The file is too big for the VRAM you have, or you reached for a Q3 to “make it fit” without an estimate. House recipes start at Q4.

Quick checks

  1. rocm-smi --showmeminfo vram

    Good output. Peak VRAM is below the card label after load. Overflow is a fit problem, not a “slow GPU.”

Likely causes

  1. Weights plus KV plus overhead exceed the card.
  2. You compared a publisher GGUF size to VRAM and ignored KV.
  3. Consumer 16–24 GB sheets are calculator estimates — not measured on the local desktop.

The fix

Use the calculator against the card you own. Q3 is a labeled estimate. Q1 and Q2 stay off. Do not invent a Q3 to force a 35B onto a 16 GB card. R9700 32 GB is the measured house card for Qwen3.6-35B-A3B UD-Q5_K_XL.

Quant + Hardware Calculator (estimates only)

Slow speed

Wrong Vulkan GPU, CPU fallback, two-card split tax, PCIe / BIOS.

Vulkan or HIP attached to the iGPU

Tokens are glacial. llama-cli --list-devices shows RAPHAEL / “AMD Radeon Graphics” first. Marketing strings are not enough.

Quick checks

  1. llama-cli --list-devices

    Good output. Pick RADV GFX1201 / the discrete index, not the iGPU. 

  2. rocm-smi --showuse

    Good output. Compute and VRAM move on the discrete card while you generate.

Likely causes

  1. Many boxes have a RAPHAEL/Strix iGPU plus the R9700. The first HIP or Vulkan device may be the iGPU.
  2. Visibility env vars were never set, or were set to the wrong index.

The fix

List devices, then pin the discrete index. Set HIP_VISIBLE_DEVICES / GGML_VK_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES only after you see the confusion. Isolation docs are AMD GPU isolation.

AMD — GPU isolation environment variables

Generation is glacial and VRAM is empty

The process “runs” but rocm-smi stays idle and VRAM never rises. This is CPU fallback, not a slow Radeon.

Quick checks

  1. rocm-smi --showproductname --showmeminfo vram --showuse

    Good output. Compute and VRAM move. An empty table while you generate means the process is not on the GPU. 

  2. llama-cli --list-devices

    Good output. The discrete device is visible to this binary.

Likely causes

  1. The binary is a CUDA-shaped wheel (torch.version.hip is None) or a CPU llama.cpp build.
  2. Ollama ROCm on R9700 Windows logged “filtering device which didn't fully initialize” and ran on CPU.
  3. HIP configure failed and you kept generating anyway.

The fix

Stop. Use llama.cpp you compiled (Vulkan first on this site). Confirm rocminfo, then --list-devices, then watch rocm-smi. Consumers should stay on llama.cpp. Uninstall a CUDA wheel from default PyPI if that is how PyTorch arrived.

Ollama · AMD — vLLM on ROCm (not default PyPI)

Two cards slower than one

A 35B that fits one 32 GB card is slower when split across two. That is a measured topology tax on the local desktop, not a broken second card.

Quick checks

  1. llama-cli --list-devices

    Good output. Both discrete GPUs are visible. Pin one process per card when you want two models, not a split.

Likely causes

  1. Official llama.cpp Vulkan overnight cards — one-card decode is faster than a split of the same Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL file.
  2. --split-mode layer (-sm layer) is the documented llama.cpp default for multi-GPU and is slower on this 35B than one card.
  3. A second card is for models that do not fit one 32 GB card (house example — Qwen 3.8 Flash UD-Q6_K_XL).

The fix

Keep the 35B daily driver on one R9700. Use two cards for a file that does not fit. Read /multi-gpu and the one-vs-two compare before you chase clocks. We do not invent a Flash split tok/s.

Dual / multi-GPU — cited local desktop 35B split tax · Compare 35B — one GPU vs split from run cards

PCIe lanes, BIOS, or power state

The card is seen but feels starved — wrong slot, mixed x16/x4, Above 4G / ReBAR off, or you are tuning C-states from an Instinct blog post.

Quick checks

  1. lspci

    Good output. The AMD VGA/3D device is present. Slot width and generation are a firmware map, not a tok/s we invent.

Likely causes

  1. AMD’s AM5 chipset table lists CPU graphics as 1×16 or 2×8. ROCm on Radeon multi-GPU wants identical width (x16+x16 or x8+x8). A mixed x16+x8 pair is listed as not recommended.
  2. ROCm BAR guidance says to enable Above 4G Decoding. ReBAR / Smart Access Memory is AMD’s name for a full CPU view of graphics memory.
  3. Instinct OS-tuning docs discuss restricting CPU C-states. That is not a measured R9700 desktop recipe — we have not measured llama.cpp tok/s with C-states on versus off on the local desktop.

The fix

Read /bios before you file “ROCm is broken” on a firmware miss. Keep GPUs on CPU-connected slots. Do not invent a motherboard for local desktop. Power and C-state switches stay labeled Instinct guidance, not a house speed fix.

AMD Socket AM5 Chipset · ROCm on Radeon — mGPU setup · ROCm — BAR access limits · Instinct OS tuning (C-states)

Crashes and hangs

MODE1 / MES reset, unstable DDR5, driver or kernel regressions.

GPU hang / MODE1 reset

The desktop dies or compute stops. dmesg shows MODE1 reset, GPU smu mode1 reset, MES unrecoverable, failed to suspend gangs, or VRAM is lost due to GPU reset.

Quick checks

  1. dmesg | grep -iE 'amdgpu|MODE1|MES'

    Good output. A clean boot has no MODE1 / MES hang. After a reset, VRAM is lost — save the log before you reboot.

Likely causes

  1. The amdgpu driver declared the GPU hung and issued a MODE1 (SMU) reset. On gfx12 this has shown up after MES failed to recover a compute queue — including during ROCm HSA selftests on linux-next (July 2026 amd-gfx thread).
  2. Fine-tune / training on ROCm 10.0 has a known-issue class on the R9700. First mitigation is TORCH_BLAS_PREFER_HIPBLASLT=0. That is not an inference daily-driver issue.
  3. A kernel upgrade vs ROCm userspace drift.

The fix

Save dmesg. If it started after a kernel upgrade, bisect kernel vs ROCm userspace before you reinstall. For inference, switch to llama.cpp Vulkan and see if the hang follows HIP. Do not apply a training workaround to inference blindly.

amd-gfx — GPU reset / MODE1 on gfx12 HSA tests (Jul 2026) · ROCm 10.0.0 release notes (KI class; confirm on the live page)

Unstable DDR5 / EXPO

Kernel oops, tainted overnight numbers, or hangs that follow the DIMM profile rather than the GPU driver.

Quick checks

  1. dmesg | grep -i amdgpu

    Good output. Separate a GPU MODE1 line from a host memory oops. The 2026-10-12 sweep oops was a RAM event, not a MODE1 row.

Likely causes

  1. First-party local desktop note — DDR5-5200 later proved unstable. A kernel oops at 00:14 CT aborted the overnight sweep. All 33 cards predate that oops.
  2. AMD EXPO is AMD’s DDR5 overclocking-profile standard for Ryzen on socket AM5. EXPO is overclocking, not the JEDEC default.
  3. The motherboard model and whether EXPO was enabled on the local desktop are not published.

The fix

Treat EXPO / XMP as an overclock profile. A two-DIMM kit profile is not a four-DIMM guarantee. We do not invent a frequency as the only cause. The 33-card sweep stays labeled measured, pending re-verify after the RAM fix. Read How we test and the benchmark caveat.

How we test — amdai-bench cards and the sweep caveat · Benchmarks — 33 cards predate the DDR5-5200 oops · AMD EXPO Technology for AMD Ryzen Processors for Socket AM5

Windows notes

What this site will say about Windows ROCm — and what it will not.

Windows ROCm fails or falls back to CPU

Ollama or HIP on Windows detects gfx1201, then logs “filtering device which didn't fully initialize” and runs on CPU. Vulkan on the same box can still generate.

Quick checks

  1. llama-cli --list-devices

    Good output. A Vulkan device list on Windows. This site does not publish a Windows rocminfo recipe.

Likely causes

  1. Ollama 0.17.7 on Windows 11 with an R9700 detected gfx1201, then filtered the device and ran on CPU (#14686). The bundled ROCm (amdhip64_6.dll / 6.x tensiles through gfx1151) had no gfx1201 kernels. Vulkan (OLLAMA_VULKAN=1) worked.
  2. HSA_OVERRIDE_GFX_VERSION still failed to initialize ROCm on that Windows report.

The fix

On Windows, use Vulkan llama.cpp (or Ollama’s Vulkan path) and do not expect a full ROCm userspace. Linux + llama.cpp is the path this site documents for HIP. Windows steps on /setup/windows stay labeled cited, not measured on the local desktop. We do not publish a Windows tok/s card.

Ollama

Still stuck?

Do not invent a fix. Write a cited report with rocminfo, kernel, and the exact log. Paste the string in the error matcher first — a miss is a miss.

Write a report Forum (off on this build) ROCm problems

The forum is off on this static build. File a cited report instead, or read the rules and come back when the forum is on.

Next steps

Real pages on this site. We do not invent a next click.

How we test — Measured cells cite official amdai-bench/run@1 cards. Estimates stay labeled. Q3 is estimate-only. Q1/Q2 stay off.

Last updated

esc

Type to filter. Enter opens the first match. Full-text search lives on /search.