The ROCm stack

The picture

ROCm is an ecosystem of compilers, runtimes, and libraries — not a single binary.^[1]For a home box, the useful mental model is six layers. Each one can be “fine” while the layer above it is the thing that is broken.

flowchart TB
  subgraph frameworks [Frameworks]
    PT[PyTorch ROCm]
    LC[llama.cpp HIP]
    VL[vLLM]
  end
  subgraph libraries [Libraries]
    BL[rocBLAS / hipBLASLt]
    MI[MIOpen]
    RC[RCCL]
    WM[rocWMMA]
    CK[Composable Kernel]
  end
  subgraph language [Language and compiler]
    HIP[HIP]
    LLVM[LLVM / clang]
    COMGR[comgr]
  end
  subgraph runtime [Runtime]
    ROCR[ROCr HSA runtime]
  end
  subgraph kernel [Kernel]
    AMDGPU[amdgpu]
    KFD[KFD]
  end
  PT --> BL
  PT --> MI
  VL --> RC
  LC --> HIP
  PT --> HIP
  VL --> HIP
  BL --> HIP
  MI --> HIP
  RC --> HIP
  WM --> HIP
  CK --> HIP
  HIP --> LLVM
  LLVM --> COMGR
  HIP --> ROCR
  ROCR --> KFD
  ROCR --> AMDGPU
  KFD --- AMDGPU

1. Kernel — amdgpu plus the compute side (KFD).
2. Runtime — ROCr, AMD’s user-mode HSA runtime.
3. Language — HIP on top of ROCr.
4. Compiler — LLVM/clang, plus comgr (code-object manager).
5. Libraries — rocBLAS, hipBLASLt, MIOpen, RCCL, rocWMMA, Composable Kernel.
6. Frameworks — PyTorch ROCm, llama.cpp HIP, vLLM.

Kernel driver: amdgpu and KFD

The in-tree Linux driver for modern AMD GPUs is amdgpu. It covers GCN, RDNA, and CDNA devices.^[2]That is the same driver that puts pixels on a monitor and the driver ROCm compute sits on. Mixing a leftover DKMS tree, a distro module, and an old runfile is how people get a black screen. One source of amdgpu. Confirm with lsmod anddmesg before you blame userspace. Thetroubleshooting page starts there.

KFD (Kernel Fusion Driver) is the kernel-side compute interface. AMD’s ROCR docs describe ROCr as talking to graphics hardware driven by the AMDGPU driver set and theROCK kernel driver.^[3]On a current Linux tree that compute path lives as amdkfd next toamdgpu. User-mode code opens /dev/kfd, learns the topology, and submits queues. If /dev/kfd is missing, no amount of pip will save you.

ROCr runtime

ROCr is AMD’s implementation of the HSA runtime: a thin user-mode API that exposes the GPU to programs above it.^[3]HIP sits on this layer. Tools like rocminfo are how you ask ROCr what agents it can see. “Agent” here means a compute device ROCr enumerated — usually your discrete GPU, and sometimes an iGPU you did not mean to use.

HIP

HIP is a C++ runtime API and kernel language for AMD GPUs. Host code allocates buffers, copies data, launches kernels, and synchronizes. Device code is the kernel that runs on the GPU.^[4]HIP is the layer llama.cpp’s HIP backend, PyTorch ROCm, and vLLM all share. It is also the layer Vulkan llama.cpp does not share — Vulkan talks to a graphics ICD instead. That split is the whole Vulkan vs HIP page.

Compiler: LLVM/clang and comgr

ROCm compiles GPU code with an LLVM/clang toolchain (you will see amdclang++ /hipcc). ROCm 10.1 moves that compiler onto LLVM 24.^[6]The ISA name in the binary — gfx1201, gfx1100, and so on — is an LLVM target. A binary built for the wrong target is thehipErrorNoBinaryForGpu class of failure. See gfx IDs.

comgr (Code Object Manager) is the library that compiles, links, and inspects GPU code objects. HIP’s runtime compiler (HIPRTC) uses comgr internally; AMD describes it as the API for linking, compiling, and inspecting those objects.^[5]You rarely invoke comgr by name. You meet it when a JIT or a wheel’s embedded kernels do not match the ROCm you installed.

Libraries

Frameworks do not implement every GEMM themselves. They call into ROCm libraries. The ones that show up in home-lab LLM logs:

• rocBLAS — ROCm’s BLAS, implemented in HIP and optimized for AMD GPUs.^[7]
• hipBLASLt — GEMM with a more flexible API than classical BLAS.^[8]Fine-tune writeups sometimes set TORCH_BLAS_PREFER_HIPBLASLT=0 when this path misbehaves; treat that as a dated workaround, not a default.
• MIOpen — AMD’s open-source deep-learning primitives library (convolutions, fusion, autotune).^[9]
• RCCL — multi-GPU / multi-node collectives over PCIe and xGMI.^[10]Two cards do not “just” double tokens. RCCL is the layer that has to work first. Our daily driver stays on one GPU.
• rocWMMA — C++ headers for mixed-precision matrix multiply-accumulate on specialized GPU matrix cores.^[11]
• Composable Kernel (CK) — a programming model for performance-critical ML kernels in HIP C++. The 10.1 blog notes CK work on attention on Radeon and more quantized matmul types.^[12][6]

Frameworks

• PyTorch ROCm — AMD documents a ROCm-enabled PyTorch install. Use that page (or the ROCm CLI managed env), not default CUDA PyPI.^[13]
• llama.cpp HIP — same GGUF runner as Vulkan, compiled withGGML_HIP against a ROCm prefix. The project lives at ggml-org/llama.cpp.^[15]Walkthrough: run a local LLM.
• vLLM — AMD documents a ROCm vLLM path. It is a serve stack (batching, many clients), not this site’s consumer default.^[14]Pins move: see versions.

Fit a quant before you climb this stack: calculator. Pick an engine on the backend chooser.