What it is
Machine Learning Compilation for LLMs: compile a model into a runtime (CLI / Python / mobile) via TVM. Backends include CUDA, Metal, ROCm, Vulkan.
Architecture in plain English
Compile once, run the artifact. Not a drop-in GGUF server. Vulkan is the wide-device path; ROCm is a separate wheel line.
What it does well
Shipping a compiled runtime to many devices; research compilation.
What it does badly
Tracking this site’s weekly GGUF; ROCm 10.1 / gfx1201 as a documented pin.
ROCm / Radeon
Official GPU page says MLC LLM only supports ROCm 6.1/6.2. Nightly wheels are mlc-llm-nightly-rocm61 / rocm62. That is not the 10.0/10.1 userspace on this site. Vulkan is documented as available in Linux packages.
Vulkan. Yes — documented. A more honest AMD on-ramp here than their ROCm 6.2 wheel on a 10.1 box.
Supported quants
MLC’s own compiled quants. Floor is still Q4-class. Do not chase IQ2 artifacts.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# Official ROCm wheels (6.1 / 6.2) — likely wrong userspace on ROCm 10.1.
# pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-rocm62 mlc-ai-nightly-rocm62
# Vulkan loader (their Linux note):
# conda install -c conda-forge gcc libvulkan-loader
# We do not have a gfx1201 recipe. Treat as not documented for this rig.
Run
# mlc_llm chat / serve — follow current docs. No R9700 command line here.
Tuning flags
If you try Vulkan, pick the discrete GFX1201 device, not the iGPU. Do not mix a 6.2 MLC wheel with a 10.1 HIP prefix and call it “ROCm worked.”
When to choose it
Rarely.
You have a reason to compile, not chat.