{
  "schema_version": "1.0",
  "as_of": "2026-10-12",
  "href": "https://amdai.net/backends",
  "default_id": "llamacpp",
  "default_name": "llama.cpp",
  "notes": [
    "Consumer default is llama.cpp. Vulkan is what we run for the 35B daily driver.",
    "First-party 35B (one R9700, llama.cpp, f16 KV). Measured on our rig, 2026-10: UD-Q5_K_XL 102.39 tok/s short / 90.52 at 32K / 66.43 at 128K / 44.4 at 256K; Q5_K_M 109.73 tok/s (24.68 GB VRAM); Q4_K_M 117.69 tok/s decode, 1902.1 tok/s prefill at 32K.",
    "Q3 is listed as an estimate or community result unless measured. Q1/Q2 and IQ1/IQ2 stay off, including draft/MTP.",
    "curl /v1 before you debug a harness. Do not let a desktop app download a CUDA llama.cpp on a Radeon."
  ],
  "engines": [
    {
      "id": "llamacpp",
      "name": "llama.cpp",
      "short": "Consumer default — GGUF, Vulkan or HIP, one user",
      "verdict": "default",
      "default": true,
      "used_on_our_rig": true,
      "href": "https://amdai.net/backends/llamacpp",
      "homepage": "https://github.com/ggml-org/llama.cpp",
      "well": "First Qwen, debugging, a loopback /v1 for a harness, tight control of context and KV type.",
      "badly": "Many concurrent clients, continuous batching, “just pip install.”",
      "when": "Start here. Always.",
      "run": "# not recorded: exact daily-driver command line / measured VRAM from the rig owner.\n./build-vk/bin/llama-cli \\\n  -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf \\\n  -ngl 99 \\\n  -c 262144 \\\n  --cache-type-k f16 \\\n  --cache-type-v f16 \\\n  -p \"Say hello from a Radeon AI PRO R9700.\" \\\n  -n 64\n# Server (harnesses curl this first):\n./build-vk/bin/llama-server -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf --host 127.0.0.1 --port 8080\n"
    },
    {
      "id": "exllama",
      "name": "ExLlama / TabbyAPI",
      "short": "EXL3/EXL2 server — ROCm is a side path",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/exllama",
      "homepage": "https://github.com/theroyallab/tabbyAPI",
      "well": "EXL3/EXL2 on NVIDIA. Fast local OpenAI API if the kernels exist for your GPU.",
      "badly": "A Radeon-first daily driver. Consumer RDNA FlashAttention/paged-attention is not first-class in their own issues.",
      "when": "You know you need ExLlama and will build against a ROCm PyTorch that lists gfx1201 — then tell the forum what happened.",
      "run": "# TabbyAPI start scripts are in their repo. No R9700 command line here.\n"
    },
    {
      "id": "ktransformers",
      "name": "KTransformers / ik_llama.cpp",
      "short": "CPU-heavy MoE forks — AMD is not the maintained path",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/ktransformers",
      "homepage": "https://github.com/kvcache-ai/ktransformers",
      "well": "Huge MoE on a lot of RAM + NVIDIA (ik) or a 7900 XTX beta (KT).",
      "badly": "gfx1201 daily driver; expecting their IQ1–IQ3 tricks (banned on this site).",
      "when": "Almost never. Use llama.cpp (or Strata for Flash).",
      "run": "# python ktransformers/local_chat.py ...  (their DeepSeek example)\n# We have no R9700 command line. Not documented for gfx1201.\n"
    },
    {
      "id": "lemonade",
      "name": "Lemonade Server",
      "short": "AMD llama.cpp wrapper — Vulkan or ROCm on :13305",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/lemonade",
      "homepage": "https://lemonade-server.ai/",
      "well": "An AMD-promoted wrapper with an explicit Vulkan vs ROCm switch, if a GPU meter moves.",
      "badly": "Matching this site’s compile flags or benches; silent CPU; treating /v1 like llama-server.",
      "when": "You want AMD’s server and you will read the backend log.",
      "run": "lemonade status\n# lemonade pull unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_XL\n# lemonade load unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_XL --ctx-size 262144 --llamacpp vulkan\ncurl -s http://127.0.0.1:13305/api/v1/models\n"
    },
    {
      "id": "lmstudio",
      "name": "LM Studio",
      "short": "Desktop app — Vulkan or a bundled ROCm runtime",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/lmstudio",
      "homepage": "https://lmstudio.ai/",
      "well": "A GUI on Windows or a first desktop when you refuse to compile — if a GPU meter moves.",
      "badly": "Reproducing this site’s HIP/ROCm notes; silent CPU.",
      "when": "Windows-only, or a desktop, and you accept “no data” for tok/s.",
      "run": "# Documented server: http://127.0.0.1:1234/v1\ncurl -s http://127.0.0.1:1234/v1/models\n"
    },
    {
      "id": "mlc",
      "name": "MLC-LLM",
      "short": "Compile-then-run — official ROCm pin is 6.1/6.2",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/mlc",
      "homepage": "https://llm.mlc.ai/",
      "well": "Shipping a compiled runtime to many devices; research compilation.",
      "badly": "Tracking this site’s weekly GGUF; ROCm 10.1 / gfx1201 as a documented pin.",
      "when": "You have a reason to compile, not chat.",
      "run": "# mlc_llm chat / serve — follow current docs. No R9700 command line here.\n"
    },
    {
      "id": "ollama",
      "name": "Ollama",
      "short": "Convenience wrapper — gfx1201 is still rocky",
      "verdict": "caution",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/ollama",
      "homepage": "https://ollama.com/",
      "well": "Convenience, if a GPU meter moves.",
      "badly": "gfx1201 as the first backend; Windows ROCm; silent CPU fallback.",
      "when": "You want the wrapper and you will read the backend log.",
      "run": "ollama serve\n# ollama run <tag>   # only after rocm-smi / amd-smi moves\ncurl -s http://127.0.0.1:11434/api/tags\n"
    },
    {
      "id": "pytorch",
      "name": "PyTorch / Transformers",
      "short": "Framework loop — not a chat server",
      "verdict": "ok",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/pytorch",
      "homepage": "https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/frameworks/pytorch/install.html",
      "well": "Fine-tune, custom heads, debugging a ROCm wheel.",
      "badly": "Daily chat; long 262K context as a first script.",
      "when": "You need a training loop or a wheel check, not a harness.",
      "run": "# from transformers import AutoModelForCausalLM\n# This is not llama-server. Do not publish a one-liner as a measured stack.\n"
    },
    {
      "id": "sglang",
      "name": "SGLang",
      "short": "Fast serving — AMD documents a Radeon image; AITER off",
      "verdict": "advanced",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/sglang",
      "homepage": "https://docs.sglang.io/",
      "well": "Serving / structured generation when you already live in SGLang.",
      "badly": "First home box; default AITER on Radeon.",
      "when": "After llama.cpp, if you need SGLang’s serving features and will disable AITER.",
      "run": "# Launch command for our Qwen GGUF is not documented on that page.\n# Use their HF/serving examples inside the container. Not measured here.\n"
    },
    {
      "id": "strata",
      "name": "Strata",
      "short": "Fast-agent try on our rig — Flash Q6",
      "verdict": "try",
      "default": false,
      "used_on_our_rig": true,
      "href": "https://amdai.net/backends/strata",
      "homepage": "https://www.stratat.com/",
      "well": "Faster Flash Q6 agent loop than llama.cpp on the same rig, once the daily driver is already proven.",
      "badly": "First install; undocumented flags; being your only engine.",
      "when": "Daily driver already works and you want the fast agent.",
      "run": "# This site’s convention: listen on 127.0.0.1:8083 so llama-server can keep :8080.\n# not recorded: exact strata command line / measured VRAM+RAM from the rig owner.\ncurl -s http://127.0.0.1:8083/v1/models\n"
    },
    {
      "id": "vllm",
      "name": "vLLM",
      "short": "Advanced / experimental for consumers",
      "verdict": "advanced",
      "default": false,
      "used_on_our_rig": false,
      "href": "https://amdai.net/backends/vllm",
      "homepage": "https://docs.vllm.ai/",
      "well": "Several clients, continuous batching, a harness that wants a real server.",
      "badly": "A first home box; mixing AMD’s 0.27 pin with community 0.31 in one venv.",
      "when": "You already have llama.cpp working and you need batching.",
      "run": "# not recorded: exact image tag / measured VRAM if 0.31 is pinned on this rig.\n# Shape only. v0.32.0+ on RDNA: the fast backend is opt-in, not the default.\n# vllm serve Qwen/Qwen3.6-35B-A3B --host 127.0.0.1 --port 8000 --attention-backend ROCM_SEGMENTED_ATTN\ncurl -s http://127.0.0.1:8000/v1/models\n"
    }
  ]
}