What it is
Two related “squeeze a giant MoE onto a workstation” projects. KTransformers mixes CPU inference with GPU kernels. ik_llama.cpp is a llama.cpp fork focused on CPU and CUDA IQK kernels.
Architecture in plain English
KT: Python local_chat + kt-kernel, GGUF on disk, CPU infer for experts. ik_llama.cpp: ggml fork. Their maintainers tell AMD users to use upstream llama.cpp.
What it does well
Huge MoE on a lot of RAM + NVIDIA (ik) or a 7900 XTX beta (KT).
What it does badly
gfx1201 daily driver; expecting their IQ1–IQ3 tricks (banned on this site).
ROCm / Radeon
KT ROCm.md is beta, developed on EPYC + Radeon 7900 XTX (gfx1100), PyTorch ROCm 6.2.4 in the doc. Marlin is unsupported. Multi-GPU HIP graph capture has reported crashes. gfx1201 is not a listed test row. ik_llama.cpp: ROCm/Vulkan are explicitly unmaintained; issues will not be resolved unless you do the work.
Vulkan. ik_llama.cpp can compile Vulkan; maintainers say it is not the focus and to use upstream llama.cpp on AMD.
Supported quants
GGUF. Their docs mention IQ2/IQ3-class kernels — this site still refuses anything below Q4, including draft/MTP.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# KTransformers ROCm.md (7900 XTX / ROCm 6.2.4 — not a 10.1 gfx1201 pin):
# export CPUINFER_USE_ROCM=1
# pip3 install torch --index-url https://download.pytorch.org/whl/rocm6.2.4
# ik_llama.cpp: build CPU or CUDA. For AMD, use upstream llama.cpp instead.
Run
# python ktransformers/local_chat.py ... (their DeepSeek example)
# We have no R9700 command line. Not documented for gfx1201.
Tuning flags
Do not import their below-Q4 quants. If HIP graphs crash on two cards, that is a known KT limitation, not a reason to HSA_OVERRIDE.
When to choose it
Not the house path.
Almost never. Use llama.cpp (or Strata for Flash).