What it is
TabbyAPI is a FastAPI OpenAI-compatible server in front of ExLlama (official API for ExLlamaV3). ExLlama is a CUDA-first inference kernel stack for EXL2/EXL3 quants.
Architecture in plain English
Python API + compiled ExLlama kernels. Continuous batching / paged attention is documented for NVIDIA Ampere+. ROCm is an install extra, not the center of gravity.
What it does well
EXL3/EXL2 on NVIDIA. Fast local OpenAI API if the kernels exist for your GPU.
What it does badly
A Radeon-first daily driver. Consumer RDNA FlashAttention/paged-attention is not first-class in their own issues.
ROCm / Radeon
Getting-Started documents pip install -U '.[amd]' against ROCm 6.1 (older text says 6.0 / 5.6). That is not ROCm 10.1 / gfx1201. ExLlamaV2 has a community PR to fix gfx1201 compile — do not treat it as merged or as a house path. FA2 on ROCm has been discussed as CDNA-oriented.
Vulkan. No.
Supported quants
EXL3 (recommended on current TabbyAPI), FP16/BF16; EXL2 on the checkpoint branch. Still never below Q4-class.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# Documented extra — ROCm 6.1, not a 10.1 pin:
# pip install -U '.[amd]'
# Confirm torch.version.hip and gfx1201 before you expect kernels.
# We have no working recipe on the R9700. Treat as TODO / not documented.
Run
# TabbyAPI start scripts are in their repo. No R9700 command line here.
Tuning flags
Do not use HSA_OVERRIDE to “make EXL2 see gfx1201.” If it does not compile, use llama.cpp.
When to choose it
Almost never, on this site.
You know you need ExLlama and will build against a ROCm PyTorch that lists gfx1201 — then tell the forum what happened.