What it is
Raw torch + Hugging Face Transformers. generate() in a notebook, a training loop, or a custom pipeline. This is the layer vLLM/SGLang sit on — not a replacement for llama-server.
Architecture in plain English
Eager or compiled PyTorch on HIP. Weights are usually HF safetensors, not GGUF. You own the loop, the tokenizer, and the OOM.
What it does well
Fine-tune, custom heads, debugging a ROCm wheel.
What it does badly
Daily chat; long 262K context as a first script.
ROCm / Radeon
Install from AMD’s PyTorch-for-ROCm page with a gfx1201 selector. Confirm torch.version.hip before from_pretrained. Fine-tune on Radeon is possible and still a thinner cookbook than CUDA. ROCm 10.0 KI — training GPU resets; try TORCH_BLAS_PREFER_HIPBLASLT=0.
Vulkan. No (this is HIP). Use llama.cpp Vulkan for chat.
Supported quants
bitsandbytes / AWQ / GPTQ / native dtypes — still never below Q4-class for anything we would serve.
House floor is still Q4. Prefer Q5. See quants and thecalculator.
Install on an R9700
# Follow the live AMD PyTorch-for-ROCm page — pick the gfx1201 / gfx12 index.
# python -c "import torch; print(torch.version.hip); print(torch.cuda.get_device_name(0))"
Run
# from transformers import AutoModelForCausalLM
# This is not llama-server. Do not publish a one-liner as a measured stack.
Tuning flags
TORCH_BLAS_PREFER_HIPBLASLT=0 if fine-tune resets (ROCm 10.0 KI). One GPU first. Confirm the device is the R9700, not the iGPU.
When to choose it
Framework work. Chat stays on llama.cpp.
You need a training loop or a wheel check, not a harness.