Run a local coding assistant on a Radeon GPU

llama.cpp first

Zero to first edit

1. 1 · Cited attach; llama.cpp measured on the local desktop

   Start llama-server — Vulkan first

   Build llama.cpp Vulkan on Ubuntu 24.04+ from /setup, then serve the house file: ./build-vk/bin/llama-server -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 127.0.0.1 --port 8080. Those are the same flags as the 35B recipe / tuning guide. HIP swap is the ROCm compile on Vulkan vs HIP — not the first token. llama-server README: OpenAI-shaped /v1, --host default 127.0.0.1, --alias for the API id. We pin --port 8080 (upstream README default is 9931). Confirm curl -s http://127.0.0.1:8080/v1/models before you open Aider.

   • Ubuntu setup (measured)
   • llama.cpp
   • Vulkan vs HIP
   • 35B recipe
   • Tuning flags

2. 2 · Cited attach; llama.cpp measured on the local desktop

   Pick Q6, Q5, or Q4 by VRAM — Q3 is a labeled estimate

   32 GB R9700: house Qwen3.6-35B-A3B UD-Q5_K_XL. 24 GB: size that same 35B on the calculator. 16 GB and below: use a /models file the calculator says fits (Qwen3.6-27B, Qwen3.8-27B, Gemma 3 27B, gpt-oss-20B) or offload 35B with both RAM lines labeled estimate. Every tier shows Q6 / Q5 / Q4 plus a cited Q3. Q1/Q2 stay off. OpenHands docs name Qwen3.6-35B-A3B as a local agent starting point — that is their cite, not a speed we invent. Flash on Strata is the fast-agent path, not this first /v1.

   • Models
   • Qwen3.6-35B
   • Qwen3.6-27B
   • gpt-oss-20B
   • Calculator
   • Can my Radeon run it?

3. 3 · Cited attach; llama.cpp measured on the local desktop

   Size context for agentic coding — 32K to 128K+

   Agents send files plus the chat. 32K is a short repo slice; 128K+ is the window that keeps a real edit loop. KV cache grows with -c, so decode slows and RAM rises. House daily driver is -c 262144. Hermes docs require at least 64K or they reject the model. Tables on this site publish 4K / 32K / 128K / 256K because speed moves: official UD-Q5 1-GPU llama.cpp is 102.39 / 90.52 / 66.43 / 44.4 tok/s at those lengths. The calculator prints Minimum system RAM needed (the load at the context you picked) and RAM needed at full context (262144 for the house 35B). Overnight UD-Q5 peaks are 37.32 GB at 4K and 42.9 GB at 256K — measured on llama.cpp, not inside Aider. Aider’s max_input_tokens cannot exceed server -c.

   • Long context
   • Calculator
   • Flash RAM offload
   • Glossary

4. 4 · Cited attach; llama.cpp measured on the local desktop

   Point Aider, Cline, and Continue at :8080/v1

   Copy the blocks below. Base URL is http://127.0.0.1:8080/v1. Model id must match GET /v1/models (house placeholder qwen; llama-server --alias sets it). API key is the non-empty placeholder local-not-a-secret — Aider requires it even locally. LM Studio is :1234 if you already followed /setup/lm-studio. House Hermes on the local desktop uses this same :8080. Attach is cited, not measured on the local desktop. For reliable tool-arg JSON, cite llama.cpp --json-schema / GBNF on /constrained-decoding — grammar decode is not a harness tok/s.

   • Harnesses
   • Aider
   • Cline
   • Continue
   • Hermes
   • Connect a harness
   • Constrained decoding
   • Tool calling

5. 5 · Cited attach; llama.cpp measured on the local desktop

   LM Studio and Ollama stay secondary

   If the desktop app is already serving /v1, Aider/Cline/Continue can use that base URL instead. LM Studio: http://127.0.0.1:1234/v1 — /setup/lm-studio. Lemonade: http://127.0.0.1:13305/api/v1 — /setup/lemonade (note /api/v1). Ollama is a wrapper, not the measured path; prefer llama-server when you want this site’s flags and cards. Do not present llama.cpp tok/s as an Ollama, LM Studio, or Lemonade result.

   • LM Studio on Radeon
   • Lemonade on Radeon
   • Backends
   • llama.cpp
   • Docker / Podman

6. 6 · Cited attach; llama.cpp measured on the local desktop

   When the assistant sees no model

   curl /v1/models first. Empty list: server not up, wrong port, or --host not 127.0.0.1. Aider “unknown model”: add openai/ prefix and model settings. Cline connection error: do not append /chat/completions to the base URL. Continue leftover context: server --context-shift / reset, not a Continue bug we invent. GPU idle: you are on CPU — check Vulkan llama-server, not the IDE. Start from /troubleshoot. Cited ROCm rows live on /rocm-problems.

   • Troubleshoot
   • Error paste
   • ROCm problems
   • FAQ
   • Glossary

Cited llama-server line

House daily-driver flags. Vulkan binary. Loopback /v1. Measured on the local desktop as llama.cpp, not as a harness.

./build-vk/bin/llama-server -m ~/models/Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf -ngl 99 -c 262144 --cache-type-k f16 --cache-type-v f16 --host 127.0.0.1 --port 8080

Copy-paste clients

Each block is cited from that tool’s OpenAI-compatible docs. Replace qwen with the id from GET /v1/models.

# Cited from Aider OpenAI-compatible docs. Key must be non-empty even locally.
export OPENAI_API_BASE=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=local-not-a-secret
aider --model openai/qwen

# Same flags on the CLI:
# aider --openai-api-base http://127.0.0.1:8080/v1 \
#   --openai-api-key local-not-a-secret \
#   --model openai/qwen

Provider: OpenAI Compatible
Base URL: http://127.0.0.1:8080/v1
API Key: local-not-a-secret
Model ID: qwen
# Do not append /chat/completions to the base URL.

name: Local llama-server
version: 0.0.1
schema: v1
models:
  - name: Local Qwen
    provider: openai
    model: qwen
    apiBase: http://127.0.0.1:8080/v1
    apiKey: local-not-a-secret
# capabilities: [tool_use]   # only if the server actually implements tools

# Native llama.cpp provider (docs: apiBase *without* /v1):
# provider: llama.cpp
# apiBase: http://127.0.0.1:8080

Models by VRAM

Only files already on /models. Q6 / Q5 / Q4 plus a cited Q3 estimate. Nothing below Q3.

35B quant picker

Speeds are measured on llama.cpp directly, not inside Aider, Cline, or Continue.

RAM lines (llama.cpp-measured)

Wherever a file spills off the GPU, the two labels stay Minimum system RAM needed and RAM needed at full context. House 35B peaks below are Ubuntu llama.cpp, not Aider.

Open in Stack Lab

Stack Lab remains Ubuntu llama.cpp-first. There is no Aider package and no public download.

Sources

• llama.cpp — llama-server README (OpenAI-shaped /v1)
• Aider — OpenAI compatible APIs
• Aider — advanced model settings
• Cline — OpenAI Compatible
• Continue — OpenAI / compatible apiBase
• Continue — llama.cpp provider
• Qwen — Qwen3.6-35B-A3B model card
• Unsloth — Qwen3.6-35B-A3B GGUF