Glossary · AMDai

A

Abliterated

Also: abliteration · refusal edit

A weight edit that weakens a model’s refusal behavior. The file still looks like a GGUF; the change is inside the tensors, not a different architecture.

On a home Radeon box. On a home Radeon box it is educational only — never the default first file. Official instruct comes first; you are responsible for what you run. Quality can drop, and AMDai has no tested-on-our-rig abliterated pick.

Go deeper

Abliteration is a tensor edit, not a new architecture. Vet the uploader and the file hash. Calculator share links exist only for the official control. Q3 is estimate/community unless measured; Q1/Q2 stay off.

See also: Official instruct · GGUF · House rules

Adapter

Also: LoRA adapter · PEFT adapter · merge_and_unload

The small trained matrices from LoRA or QLoRA. The base model stays frozen. After training you merge the adapter into the base (PEFT merge_and_unload) before converting to GGUF for llama.cpp.

On a home Radeon box. On AMDai, training is a cited side path — inference stays llama.cpp first. Speeds are not yet measured on our R9700s. Q3 stays a labeled estimate after you quantize the merged file.

Go deeper

AMD’s Radeon playbook merges with merge_and_unload then save_pretrained. llama.cpp convert_hf_to_gguf.py turns that folder into a GGUF. Quantize Q6_K / Q5_K_M / Q4_K_M; Q3_K_M is a labeled estimate.

See also: LoRA · QLoRA · GGUF · imatrix

AMDai

Also: amdai.net · R9700 Lab Notes · og:site_name

The display wordmark for this notebook (AMDai · R9700 Lab Notes). Domain, URLs, package names, and amdai-bench stay lowercase. Open Graph site_name is AMDai.

On a home Radeon box. The header says AMDai; the address bar says amdai.net. Same project — mixed-case is display-only so it is not confused with AMD’s logo or an official AMD publication.

APU

Also: Strix Halo · Ryzen AI Max · Ryzen AI Max+ 395 · Radeon 8060S · gfx1151 · 780M · 890M · Ryzen iGPU

A chip that combines CPU and GPU and shares one memory pool. There is no discrete VRAM sticker. The shopper box on AMDai is Ryzen AI Max+ 395 with Radeon 8060S (gfx1151) — Strix Halo in mini-PCs and laptops.

On a home Radeon box. A 64 GB or 128 GB kit is both system RAM and GPU memory. Size the file on the calculator before you assume a 35B Q5 “fits like an R9700.” Official tok/s on this site are still the discrete R9700 cards.

Go deeper

AMD’s ROCm 10.1.0 GPU table prints VRAM for this SKU as Dynamic + carveout. BIOS UMA reserves a slice; GTT / dynamic allocation grows into the rest of the same kit. llama.cpp Vulkan first. Not measured on the local desktop.

See also: UMA · GTT · VRAM vs system RAM · System RAM

amdai-bench

Also: amdai-bench/run@1 · run@1 · bench schema

The first-party measurement schema this site uses. A validating card is tagged amdai-bench/run@1 and records hardware, software pins, the launch command, and tok/s. The product name stays lowercase.

On a home Radeon box. Home numbers on AMDai come from these cards, not from a scraped leaderboard. If a page cites 102.39 tok/s or 42.9 GB RAM, a run card is the source.

Go deeper

Schema id is amdai-bench/run@1. Cards record GPU layout (single / split / dual-instance), the launch command, prefill and decode tok/s, VRAM per card, and peak_system_ram_gb when sampled. The 2026-10-12 overnight drop is 33 cards, pending re-verify after the DDR5 oops.

See also: Run card · Prefill vs decode · local desktop

B

Backend

Also: inference engine · engine

The program that loads weights and emits tokens — llama.cpp, Strata, or vLLM on this site. A harness (Hermes, a chat UI) sits above a backend; it is not a substitute for one.

On a home Radeon box. Pick the engine before you debug the card. AMDai’s consumer default is llama.cpp (Vulkan first). Strata is the Flash try; vLLM is a multi-client serve try.

Go deeper

Chooser order: llama.cpp (Vulkan if HIP CMake sulks), then Strata for Flash, then vLLM only for several clients. Ollama and desktop GUIs are try-if-the-meter-moves. Compare them on /backends — no invented tok/s.

Batch (-b)

Also: -b · --batch-size · logical batch · LLAMA_ARG_BATCH

Official llama.cpp logical maximum batch size. llama-server README: -b / --batch-size N (default 2048; env LLAMA_ARG_BATCH). How much prompt the engine plans for. The GPU still computes in -ub-sized chunks.

On a home Radeon box. Long Hermes, coding-tool, and RAG prompts are prefill-bound. House llama-server does not record a -b pin. Overnight llama-bench prefill cells are not a llama-server batch A/B. Measure Stack Lab prefill-ab/1.

Go deeper

The CLI README documents the same -b / --batch-size flag. A larger -b with a smaller -ub still processes the prompt in physical micro-batches. llama.cpp #14836 failed compute pp buffers at -b 4096 -ub 4096. Do not invent a house pin.

C

Context length

Also: context · 4K · 32K · 128K · 256K · 262K · -c

How many tokens of prompt plus conversation the model can keep in the KV cache. House daily driver on one R9700 is 262144 (262K). Tables on this site also show 4K, 32K, 128K, and 256K because speed and RAM change with length.

On a home Radeon box. Longer context grows KV and usually slows decode. Official UD-Q5_K_XL on the local desktop: 102.39 tok/s at 4K and 44.4 tok/s at 256K — same file, one GPU, f16 KV. Size the window on the calculator before you set -c.

Go deeper

House -c 262144 is the 262K daily-driver window. Tables also publish 4K, 32K, 128K, and 256K because KV and decode move. Official UD-Q5_K_XL 1-GPU: 102.39 / 90.52 / 66.43 / 44.4 tok/s at those four lengths.

See also: Context ceiling · KV cache · Prefill vs decode · Quant

Context ceiling

Also: max context · ctx-ceiling · biggest context · ab-ctx-ceiling

The largest llama.cpp -c that still loads and serves on a given box. Native max is the model-card window. The ceiling is what VRAM and RAM actually hold.

On a home Radeon box. Shoppers search max context Radeon and 262K VRAM. Stack Lab ctx-ceiling/1 (amdai-bench ab-ctx-ceiling) climbs a lean ladder on one GPU and on all GPUs. AMDai has not ingested a winner on the local desktop — not yet measured.

Go deeper

House overnight llama-bench ran 262144 on UD-Q5, one R9700, f16 KV. That is not a ctx-ceiling winner. Minimum system RAM needed is the lean step rounded to a kit. RAM needed at full context is the winner peak first. The shopper page is /context-ceiling.

See also: Context length · KV cache · VRAM vs system RAM

Constrained decoding

Also: grammar sampling · JSON schema decode · structured JSON

llama.cpp grammar sampling so every next token still matches a GBNF grammar or a JSON Schema converted to GBNF. Used for tool args, calculator output, and recipe export.

On a home Radeon box. Hermes and coding tools need JSON that parses. Official flags are --grammar / --json-schema and response_format. Grammar decode versus free-form is not measured on the local desktop — run Stack Lab grammar-ab/1. The schema is not injected into the prompt except during tool calling.

Go deeper

llama.cpp server README (checked 2026-10-14): --grammar, --grammar-file, -j / --json-schema, -jf / --json-schema-file, plus grammar / json_schema body fields. grammars/README.md: JSON Schemas → GBNF; performance gotchas on issue #4218. Shopper how-to: /constrained-decoding.

See also: GBNF · llama.cpp · Official instruct

Container (Docker / Podman)

Also: Docker · Podman · container image

An isolated process that runs an official llama.cpp image instead of a host build. On AMDai that is ggml-org’s GHCR server-vulkan first, server-rocm second. The host still owns /dev/dri and, for HIP, /dev/kfd.

On a home Radeon box. People search “rocm docker llama.cpp.” The cited path is /setup/docker. It is not a measured local desktop card. We do not invent a container tok/s or a speed delta versus bare metal. Bind the published port to 127.0.0.1.

Go deeper

docker.md names ghcr.io/ggml-org/llama.cpp:server-vulkan and :server-rocm and says GPU images are not currently tested by CI beyond being built. AMD’s ROCm 10.1 container page documents --device /dev/kfd --device /dev/dri and optional seccomp=unconfined. Rootless Podman needs --group-add keep-groups; Fedora SELinux needs container_use_devices plus :z / :Z on the model volume.

See also: llama.cpp · Device passthrough · NVMe staging

ComfyUI

Also: Comfy · image gen UI

A node-graph UI for local diffusion checkpoints on HIP/ROCm PyTorch. On AMDai it is the first image-gen path. It does not speak llama-server /v1.

On a home Radeon box. People search “ComfyUI ROCm Radeon.” First-run install is /setup/comfyui. Fit tables stay on /image-gen. Speeds are not yet measured by AMDai. llama.cpp stays first for LLMs. Q3 stays a labeled estimate on any LLM that shares the box.

Go deeper

AMD documents git clone, pip install -r ComfyUI/requirements.txt, then python ComfyUI/main.py → http://127.0.0.1:8188. Checkpoints go in models/checkpoints. The official Windows AMD portable is ComfyUI_windows_portable_amd.7z; the Nvidia zip is not a Radeon path. Instinct Docker tags are not R9700 pins.

Chat template

Also: --jinja · --chat-template · --chat-template-file · chat_template_tool_use

The Jinja string llama-server uses to wrap messages, tools, and reasoning. It usually ships in the GGUF metadata. --jinja turns the engine on; --chat-template and --chat-template-file override the file.

On a home Radeon box. OpenAI-style tool calls need a tools-aware template. Inspect GET /props for chat_template and chat_template_tool_use. A mismatch is the usual “tools ignored” or “markup in content” cause. House pin still passes --jinja.

Go deeper

llama.cpp function-calling.md: native handlers exist for several families (Llama 3.1 / 3.3, GPT-OSS Harmony, Hermes 2/3, …). Unrecognized templates log Chat format: Generic. --chat-template chatml is the documented YMMV fallback. --skip-chat-parsing dumps markup into content.

See also: Jinja · Special tokens · Tool calling · Official instruct · llama.cpp · GBNF

Cache-reuse (--cache-reuse)

Also: --cache-reuse · n_cache_reuse · KV-shift reuse · LLAMA_ARG_CACHE_REUSE

Official llama-server min chunk size for reuse via KV shifting. --cache-reuse N (default 0 = disabled; env LLAMA_ARG_CACHE_REUSE). Requires prompt caching to be on. n_cache_reuse on the API matches that flag.

On a home Radeon box. Default 0 means KV-shift reuse is off even when --cache-prompt is on. House recipes do not record a --cache-reuse pin. Do not invent 256. Measure prompt-cache-ab/1 before you keep a new value.

Go deeper

llama.cpp server README is the cite. --cache-ram (default 8192 MiB; PR #16391) is the system-RAM ceiling for cached prefixes, not VRAM. Extra cache RAM on top of the measured 37.32 / 42.9 GB peaks is a labeled estimate.

See also: Prompt cache · Prefix cache · KV cache · TTFT

Coding assistant

Also: Aider · Cline · Continue · local coding agent

An editor or CLI agent that talks to a local OpenAI-compatible /v1. On AMDai that server is llama.cpp llama-server on 127.0.0.1:8080. Aider, Cline, and Continue are the cited first three.

On a home Radeon box. People search “aider llama.cpp radeon” and “cline local model.” The cited walk is /setup/coding-assistant. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not Aider tok/s. Vulkan llama-server first; LM Studio :1234 is secondary.

Go deeper

Aider prefixes the model with openai/ and requires a non-empty API key. Cline uses the OpenAI Compatible provider — do not append /chat/completions. Continue openai provider wants apiBase including /v1. House file is still Qwen3.6-35B-A3B UD-Q5_K_XL. Hermes is the house agent on the local desktop against the same :8080.

See also: llama.cpp · LM Studio · Context length

D

Daily driver

Also: qwen-reliable · house default

The stack AMDai tells a newcomer to run first: Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL on one R9700, llama.cpp Vulkan, 262K context, f16 KV, no RAM offload.

On a home Radeon box. It is the measured in-VRAM path on a 32 GB card. Official 1-GPU cards: 102.39 tok/s at 4K, 90.52 at 32K, 66.43 at 128K, 44.4 at 256K. Do not start with Flash, vLLM, or an abliterated file.

Go deeper

Recipe id qwen36-35b.llamacpp-vulkan.262k — one R9700, Vulkan, 262K, UD-Q5_K_XL. Stack Lab profile qwen-reliable is the same pick. Q3 35B listings stay estimates unless a card says measured.

See also: Qwen3.6-35B-A3B · llama.cpp · Quant

Device passthrough

Also: --device · /dev/kfd · /dev/dri · keep-groups

Giving a container the host GPU nodes it needs. Vulkan llama.cpp uses /dev/dri. ROCm / HIP also needs /dev/kfd. The login (or --group-add) must match render and video.

On a home Radeon box. Permission denied on /dev/kfd is the usual container miss. AMD’s ROCm 10.1 container page documents both nodes and optional seccomp=unconfined. Rootless Podman bind-mounts devices and needs --group-add keep-groups; Fedora SELinux may need container_use_devices.

Go deeper

Passing all of /dev/dri exposes every GPU. AMD shows renderD128 / renderD129 to restrict. Volume labels :z / :Z are for the GGUF mount, not for /dev. Full walk: /setup/docker.

See also: Container (Docker / Podman) · ROCm · HIP

Dual-GPU

Also: two cards · two copies · split · tensor parallel

Two R9700s can run two full copies of a model (two clients) or one split / tensor-parallel copy (one larger or shared request). They are different layouts; AMDai does not treat “two GPUs” as free 2× speed.

On a home Radeon box. On local desktop, two copies of UD-Q5_K_XL sit near 100 tok/s per request. A split 4K pass is 82.78 tok/s total versus 102.39 on one card. Prefer two copies unless you deliberately need one shared context. House rule: one LLM per GPU unless you combine on purpose.

Go deeper

Layouts on amdai-bench cards: single, split, dual-instance. Two copies: about 100 tok/s per request, 172 at two concurrent, 177 at 4–8, 423 soak requests, 0 errors. Split 4K max total 82.78 versus 102.39 on one card. Consumer why: /multi-gpu. Hands-on llama.cpp flags: /setup/multi-gpu.

See also: Radeon AI PRO R9700 · Offload · Run card

E

Expert offload

Also: -ot · --n-cpu-moe · CPU MoE · tensor override

A llama.cpp MoE trick: keep attention on the GPU and park expert tensors in system RAM. The flags are -ot (tensor override regex) and --n-cpu-moe (how many experts stay on CPU).

On a home Radeon box. Flash-class GGUFs do not fit 32 GB VRAM. The launch shape lives on Flash RAM offload — exact -ot regex and --n-cpu-moe N from the rig owner are not recorded. Do not invent flag values; copy the shape and measure.

Go deeper

Shape only: llama-server -ngl 99 -c 262144 --n-cpu-moe not recorded -ot not recorded --cache-type-k/v f16. Copy that skeleton from Flash RAM offload; do not invent the regex or N.

See also: Offload · MoE vs dense · Qwen 3.8 Flash

F

Fast agent

Also: qwen-fast-agent · agent stack

The house speed path after the daily driver works: Qwen 3.8 Flash Unsloth UD-Q6_K_XL on Strata v0.1.42, usually with RAM offload, pointed at a harness.

On a home Radeon box. First-party Flash on our rig is about 32.6 tok/s on Strata versus about 16 on llama.cpp (2026-10). It is not the first box you should build; get official instruct generating first.

Go deeper

Stack Lab profile qwen-fast-agent is this path. Get official instruct generating on llama.cpp first; then point a harness at Strata :8083. Q3 Flash rows stay estimates unless measured.

See also: Qwen 3.8 Flash · Strata · Daily driver

Qwen 3.8 Flash

Also: Flash · Flash-Next · UD-Q6_K_XL

The large fast-agent GGUF on this site (Unsloth UD-Q6_K_XL listing ~169 GB). It is a different model from the 35B daily driver and usually needs DDR5 expert offload on a pair of 32 GB cards.

On a home Radeon box. Do not download Flash to “make the 35B faster.” Size it on Flash daily / RAM offload; Q3 Flash rows are estimates unless measured. Q1/Q2 stay off, including draft heads.

Go deeper

Publisher UD-Q6_K_XL folder ~169 GB. Two-card Strata at 262K: 32.8 tok/s at 8K/32K, 31.8 at 100K on the Flash daily page. Recipe: qwen38-flash.strata.262k. Q3 Flash rows stay estimates unless measured.

See also: Fast agent · Expert offload · Quant

Flash Attention (-fa)

Also: -fa · --flash-attn · flash attn · flash-attn

A fused attention kernel in llama.cpp. Official flag is -fa / --flash-attn on|off|auto (default auto). Overnight house 35B Vulkan llama-bench recorded -fa 1.

On a home Radeon box. Usually cuts attention VRAM and can help long context. llama.cpp multi-gpu.md requires it for tensor split and/or a quantized V cache. llama-server FA on versus off is not measured on the local desktop — run Stack Lab Flash Attention A/B.

Go deeper

House default KV is f16. Turn -fa on before -ctv q8_0 or -ctv q4_0. HIP / ROCm documents the same flag; HIP FA deltas stay not measured. That A/B is not an allowlisted Open-in-Stack-Lab deep link. House recipes stay Q4 and up.

See also: KV cache · llama.cpp · Prefill vs decode · Vulkan · Prompt cache

G

gfx1201

Also: LLVM target · GFX1201 · RDNA4 target

The LLVM / ISA target for RDNA 4 Radeon cards — Radeon AI PRO R9700 and the RX 9070 series on current ROCm tables. rocminfo and llama.cpp --list-devices should show it before you download 20 GB.

On a home Radeon box. Wrong gfx means HIP kernels miss and you silently land on CPU. Confirm gfx1201 after install; do not use unofficial HSA_OVERRIDE on this SKU.

Go deeper

HSA_OVERRIDE_GFX_VERSION is unofficial and not needed on official gfx1201 + current ROCm. Confirm with rocminfo and llama.cpp --list-devices before you download weights.

See also: RDNA 4 · Radeon AI PRO R9700 · ROCm

GBNF

Also: GGML BNF · .gbnf · grammar file

GGML Backus–Naur Form — the grammar language llama.cpp uses to constrain decode. A JSON Schema is converted to GBNF. Samples live in llama.cpp grammars/.

On a home Radeon box. Use GBNF when the output is not JSON, or the official schema subset cannot express the shape. Use a JSON Schema when the output is a JSON object. Shopper how-to: /constrained-decoding.

Go deeper

Official grammars/README.md: production rules, a root rule, and --grammar / --grammar-file. Server README also accepts -j / --json-schema. Do not invent a tok/s cost; measure Stack Lab grammar-ab/1.

GGUF

Also: .gguf · ggml file

The llama.cpp weight-file format: one (or a few) files that already include a quant, tokenizer, and metadata. It is not a PyTorch checkpoint and not what vLLM wants by default.

On a home Radeon box. Every house recipe on a Radeon box starts from a GGUF on NVMe. Pin a revision; prefer the Unsloth UD names on /stacks. Stage the file before you tune flags.

Go deeper

Daily-driver file: unsloth/Qwen3.6-35B-A3B-GGUF · Qwen3.6-35B-A3B-UD-Q5_K_XL.gguf (publisher 26.6 GB). Pin a revision. vLLM does not want GGUF as its default path.

See also: Quant · Unsloth UD · NVMe staging · imatrix

GTT

Also: Graphics Translation Table · dynamic allocation · dynamic VRAM

The path where the driver maps more of the same system-memory kit to the iGPU after boot, instead of living only inside a fixed BIOS carve-out.

On a home Radeon box. On Strix Halo, a tiny UMA reserve plus GTT can still load a large GGUF — and it can also starve the OS if the model plus KV eat the kit. The calculator’s 70% slice is an estimate of that split, not a firmware reading.

Go deeper

AMD’s ROCm 10.1.0 APU table calls this Dynamic + carveout. Change the calculator usable percent if your BIOS printout disagrees. We do not invent a megabyte GTT cap.

See also: UMA · APU · VRAM vs system RAM

H

HIP

Also: AMD HIP · GGML_HIP · rocm backend

AMD’s C++ GPU compute API. llama.cpp can compile a HIP backend when /opt/rocm is clean. HIP is not CUDA and not a drop-in replacement.

On a home Radeon box. On AMDai, HIP is optional — Vulkan llama.cpp is what we run for the 35B daily driver. Try HIP after rocminfo is boring; do not block first tokens on a HIP CMake fight.

Go deeper

Compile GGML_HIP only after /opt/rocm and rocminfo show gfx1201. Family packages are gfx-specific; a gfx1100 tree is the wrong binary. HIPRTC and rocBLAS live on the ROCm-guide pages, not here.

House rules

Also: consumer house rules · how we pick

How AMDai picks a stack: llama.cpp first; prefer Q5 over Q4; Q3 listed as estimate or community unless measured; Q1/Q2 stay off; GGUFs on NVMe, not a spinning disk; one LLM per GPU unless you deliberately combine.

On a home Radeon box. These rules keep a first Radeon box out of folklore. Q3 rows exist and are labeled estimate unless measured. They are also not a license to invent tok/s.

Go deeper

llama.cpp over vLLM for a first box; Q5 over Q4 when both fit; Q3 listed; Q1/Q2 excluded; NVMe; one LLM per GPU. Budget and calculator follow the same list — Q3 rows appear as labeled estimates.

See also: Quant · llama.cpp · NVMe staging

I

Official instruct

Also: instruct GGUF · chat template · jinja

The publisher’s chat / instruct GGUF — official Qwen weights plus a Jinja template — not an abliterated merge or an unnamed “uncensored” file.

On a home Radeon box. First tokens on a R9700 should come from official instruct. Enable --jinja on current Qwen templates. Abliterated files come after that pipe is boring.

See also: Abliterated · Qwen3.6-35B-A3B · GGUF · Chat template

imatrix

Also: importance matrix · llama-imatrix

An importance matrix llama.cpp builds from a model and a calibration text, then feeds to llama-quantize to reduce quality loss versus a naive quant.

On a home Radeon box. Use it when you convert a new GGUF and want your own calibration text. It does not turn a stock Q5_K_M into an Unsloth UD-Q5_K_XL file. The how-to is /guides/quantize-gguf.

Go deeper

Official llama-imatrix needs -m and -f. Default output is imatrix.gguf. Unsloth Dynamic docs describe a different calibration set and per-layer mix — download UD-*_XL; do not claim stock tools reproduce it.

See also: Quant · Unsloth UD · KL-divergence · GGUF

J

Jinja

Also: --jinja · Jinja2 · LLAMA_ARG_JINJA

The template engine llama-server uses to wrap chat messages. Official --jinja / --no-jinja (default enabled). House recipes still pass --jinja.

On a home Radeon box. Without it, roles, tools, and thinking tags leak as raw text. function-calling.md requires --jinja for OpenAI-style tools. Inspect GET /props.

Go deeper

llama-server README (checked 2026-10-14): --chat-template and --chat-template-file override GGUF tokenizer.chat_template. --chat-template-kwargs passes JSON into the parser. The shopper page is /chat-templates.

See also: Chat template · Special tokens · Tool calling

JSON schema

Also: json_schema · --json-schema · response_format · json_object

A JSON Schema object that llama-server can turn into a GBNF grammar. Send it as response_format or -j / --json-schema when the whole reply must be one JSON object.

On a home Radeon box. Different from a tools array. Tools + --jinja produce tool_calls for Hermes, Cline, and Continue. Schema / GBNF constrain the message body. Official grammars/README.md: the schema is not injected except during tool calling, so describe the fields in the prompt.

Go deeper

Server README: {"type":"json_object"} or a schema-constrained object. Measure free-form versus grammar on Stack Lab grammar-ab/1. Do not copy a free-form 35B tok/s as a grammar figure.

See also: GBNF · Tool calling · Chat template · llama.cpp

K

KV cache

Also: f16 KV · q8_0 KV · cache-type-k · cache-type-v

Key/value attention memory that grows with context length. Separate from the weight file. House types: f16 (2 bytes, daily driver on our rig) and q8_0 (1 byte, smaller KV; quality trade not measured here).

On a home Radeon box. At 262K, KV is a real VRAM line on a 32 GB card. Official 35B cards use f16. Do not confuse a KV type with a weight quant — q8_0 as cache is not “running Q8 weights.”

Go deeper

The calculator’s KV estimate is 2 × kv_layers × kv_heads × head_dim × context × bytes. Hybrid Qwen cards only grow KV on full-attention layers — that count comes from the public model card. Official 35B sweep cards use f16.

KL-divergence

Also: KLD · Kullback–Leibler · kl-divergence-base

A llama.cpp perplexity metric that compares a quantized model’s token distribution to an f16 / bf16 baseline. Zero means the distributions match.

On a home Radeon box. Use it after you make a GGUF to measure accuracy loss. Record baseline logits with --kl-divergence-base, then rerun the quant with --kl-divergence. AMDai has no house KLD score — run the tool.

Go deeper

The llama.cpp perplexity README also reports PPL. The logit file is very large (11 GiB for LLaMA 2, 37 GiB for LLaMA 3 on Wikitext-2). Stage leftover on NVMe. Then record an amdai-bench card for speed.

See also: Quant · imatrix · amdai-bench

L

llama.cpp

Also: llama-server · llama-cli · llama-bench · ggml-org

The consumer default inference engine on AMDai: a C/C++ GGUF runner from ggml-org. You compile a backend, load a file, and optionally serve an OpenAI-shaped /v1 on llama-server (127.0.0.1:8080).

On a home Radeon box. Vulkan first on our rig (commit c060ca974 on the official 35B sweep); HIP is optional. llama.cpp beats vLLM for a single-user Radeon box. Build it before you install a desktop app that might pull a CUDA binary.

Go deeper

Official 35B sweep: llama.cpp c060ca974, Vulkan, f16 KV. llama-server is the same process as llama-cli plus HTTP on 127.0.0.1:8080. Do not let a desktop app download a CUDA llama.cpp on a Radeon.

See also: Vulkan · HIP · GGUF · mmproj · LM Studio · Lemonade Server · Flash Attention (-fa)

LM Studio

Also: lms · lmstudio · :1234

A desktop chat app that wraps llama.cpp. You download a Vulkan or ROCm llama.cpp runtime, load a GGUF, and optionally serve an OpenAI-compatible /v1 on 127.0.0.1:1234.

On a home Radeon box. People search “LM Studio Radeon.” The cited first-run is /setup/lm-studio. It is not a measured local desktop card. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not measured inside LM Studio. Vulkan first; bundled ROCm only if the picker lists this gfx.

Go deeper

AMD’s playbook documents Ctrl+Shift+R runtime picker, GPU Offload max, Flash Attention On, and http://127.0.0.1:1234. Linux is an AppImage (libfuse2); Ubuntu newer than 22 is not well tested per LM Studio. House file is still Qwen3.6-35B-A3B UD-Q5_K_XL at 262K. Leave the app for compile flags and matching cards — /setup and /backends/llamacpp.

See also: llama.cpp · Vulkan · Offload · Coding assistant · Ollama

Lemonade Server

Also: lemonade · :13305 · lemonade-server · /api/v1

AMD’s own local AI server. The primary LLM recipe wraps llama.cpp (Vulkan or ROCm). OpenAI-compatible API on 127.0.0.1:13305/api/v1 — note /api/v1, not /v1.

On a home Radeon box. People search “Lemonade AMD GPU.” The cited first-run is /setup/lemonade. It is not a measured local desktop card. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not measured inside Lemonade. llama.cpp you compile stays the daily driver. Vulkan first; ROCm only on the gfx their docs name.

Go deeper

Ubuntu 24.04+ PPA is ppa:lemonade-team/stable; Windows is the MSI. Latest GitHub tag we checked is v2026.41.1 (2026-10-07) — ROCm backends move to ROCm 10. lemonade backends install llamacpp:vulkan or llamacpp:rocm. Nightly ROCm bins: gfx1150, gfx1151, gfx103X, gfx110X, gfx120X. House family is still Qwen3.6-35B-A3B. AMDai runs Lemonade on the local desktop; there is still no Lemonade tok/s card.

See also: llama.cpp · Ollama · LM Studio · Open WebUI

LoRA

Also: Low-Rank Adaptation · PEFT LoRA

Parameter-efficient fine-tuning: freeze the base weights and train small low-rank matrices. Unsloth’s table treats LoRA as 16-bit. Hugging Face PEFT/TRL and AMD’s Radeon playbook document it.

On a home Radeon box. Cheaper than a full retrain, hungrier than QLoRA. Unsloth’s 27B LoRA minimum is 64 GB — not a 16 GB card. Training speed is not yet measured on our R9700s.

Go deeper

AMD’s playbook (Gemma 3 4B) lists LoRA typical VRAM 24–32 GB and requires at least 16 GB dedicated VRAM plus 32 GB system RAM on Linux. That is their 4B example, not a 35B number we invent.

See also: QLoRA · Adapter · VRAM vs system RAM

--load-mode

Also: -lm · LLAMA_ARG_LOAD_MODE · load mode

Official llama.cpp model-loading switch. Values: auto (default), none, mmap, mlock, mmap+mlock, dio. Env LLAMA_ARG_LOAD_MODE. Replaces --mmap / --no-mmap / --mlock (PR #20834).

On a home Radeon box. Shopper A/B is mmap default vs --no-mmap vs --mlock. Map those onto auto / none / mlock. House recipes do not record the flag. Load time and RSS stay not yet measured.

Go deeper

auto = mmap unless a device cannot. none = no special loading. dio = Direct I/O when available. --lazy-mode requires mmap. Do not invent a house pin.

See also: mmap · --no-mmap · --mlock · llama.cpp

local desktop

Also: house box · house rig · test rig

The house test host: Ryzen 9 9950X, 4×64 GB DDR5-5200, two Radeon AI PRO R9700 32 GB, Ubuntu 24.04.5. Official 35B cards credit AMDai and llama.cpp c060ca974 Vulkan, ROCm 7.2.0, Mesa RADV 25.2.8.

On a home Radeon box. Every “measured on our rig” figure on AMDai is this box. The 2026-10-12 overnight sweep is 33 cards — measured, pending re-verify after a DDR5-5200 oops at 00:14 CT. Do not treat a different kit as the same RAM line.

Go deeper

Software on the official 35B cards: Ubuntu 24.04.5, llama.cpp c060ca974 Vulkan, ROCm 7.2.0, Mesa RADV 25.2.8, f16 KV. Whole sweep labeled measured, pending re-verify after RAM fix. Flash / Strata / Q3 were excluded from that drop.

See also: Radeon AI PRO R9700 · Run card · amdai-bench

M

mmproj

Also: multimodal projector · libmtmd · llama-mtmd-cli

The multimodal projector GGUF that llama.cpp uses to encode an image into embeddings the text model can read. You typically need two files: the text GGUF and a matching mmproj. libmtmd is the current llama.cpp path.

On a home Radeon box. Local vision on AMDai is image in, text out — not ComfyUI. Pairing is architecture-specific. Speeds are not yet measured by AMDai. The house 35B daily driver is text-only; do not invent an mmproj for that file. Q3 stays a labeled estimate.

Go deeper

Cited tools: llama-mtmd-cli -m model.gguf --mmproj mmproj.gguf --image file.jpg, or llama-server --mmproj. --no-mmproj-offload keeps the projector off the GPU. ggml-org publishes ready multimodal GGUFs. Start from that collection, not a retired Qwen family.

MoE vs dense

Also: mixture of experts · dense · A3B · active parameters

A mixture-of-experts (MoE) model activates a subset of experts per token; a dense model runs every layer. Qwen3.6-35B-A3B is MoE (the A3B names the active slice). Qwen 3.6 27B and Qwen 3.8 27B on this site are dense.

On a home Radeon box. MoE is why a 35B-class GGUF fits one 32 GB R9700 at UD-Q5_K_XL (25.97 GB VRAM at 4K on the official card). Dense 27B files are a different fit; do not assume the same VRAM. Expert offload is an MoE-only lever.

Go deeper

Qwen3.6-35B-A3B activates a 3B-class slice; that is why UD-Q5_K_XL fits one 32 GB card (25.97 GB VRAM at 4K). Flash is a different MoE and usually needs DDR5 expert offload. Dense 27B rows on /qwen are a separate fit.

See also: Expert offload · Qwen3.6-35B-A3B · Quant

mmap

Also: memory-map · --mmap · --load-mode mmap · --load-mode auto

Official llama.cpp default model load. --load-mode auto memory-maps the GGUF unless a device cannot (env LLAMA_ARG_LOAD_MODE). The kernel pages the file from NVMe instead of copying the whole weight blob into anonymous RAM.

On a home Radeon box. Shoppers search llama.cpp mlock and why is system RAM full after load. mmap still faults pages into RSS as you run. AMDai has not measured mmap versus --no-mmap RSS on the local desktop — not yet measured. Run Stack Lab Model loading A/B.

Go deeper

PR #20834 folded --mmap / --no-mmap / --mlock into --load-mode. --lazy-mode can leave huge tensors on disk and requires mmap. House llama-server does not record a load-mode pin. The shopper page is /mmap.

See also: --no-mmap · --mlock · --load-mode · System RAM

--mlock

Also: mlock · --load-mode mlock · mmap+mlock · LLAMA_ARG_MLOCK

Deprecated llama.cpp flag that asks the OS to keep model pages in RAM. Official replacements: --load-mode mlock (no mmap + lock, old --no-mmap --mlock) and --load-mode mmap+mlock (map + lock).

On a home Radeon box. Use it when the box is swapping the weight file. It does not free VRAM and does not fix ErrorOutOfDeviceMemory. Locked-resident peak RAM stays not yet measured.

Go deeper

Discussion #27912: mlock does not mmap; mmap+mlock does. House llama-server does not record --mlock. Print both Minimum system RAM needed and RAM needed at full context.

See also: mmap · --no-mmap · --load-mode · System RAM

Min-p (--min-p)

Also: --min-p · min_p · LLAMA_ARG_MIN_P

Drops tokens below a fraction of the most likely token. Official llama.cpp --min-p default 0.05; 0.0 disables (env LLAMA_ARG_MIN_P). The /completion field is min_p.

On a home Radeon box. Official Qwen cards set min_p=0.0 on their recommended rows. That is not the llama.cpp CLI default. House recipes do not record --min-p.

Go deeper

/v1/chat/completions accepts min_p as a llama.cpp extra on the OpenAI-shaped body. Sampler speed stays not measured yet. The shopper page is /sampling.

N

NVMe staging

Also: NVMe · SSD · spinning disk · HDD

Putting the GGUF on a fast NVMe SSD before you run it. A spinning disk (HDD) can feed weights slower than the GPU can prefill, so the card sits idle while the file is paged.

On a home Radeon box. House rule on a home Radeon box: NVMe, not HDD. If prefill looks broken and rocm-smi is idle, check the disk before you change quants. Staging is not the same as RAM offload.

Go deeper

Prefill stalls with idle rocm-smi often mean the GGUF is still on a spinning disk. Staging to NVMe is not RAM offload and not a quant change. House rule: move the file before you tune flags.

See also: GGUF · Prefill vs decode · House rules

--no-mmap

Also: no mmap · --load-mode none · LLAMA_ARG_MMAP

Deprecated llama.cpp flag that copies the GGUF into process RAM instead of mapping it. Official replacement is --load-mode none (discussion #26469).

On a home Radeon box. Can help when mmap progress looks stuck or a platform pages the map out. Extra RSS for the weight copy stays not yet measured. Do not combine --no-mmap with --load-mode — only the last flag wins.

Go deeper

Lemonade renamed a deprecated --no-mmap option to --load-mode none for llama.cpp. House recipes do not record either flag. Measure Model loading A/B from the planner.

See also: mmap · --mlock · --load-mode · System RAM

O

Ollama

Also: :11434 · ollama serve · ollama ps

A pull-and-run daemon that wraps a packaged llama.cpp / ROCm or Vulkan runtime. Local API on 127.0.0.1:11434, including an OpenAI-shaped /v1.

On a home Radeon box. People search “Ollama AMD GPU.” The cited first-run is /setup/ollama. It is not a measured local desktop card. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not measured inside Ollama. llama.cpp you compile stays the daily driver.

Go deeper

Official Linux install is their script plus an AMD ROCm tarball. gpu.mdx lists R9700 / 9070 XT on Linux ROCm v7; the Windows ROCm table does not. ollama ps is how you prove GPU vs CPU. Default context is 4096; KV grows with OLLAMA_NUM_PARALLEL. House family is still Qwen3.6-35B-A3B. gfx1201 tensiles and Windows R9700 CPU fallback live on /rocm-problems.

See also: llama.cpp · LM Studio · Lemonade Server · Open WebUI · Context length

Open WebUI

Also: openwebui · :3000 · self-hosted chat UI

A self-hosted browser chat UI. It talks to an OpenAI-compatible /v1. On AMDai that server is llama.cpp llama-server on 127.0.0.1:8080. The UI itself does not run the GGUF.

On a home Radeon box. People search “Open WebUI AMD GPU.” The cited first-run is /setup/open-webui. It is not a measured local desktop card. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not measured inside Open WebUI. llama.cpp you compile stays the daily driver.

Go deeper

Official Docker maps host 3000 to container 8080. pip defaults to :8080 — use --port 3000 so it does not steal house llama-server. Admin → Connections → OpenAI, Provider llama.cpp. Do not pull the :cuda image on a Radeon. Ollama :11434 is a secondary path. House file is still Qwen3.6-35B-A3B UD-Q5_K_XL.

See also: llama.cpp · Ollama · LM Studio

Offload

Also: RAM offload · layer offload · -ngl · --fit-target

Some weights or KV live in system RAM instead of VRAM. llama.cpp -ngl 99 means “offload what fits”; leftover layers spill to DDR5. Expert offload is the MoE-specific form (-ot, --n-cpu-moe).

On a home Radeon box. The 35B daily driver on our rig uses 0% RAM offload. Q6_K 1-GPU 4K on the sweep is partly on DDR5 (--fit-target 1024) — decode there is not an in-VRAM figure. Flash usually needs offload; measure, do not guess tok/s.

Go deeper

-ngl 99 means offload what fits. Q6_K 1-GPU 4K on the sweep used --fit-target 1024 and is partly on DDR5 — decode there is not in-VRAM. Expert offload (-ot, --n-cpu-moe) is the MoE form; exact values from the rig owner are not recorded.

See also: Expert offload · System RAM · VRAM vs system RAM · mmap · --mlock

Out of VRAM

Also: OOM · ErrorOutOfDeviceMemory · hipErrorOutOfMemory · failed to allocate

llama.cpp could not allocate GPU memory. Vulkan prints ErrorOutOfDeviceMemory or failed to allocate Vulkan0 buffer. HIP prints hipErrorOutOfMemory or failed to allocate ROCm0 buffer. Those strings are quoted from upstream issues.

On a home Radeon box. The shopper ladder is /out-of-vram: cheapest first, then size the file on the calculator. Prefer a smaller -c or q8_0 KV before you drop the weight quant. House recipes stay Q4 and up. Q3 is a labeled estimate.

Go deeper

Cited strings: llama.cpp #14836 / #15120 / #9271 (Vulkan), #16725 / #8618 (HIP), Ollama #8734 (system memory). LM Studio wraps llama.cpp — AMD shoppers still see the Vulkan or HIP strings. House 35B UD-Q5 on one R9700 is in-VRAM at 262144 f16 KV.

See also: VRAM vs system RAM · KV cache · Offload · llama.cpp

P

Parallel slots (-np)

Also: -np · --parallel · llama-server slots · concurrency-ab

How many concurrent sequences one llama-server accepts. Official flag is -np / --parallel (default auto). Each slot needs KV. Usable context per client is total ctx / np unless you raise -c or set --kv-unified-per-slot.

On a home Radeon box. Hermes, Open WebUI, and a coding tool often share one always-on unit. House daily driver stays one slot. AMDai has not measured -np 2/4/8 on one R9700 — run Stack Lab concurrency-ab/1. Two GPU copies are a different layout.

Go deeper

llama.cpp server README: -np / --parallel N, --kv-unified-per-slot N, GET /health total_slots. Pack concurrency-ab/1 is amdai-bench ab-concurrency. That pack is not an allowlisted Open-in-Stack-Lab deep link. Do not copy a one-client 35B tok/s as an -np 4 figure.

Prompt cache

Also: prompt caching · --cache-prompt · cache_prompt · LLAMA_ARG_CACHE_PROMPT

llama-server keeps processed prompt tokens so a later request that shares a prefix skips that prefill. Official --cache-prompt / --no-cache-prompt (default on; env LLAMA_ARG_CACHE_PROMPT). The completion API field cache_prompt does the same per request (default true).

On a home Radeon box. Hermes, coding tools, and chat UIs resend a long system prompt every turn. The cache helps when that prefix stays put. AMDai has not measured TTFT with versus without the cache on the local desktop — not yet measured. Run Stack Lab prompt-cache-ab/1.

Go deeper

Official docs warn logits are not guaranteed bit-for-bit identical across batch sizes, so cache_prompt can be nondeterministic. House llama-server does not record a --cache-reuse, --cache-ram, or --slot-save-path pin. The shopper page is /prompt-cache.

Prefix cache

Also: cached prefix · common prefix · prefix reuse

Another name for llama.cpp prompt cache: the shared leading tokens of two requests. Official cache_prompt skips that prefix and prefills only the new suffix.

On a home Radeon box. A stable Hermes system prompt, thinking instructions, or a RAG preamble can reuse the prefix. A one-shot prompt or a prefix that changes every turn cannot. Cache-hit TTFT stays not yet measured.

Go deeper

Shoppers search llama.cpp prompt cache AMD and llama-server cache-reuse. The dedicated page is /prompt-cache. Do not copy an overnight llama-bench decode cell as a cache-hit first-token figure.

Prefill vs decode

Also: prefill · decode · tok/s · tokens per second · pp · tg

Prefill (prompt processing) ingests the prompt in a burst — high tok/s, one shot. Decode (token generation) is the streaming rate people mean by “how fast is it.” AMDai cards report both.

On a home Radeon box. They are not interchangeable. Official UD-Q5_K_XL 4K on one R9700: 2761.6 tok/s prefill and 102.39 tok/s decode. At 256K the same file is 519.2 / 44.4. Quote decode for chat feel; look at prefill when long prompts stall.

Go deeper

Official UD-Q5_K_XL 4K: 2761.6 tok/s prefill, 102.39 decode. At 256K: 519.2 / 44.4. Q4_K_M 32K sibling: 1902.1 prefill, 117.69 decode. Quote the card, not a blended average.

Q

Quant

Also: Q6 · Q5 · Q4 · Q3 · Q6_K · Q5_K_M · Q4_K_M · quantization · bpw

A stored-weight encoding that shrinks the GGUF. AMDai lists Q3 through BF16. Q6 / Q5 / Q4 are measured on the local desktop for the 35B file. Q3 rows are options labeled estimate or community unless a run card says measured.

On a home Radeon box. Prefer Q5 over Q4 when the file still fits; UD-Q5_K_XL is the daily driver. Q3 is a listed size step, not a secret “must stay at Q4” floor. This site never invents Q1 or Q2 guidance — those stay off, including draft heads.

Go deeper

Nominal llama.cpp-family bits in the calculator: Q3_K_S 3.50, Q3_K_M / UD-Q3_K_XL 3.91, Q4_K_M 4.85, Q5 family 5.69, Q6 family 6.56, Q8_0 8.5, BF16 16. Q3 file sizes are Unsloth listings, not local desktop quality scores. Q1/Q2 stay off.

See also: GGUF · Unsloth UD · KV cache · imatrix

QLoRA

Also: Quantized LoRA · 4-bit LoRA · load_in_4bit

LoRA plus a 4-bit quantized base. Unsloth’s table treats QLoRA as 4-bit. bitsandbytes is the usual loader. AMD’s Radeon playbook: QLoRA is Linux-only because bitsandbytes is not well supported on Windows.

On a home Radeon box. The memory-efficient path on a home Radeon. Unsloth 27B QLoRA minimum is 22 GB; 70B is 41 GB. Those are Unsloth minima, not local desktop runs. Training speed is not yet measured on our R9700s.

Go deeper

HF TRL documents load_in_4bit with peft_config. Unsloth documents bitsandbytes QLoRA on Radeon and CDNA. After training, merge the adapter and convert to GGUF — llama.cpp first for tokens.

Qwen3.6-35B-A3B

Also: 35B · Qwen 3.6 · A3B

The house daily-driver model: a 35B-class MoE instruct GGUF. The file we measure is Unsloth UD-Q5_K_XL (publisher listing 26.6 GB). Sibling 35B points on the same sweep include Q5_K_M, Q4_K_M, and Q6_K.

On a home Radeon box. It is the first Qwen AMDai names — not a smaller “starter” from an older Qwen line. Fit and speed live on /qwen and the official cards; Q3 35B listings are estimates unless measured.

Go deeper

Sibling 35B points on the same sweep include Q5_K_M, Q4_K_M, and Q6_K. Recipe qwen36-35b.llamacpp-vulkan.262k is the daily-driver launch. Q3 35B listings stay estimates unless measured.

See also: Daily driver · MoE vs dense · Quant

R

Radeon AI PRO R9700

Also: R9700 · R9700 Pro · AI PRO R9700

The AMD workstation / local-AI card this notebook is grounded in: 32 GB, gfx1201, RDNA 4. Official product name — there is no consumer “RX 9700.” The gaming sibling is the RX 9070 series (also gfx1201, 16 GB on the XT).

On a home Radeon box. local desktop has two of them. One card holds the daily-driver 35B Q5 in VRAM; two cards are for copies or Flash offload, not automatic 2× speed. Check the GPU directory before you buy a PSU.

Go deeper

32 GB workstation SKU, gfx1201. Official product name is AMD Radeon AI PRO R9700. One card is the daily driver; two cards are copies or Flash offload, not automatic 2× decode. Size a PSU on /hardware — we do not invent a wattage.

RDNA 4

Also: RDNA4 · RDNA4/gfx1201 · gfx12

The Radeon architecture family whose LLVM target is gfx1201 on current ROCm hardware tables. Covers the Radeon AI PRO R9700 and RX 9070-class cards — not RDNA 3 (gfx1100, 7900 XTX) and not Instinct CDNA.

On a home Radeon box. ROCm packages and Vulkan drivers are family-specific. A 7900 XTX recipe is a different gfx; do not paste gfx1100 HIP flags onto a R9700 and expect them to be official.

Go deeper

RDNA 3 (gfx1100, 7900 XTX) is a different ROCm family package. RX 9070 XT is gfx1201 with 16 GB — same generation, half the VRAM. There is no consumer RX 9700.

See also: gfx1201 · Radeon AI PRO R9700 · ROCm · Used Radeon

Resizable BAR

Also: ReBAR · Smart Access Memory · SAM · Above 4G

A BIOS feature that lets the CPU see the GPU’s full VRAM window instead of a small aperture. AMD’s consumer name is Smart Access Memory. Menu labels vary by board.

On a home Radeon box. On a home Radeon box, turn it on with Above 4G Decoding and CSM off — RDNA 4 is UEFI-only. AMDai has not measured llama.cpp or Strata tok/s with Resizable BAR on versus off on the local desktop, so do not invent a speed claim.

Go deeper

ROCm BAR notes also mention keeping the high MMIO aperture inside the device physical-address limit. Two 32 GB cards plus a 256 GB host map a large prefetchable BAR. Record the exact BIOS string on a board row; do not assume one label.

ROCm

Also: Radeon Open Compute · Core SDK · TheRock

AMD’s open GPU compute platform: compilers, runtimes, and libraries for Linux (and a thinner Windows story). Historically Radeon Open Compute. It is not CUDA, and Canonical’s archive rocm is not the current Core SDK.

On a home Radeon box. You need a current AMD pin for gfx1201 — Core SDK 10.x / ROCm CLI, not a mixed apt install. Vulkan llama.cpp can generate before HIP userspace is perfect; still verify rocminfo when you want HIP.

Go deeper

Core SDK 10.x (ROCm CLI v0.2.0 or AMD’s 10.1 repo) is the current consumer pin for gfx1201. Do not mix those with Ubuntu archive rocm. Moving versions, pinning, or recovering a half-upgraded box is /rocm/upgrade. The ROCm-guide glossary cites AMD sentences for HIP, ROCr, and TheRock.

Run card

Also: test-run card · results card · official card

One official amdai-bench/run@1 measurement published on /results: who ran it, which GPU layout, the exact launch command, prefill/decode tok/s, VRAM, and peak system RAM when recorded.

On a home Radeon box. AMDai does not invent a number to fill a table cell. If the 35B page shows 102.39 or 42.9 GB, click through to the card. The 2026-10-12 overnight sweep is 33 cards from local desktop — measured, pending re-verify after the DDR5 oops.

Go deeper

Each /results page shows the launch command and pins. The 35B table links cells to cards instead of inventing a missing figure. Q6_K 1-GPU 4K uses the 23:55 card (103.23 tok/s). The earlier 82.62 card stays on /results and is not averaged.

See also: amdai-bench · local desktop · Prefill vs decode

Repeat penalty

Also: --repeat-penalty · repeat_penalty · repetition_penalty · presence_penalty · LLAMA_ARG_REPEAT_PENALTY

Official llama.cpp --repeat-penalty penalizes repeating token sequences (CLI default 1.00 = disabled; env LLAMA_ARG_REPEAT_PENALTY). --presence-penalty and --frequency-penalty default 0.00. Qwen cards write repetition_penalty; llama.cpp maps that to repeat_penalty.

On a home Radeon box. Loops and repetition are often this knob, not the GPU. Official Qwen cards allow presence_penalty 0–2. The /completion field repeat_penalty is documented as default 1.1 — an official discrepancy versus the CLI 1.00 default.

Go deeper

DRY (--dry-multiplier, default 0 = off) and XTC (--xtc-probability, default 0 = off) are separate official samplers. House recipes do not record any of them. The shopper page is /sampling.

S

Special tokens

Also: im_start · im_end · bos · eos · think tags

Control tokens the chat template inserts for roles, stops, and thinking blocks. They live in the GGUF tokenizer, not in the GGUF weights.

On a home Radeon box. A missing or wrong template drops those wrappers, so the model talks as the user or never stops thinking. Do not swap the quant to “fix” them.

Go deeper

Official inspect path is GET /props and POST /apply-template. Leaked think tags are /thinking. The shopper page is /chat-templates.

See also: Chat template · Jinja · GGUF

Slot save/restore

Also: --slot-save-path · POST /slots save · slot restore · LLAMA_ARG_SLOT_SAVE_PATH

Official --slot-save-path PATH enables saving a slot’s KV / prompt cache to disk (default disabled; env LLAMA_ARG_SLOT_SAVE_PATH). POST /slots/{id_slot}?action=save and action=restore take a filename under that directory.

On a home Radeon box. Useful when you want a parked prefix on disk instead of only in RAM. House llama-server does not record the flag. Slot TTFT stays not yet measured. -np sizes the slot count; --kv-unified shares one KV buffer.

Go deeper

--cache-idle-slots (default on; requires --cache-ram) parks an idle slot into the prompt cache so a unified buffer can take the next task. The shopper pages are /prompt-cache and /concurrency-ab.

See also: Prompt cache · Parallel slots (-np) · KV cache · TTFT

Speculative decoding

Also: draft model · model-draft · --model-draft · -md · spec decode

A smaller draft GGUF guesses the next tokens; the target accepts or rejects them. llama.cpp documents -md / --model-draft on the server README. AMDai has not measured a draft pair on the local desktop.

On a home Radeon box. A good draft can raise decode. A bad one wastes VRAM. Draft files stay Q4 and up. Q3 is a labeled estimate for the main file, not a house draft. MTP heads stay off.

Go deeper

Official current flags are --spec-draft-model, --spec-draft-n-max, --spec-draft-n-min, and -ngld. Old --draft-max / --draft-min were removed. Measure draft off vs on with the Stack Lab A/B. House 35B launches have no draft. Hermes’s Flash path is Strata with no MTP.

See also: llama.cpp · Quant · Prefill vs decode · Strata

Stack Lab

Also: Radeon Stack Lab · qwen-reliable · companion app

The Ubuntu 24.04+ companion that ships the same catalog as this site (Origin repo / local install — packages not published yet, no public download URL). Profile qwen-reliable is the daily driver; qwen-fast-agent is Flash on Strata.

On a home Radeon box. Use it when you want the YAML catalog on the box, not as a replacement for reading /setup. It does not invent tok/s; it records amdai-bench cards AMDai publishes.

Go deeper

Jobs: check my rig, setup doctor, model manager, apply recipes, start a profile, live dashboard, amdai-bench, share cards. Profiles qwen-reliable (35B Q5 llama.cpp) and qwen-fast-agent (Flash on Strata) match /stacks. Packages are not published yet.

See also: Daily driver · Fast agent · AMDai

Strata

Also: stratat · v0.1.42 · :8083

A GGUF server AMDai uses for the Flash fast-agent try. The pin on our rig is v0.1.42 (dated 2026-10-10). It is not the daily-driver engine.

On a home Radeon box. First-party Flash is faster on Strata than on llama.cpp (~32.6 vs ~16 tok/s, 2026-10). Get llama.cpp generating official instruct first; then point a harness at Strata :8083 if you want the agent path.

Go deeper

Pin v0.1.42 dated 2026-10-10. First-party Flash ~32.6 tok/s versus llama.cpp ~16. Draft / MTP heads that would be a Q1/Q2 path stay off. Point a harness at :8083 after llama.cpp already works.

See also: Fast agent · Qwen 3.8 Flash · llama.cpp

System RAM

Also: DDR5 · DIMM · host RAM · usable RAM

Memory on the motherboard (DDR5 on the local desktop: 4×64 GB, 256 GB kit). Separate from VRAM. It holds the OS, offloaded tensors, and some KV when the file does not fit the GPU.

On a home Radeon box. Official UD-Q5_K_XL 1-GPU at 256K peaked at 42.9 GB system RAM — that is why the calculator can show a measured RAM line. Flash offload needs a much larger kit; spinning the DIMMs is not free speed.

Go deeper

local desktop kit is 256 GB (4×64 GB DDR5-5200); usable pool in the calculator is 249 GB. Daily-driver 256K peak 42.9 GB is a measured overlay, not an estimate. Flash offload kits are labeled estimates against that usable pool.

See also: VRAM vs system RAM · Offload · KV cache · APU

T

TTFT

Also: time to first token · first token · cache-hit TTFT · cache-miss TTFT

Time to first token — how long the user waits after send before the first decode token. Prompt / prefix cache aims to cut that wait when a long system prompt is resent.

On a home Radeon box. Shoppers search why is time to first token slow. AMDai has not measured llama-server TTFT with versus without the cache on the local desktop. Every cache-hit, miss, and first-token cell on /prompt-cache stays not yet measured.

Go deeper

Do not copy the overnight 102.39 tok/s 4K decode cell as a cache-hit TTFT. That number is llama-bench, one client, not a prefix-cache A/B. Run amdai-bench ab-prompt-cache and ingest a lab-notes-report/1 pack.

See also: Prompt cache · Prefix cache · Prefill vs decode · tok/s

tok/s

Also: tokens per second · throughput

Tokens per second. Decode tok/s is chat feel; prefill tok/s is prompt ingest. AMDai never fills a missing cell with a guessed rate.

On a home Radeon box. A home Radeon box should quote a card, not a viral screenshot. Short-context decode on the official 35B Q5 is 102.39 tok/s; that number does not apply to Flash, to Q3 estimates, or to a split layout.

See also: Prefill vs decode · Run card · amdai-bench · Thinking mode

Thinking mode

Also: reasoning mode · --reasoning · enable_thinking · reasoning_effort

A chat-template behavior where the model writes a hidden thought block, then a final answer. llama.cpp documents --reasoning on|off|auto, --reasoning-effort, --reasoning-budget, and chat_template_kwargs.enable_thinking.

On a home Radeon box. Official Qwen3.6 cards think by default. Hidden tokens burn context and wall-clock. Turn it off with --reasoning off or reasoning_effort none. Thinking-token counts are not measured on the local desktop.

Go deeper

Server README (checked 2026-10-14): reasoning_effort none disables thinking when the template honors it. Official Qwen3.6 card does not support /think or /nothink. --reasoning-format deepseek fills message.reasoning_content. Strata Flash thinking is not verified here.

Temperature (--temp)

Also: --temp · --temperature · temperature · LLAMA_ARG_TEMPERATURE

llama.cpp sampler that scales logits before the rest of the chain. Official --temp / --temperature default 0.80 (env LLAMA_ARG_TEMPERATURE). The /completion field is temperature (default 0.8). 0 is greedy.

On a home Radeon box. Shoppers search llama.cpp temperature AMD when output looks random or stuck. House recipes do not record a --temp pin. Official llama.cpp defaults then apply unless a client sends the field. Sampler speed stays not measured yet.

Go deeper

Copy a value only from the official model card. Qwen3.6-35B publishes thinking 1.0 / coding 0.6 / instruct 0.7. gpt-oss, Gemma 3 27B, and Llama 3.3 70B do not publish a recommended set. The shopper page is /sampling.

Top-p (--top-p)

Also: --top-p · top_p · nucleus · LLAMA_ARG_TOP_P

Nucleus sampling. Official llama.cpp --top-p default 0.95; 1.0 disables (env LLAMA_ARG_TOP_P). The /completion and OpenAI-shaped field is top_p.

On a home Radeon box. Qwen cards pair top_p=0.95 with thinking and top_p=0.80 with instruct. House recipes do not record --top-p. Do not invent 0.9.

Go deeper

Works with --top-k (default 40; 0 disables) and --min-p (default 0.05). Official --samplers order includes top_p before min_p and temperature. The shopper page is /sampling.

Tool calling

Also: function calling · tools array · tool_calls · parallel_tool_calls

An OpenAI-shaped /v1/chat/completions request that includes a tools array. The model returns tool_calls when --jinja is on and the chat template understands tools.

On a home Radeon box. Hermes, Cline, Continue, and Roo need this path. Official docs are llama.cpp function-calling.md and the server README. Parallel / multiple tool calls stay off unless the payload sets parallel_tool_calls: true. Success is not measured on the local desktop.

Go deeper

Native handlers exist for several families; unrecognized templates fall back to Generic. House 35B: start llama-server with --jinja on :8080. Strata :8083 tool calls are not verified here. Structured JSON is a different knob — response_format / GBNF. Measure grammar-ab/1.

See also: Chat template · GBNF · JSON schema · llama.cpp

U

Micro-batch (-ub)

Also: -ub · --ubatch-size · ubatch · physical batch · LLAMA_ARG_UBATCH

Official llama.cpp physical maximum batch size. llama-server README: -ub / --ubatch-size N (default 512; env LLAMA_ARG_UBATCH). The chunk that actually hits the GPU during prefill.

On a home Radeon box. Larger -ub can raise prefill when unused VRAM remains for the compute buffer. Smaller -ub can fit a tight card. Never set -ub 65–256 on Qwen3.5 or Qwen3.6. House llama-server does not record a pin. llama-server -ub stays not measured yet — run amdai-bench ab-prefill.

Go deeper

Unused VRAM after weights + KV is the shopper headroom. House 32 GB R9700 at 256K UD-Q5 leaves about 2.7 GB after the measured 29.34 GB peak. llama.cpp #14836 failed to allocate Vulkan compute pp buffers at -b 4096 -ub 4096. Cut -ub when that allocate appears.

Unsloth UD

Also: UD-Q5_K_XL · UD-Q6_K_XL · UD-Q3_K_XL · dynamic quant

Unsloth’s dynamic GGUFs: a named bucket (UD-Q5_K_XL, UD-Q6_K_XL, UD-Q3_K_XL) that keeps some tensors at higher precision. When the publisher lists a file size, the calculator uses that size instead of params × bits.

On a home Radeon box. UD-Q5_K_XL is the daily-driver 35B file; UD-Q6_K_XL is the Flash agent file. UD-Q3_K_XL is the Q3 size step on the same cards — labeled estimate unless measured. We do not invent a quality delta.

Go deeper

When Unsloth lists a file size, the calculator uses that instead of params × bits. UD-Q3_K_XL 35B listing 16.8 GB is a size step, labeled estimate. Prefer the UD-Q5_K_XL / UD-Q6_K_XL names on /stacks.

See also: Quant · GGUF · Qwen3.6-35B-A3B · imatrix

UMA

Also: UMA frame buffer · VRAM carve-out · dedicated GPU memory

A BIOS reserve that hands the iGPU a fixed slice of system memory at boot. The rest of the kit stays with the OS unless GTT / dynamic allocation grows into it.

On a home Radeon box. A huge carve-out on a 64 GB Strix Halo box leaves little for the desktop. A tiny carve-out leans on GTT. Check the board menu; AMDai does not pin a megabyte number.

Go deeper

Same pool as system RAM. The two shopper labels stay Minimum system RAM needed and RAM needed at full context, adapted for what is left after the GPU share. Link /bios, /gpus/strix-halo, and /gpus/ryzen-igpu.

Used Radeon

Also: used GPU · used 7900 XTX · buying used Radeon

A second-hand consumer or workstation Radeon bought for local LLMs. AMDai sizes it by VRAM and gfx first — not by a street price — and vets it with lspci, rocminfo, rocm-smi or amd-smi, and vulkaninfo.

On a home Radeon box. Shoppers search “used 7900 XTX for llama.cpp.” The cited how-to is /guides/buying-used-radeon. Sheets stay on /gpus. llama.cpp Vulkan is the always-works path. Official ROCm follows the 10.1 matrix; consumer RDNA 2 and gfx1031 HSA_OVERRIDE stay unofficial. No invented prices or failure rates.

Go deeper

Used-market sheets on this site include RX 6800 XT, 6700 XT, 6950 XT / 6900 XT, 7900 XTX / XT, and 7800 XT. 32 GB measured recipes stay on the R9700. Q6/Q5/Q4 plus a labeled UD-Q3_K_XL estimate. Both RAM lines print where the file spills.

See also: VRAM vs system RAM · RDNA 4 · Vulkan · Radeon AI PRO R9700

V

vLLM

Also: vllm · ROCm vLLM · continuous batching

A high-throughput Python serving stack with continuous batching. Default pip install vllm is CUDA. ROCm builds are a separate pin — AMD’s conservative 10.0 combo is still 0.27; community 0.31 is this week’s serve try.

On a home Radeon box. Not the daily driver on a home Radeon box. Use it when several clients need one server, after llama.cpp already works. Do not mix vLLM pins in one venv, and do not start a first box here.

Go deeper

AMD’s 10.0 supported combo is still 0.27 + PyTorch 2.13. Community / ROCm 0.31 is this week’s serve try. Default pip install vllm is CUDA. Do not mix pins in one venv.

VRAM vs system RAM

Also: VRAM · GPU memory · 32 GB

VRAM is memory on the GPU; system RAM is DIMMs on the motherboard. A 32 GB R9700 holds weights + KV + overhead. Overflow goes to DDR5 and decode usually drops.

On a home Radeon box. Official 35B UD-Q5_K_XL on one card: 25.97 GB VRAM at 4K and 29.34 GB at 256K. That is in-VRAM. Flash and Q6-with-fit-target are not the same story — read the card before you assume the file “fits in 32.”

Go deeper

Official UD-Q5_K_XL 1-GPU VRAM: 25.97 GB at 4K, 26.7 at 32K, 28.82 at 128K, 29.34 at 256K. Peak system RAM at 256K is 42.9 GB. Size a different file on the calculator; do not reuse these as a universal 32 GB rule.

See also: System RAM · KV cache · Offload · Out of VRAM · APU · Used Radeon

Vulkan

Also: GGML_VULKAN · RADV · Mesa · build-vk

A Khronos graphics/compute API. llama.cpp can use it instead of HIP. On our rig that is Mesa RADV 25.2.8 talking to gfx1201 — the build-vk tree, not a consolation prize.

On a home Radeon box. Vulkan is the daily-driver path for the official 35B sweep. If HIP configure is a mess, generate on Vulkan first. After --list-devices, hide the iGPU if two devices appear.

Go deeper

House 35B cards use Mesa RADV 25.2.8. After llama.cpp --list-devices, pick the RADV GFX1201 device and hide the iGPU if two Vulkan devices appear. Vulkan is an on-ramp, not a fallback — it is what we run.

See also: llama.cpp · HIP · gfx1201 · Flash Attention (-fa)

W

WSL2

Also: WSL · Windows Subsystem for Linux · ROCm WSL2

Windows Subsystem for Linux 2. ROCm 10.1.0 lists it as a technical preview: Ubuntu guest, Windows host Adrenalin, GPU via /dev/dxg. Profiling, debugging, and KFD-dependent tools are out of scope. AMD SMI is discovery and monitoring only.

On a home Radeon box. People search “ROCm WSL2 Radeon” and “llama.cpp WSL AMD GPU.” The cited walk is /setup/wsl. It is not a measured local desktop card. Native Linux stays the measured path. Native Windows first token is llama.cpp Vulkan.

Go deeper

TheRock artifacts ship Ubuntu .deb and Python wheels, not RPM. rocminfo prints “WSL2 environment detected.” Missing /dev/kfd is a cited preview limit. Microsoft documents .wslconfig memory= (default 50% of Windows RAM). llama.cpp’s documented GPU path in a Linux guest is HIP (GGML_HIP=ON). We do not invent Vulkan-in-WSL.

See also: ROCm · HIP · Vulkan · llama.cpp