# AMDai > Local AI on consumer AMD Radeon — ROCm notes from a real R9700 test rig. The consumer AMD ROCm notebook: llama.cpp and Qwen on Radeon GPUs, grounded in a two-R9700 test rig. Daily driver, fast agent stack, GPU directory, and a quant + hardware calculator. Measured numbers come from amdai-bench/run@1 cards on a two-R9700 local desktop. Estimates stay labeled. Q3 is estimate-only. Q1/Q2 stay off. AMDai is not an AMD publication. ## Docs - [R9700 Lab Notes](https://amdai.net/): The consumer AMD ROCm notebook: llama.cpp and Qwen on Radeon GPUs, grounded in a two-R9700 test rig. Daily driver, fast agent stack, GPU directory, and a quant + hardware calculator. - [New models](https://amdai.net/new-models/): Daily Hugging Face Hub table of new open-weight text and vision releases. GGUF Q3–Q8. Runs-on from cited engine lists. Q4_K_M fit is a labeled estimate. - [Lab log](https://amdai.net/lab-log/): Results log of measured and cited findings, plus informative failures. Dual R9700 speed and the hourly Idea lab. No invented tok/s. - [Getting started](https://amdai.net/start/): Check a rig or budget, first-run Ubuntu setup, then a measured 35B recipe. - [First-run setup](https://amdai.net/setup/): Ubuntu 24.04+ to first token on a Radeon box. llama.cpp Vulkan first. - [GPU directory](https://amdai.net/gpus/): Radeon cards with gfx IDs, VRAM, official ROCm rows, and tested recipes. - [Radeon AI PRO R9700](https://amdai.net/gpus/r9700/): Measured 35B llama.cpp on the house 32 GB card — one GPU vs two. - [Where to buy the Radeon AI PRO R9700](https://amdai.net/where-to-buy-r9700/): US store listings for the 32 GB R9700. Dated Newegg snapshot. No invented price. - [Inference engines](https://amdai.net/backends/): llama.cpp first. Vulkan daily driver. Cited engines, no invented benches. - [Serve many users on two Radeon cards](https://amdai.net/serving/): Several users on 1–2 Radeon cards. llama.cpp -np and --split-mode first. Third-party figures labeled. Our lab 35B and Flash. - [Quant + hardware calc](https://amdai.net/calculator/): Size a GGUF against Radeon VRAM. Estimates stay labeled. No prices. - [Public API](https://amdai.net/api/): Static JSON at /api/v1/. GPU, model, stack, measured, and ROCm catalogs for assistants. MCP server in this repo. - [API index.json](https://amdai.net/api/v1/index.json): Endpoint list, schema version, and as_of for the public static API. - [Benchmarks](https://amdai.net/benchmarks/): Cited 35B llama.cpp matrix from local desktop amdai-bench/run@1 cards. - [Official run cards](https://amdai.net/results/): amdai-bench/run@1 cards credited to AMDai, with the exact launch command. - [How we test](https://amdai.net/methodology/): Measured vs estimate. Q3 is estimate-only. Q1/Q2 stay off. - [FAQ](https://amdai.net/faq/): First-time Radeon questions. Points at measured pages. - [Glossary](https://amdai.net/glossary/): Plain-English ROCm, GGUF, quant, and run-card terms. - [About AMDai](https://amdai.net/about/): Independent notebook. Not an AMD publication. - [Lab Notes digest](https://amdai.net/newsletter/): Email-only digest. Double opt-in. No tracking pixel. - [Sponsor](https://amdai.net/sponsor/): Sponsorship slots for AMD/Radeon-relevant vendors: homepage banner, featured GPU and backend listings, newsletter mention. Labeled Sponsored. Contact for rates. - [Windows setup](https://amdai.net/setup/windows/): Cited llama.cpp Vulkan first-run on Windows. - [Compare 35B](https://amdai.net/compare/): One GPU vs split, Q6/Q5/Q4 plus a labeled Q3 estimate. - [Compare GPUs](https://amdai.net/compare/gpus/): Pick two Radeon sheets. Fit estimates. No invented prices. - [Compare models](https://amdai.net/compare/models/): Pick two house or cited model sheets. No invented tok/s. - [Radeon vs NVIDIA](https://amdai.net/compare/radeon-vs-nvidia/): Cited specs. Measured R9700 35B. No invented tok/s. - [Can my Radeon run it?](https://amdai.net/can-run/): Model × GPU fit pages from the same sheets as the calculator. - [35B requirements](https://amdai.net/models/qwen36/requirements/): Qwen3.6-35B-A3B VRAM and GPU needs at 256K. - [Flash requirements](https://amdai.net/models/flash/requirements/): Qwen 3.8 Flash RAM and VRAM at 256K. 96 GB plus two R9700s for Q4. - [GLM Flash requirements](https://amdai.net/models/glm-53-flash/requirements/): GLM 5.3 Flash RAM kits at 256K. Estimates only. - [Best AMD GPU for local LLMs](https://amdai.net/best-amd-gpu-for-llm/): R9700 first at 256K. No invented street prices. - [Local LLM PC build](https://amdai.net/local-llm-pc-build/): Two AM5 + R9700 builds. 96 GB DDR5. No invented prices. - [R9700 review for local LLMs](https://amdai.net/gpus/r9700/review/): Measured 35B llama.cpp review from the local desktop. - [Best local coding model for 32 GB](https://amdai.net/best-local-coding-model-32gb/): Qwen3.6-35B-A3B Q5 on one R9700. Official scores plus measured speed. - [Harnesses](https://amdai.net/harnesses/): Local frontends that talk to a Radeon llama.cpp or Strata server. - [Recipes](https://amdai.net/recipes/): First-party launch recipes: one model on one card with a sourced proof. - [Models](https://amdai.net/models/): House Qwen plus cited open-weight pages. Q3 is a labeled estimate. - [Quants Q3–Q8](https://amdai.net/quants/): Why Q1/Q2 stay off this notebook. Unsloth UD is the house default. - [Stack Lab](https://amdai.net/stack-lab/): Ubuntu 24.04+ companion. Packages not published yet. - [Buying a used Radeon](https://amdai.net/guides/buying-used-radeon/): Cited shopper checklist. No invented street prices. - [Make GGUF quants](https://amdai.net/guides/quantize-gguf/): Cited llama.cpp convert + imatrix + Q6/Q5/Q4 how-to. - [Tool calling](https://amdai.net/guides/tool-calling/): Cited llama.cpp tools on a Radeon GPU. - [What is ROCm](https://amdai.net/rocm/): Stack, gfx IDs, Vulkan vs HIP, versions. - [ROCm problems](https://amdai.net/rocm-problems/): Cited consumer ROCm issues and workarounds. - [Dual / multi-GPU](https://amdai.net/multi-gpu/): Two R9700s: one model split, or two copies. Measured cards only. - [Dual x8/x8 motherboards](https://amdai.net/motherboards/): Cited AM5 boards that split CPU PCIe into x8/x8. No invented prices. - [Lab log as Markdown](https://amdai.net/lab-log.md): Every lab-log entry with metrics and sources, no JavaScript. - [Contact](https://amdai.net/contact/): Honest routes. No invented inbox. - [Affiliate disclosure](https://amdai.net/disclosure/): Amazon Associates and other programs when IDs are set. - [Privacy](https://amdai.net/privacy/): What optional accounts and the digest store. No tracking pixel. - [llms.txt](https://amdai.net/llms.txt): Short machine-readable map of the key AMDai pages. ## Sections - [Compare](https://amdai.net/compare/): 228 pages — 35B shopper questions, GPU-vs-GPU, and model-vs-model sheets. - [Can my Radeon run it?](https://amdai.net/can-run/): 162 pages — Model × GPU fit from the same calculator sheets. - [Harnesses](https://amdai.net/harnesses/): 4 pages — Local frontends, connect notes, quiz, and agent pages. - [Official run cards](https://amdai.net/results/): 49 cards — amdai-bench/run@1 cards credited to AMDai. - [GPU directory](https://amdai.net/gpus/): 27 pages — Radeon sheets with gfx IDs, VRAM, and recipes. - [Recipes](https://amdai.net/recipes/): 7 pages — First-party launch recipes with sourced proofs. - [Lab log](https://amdai.net/lab-log/): 101 entries — Measured and cited findings as of 2026-10-28. ## Key pages ### R9700 Lab Notes The consumer AMD ROCm notebook: llama.cpp and Qwen on Radeon GPUs, grounded in a two-R9700 test rig. Daily driver, fast agent stack, GPU directory, and a… Canonical: https://amdai.net/ ### Getting started Check a rig or budget, first-run Ubuntu setup, then a measured 35B recipe — llama.cpp first, then benchmarks and optional Stack Lab. Canonical: https://amdai.net/start/ ### First-run setup Ubuntu 24.04+ first token on a Radeon, plus an honest Windows path. llama.cpp first. Measured local desktop cards, no invented tok/s. Canonical: https://amdai.net/setup/ ### How we test How AMDai produces measured tok/s (amdai-bench/run@1), how estimates are labeled including Q3, and how citations and corrections work. Canonical: https://amdai.net/methodology/ ### About AMDai Who runs AMDai, what the local desktop Radeon test rig is, editorial independence, and how affiliate links work. Canonical: https://amdai.net/about/ ### GPU directory Radeon cards with gfx IDs, VRAM, official ROCm rows, and tested recipes. Spec-only cards stay labeled. Canonical: https://amdai.net/gpus/ ### Official run cards amdai-bench/run@1 cards from overnight local desktop tests — credited to AMDai, with the exact launch command. Canonical: https://amdai.net/results/ ### Compare Cited 35B decisions from official local desktop llama.cpp cards: one GPU versus split, and Q6 / Q5 / Q4 plus a labeled Q3 estimate. Canonical: https://amdai.net/compare/ ### Serve many users on two Radeon cards Serve several users on 1–2 Radeon cards. llama.cpp -np and --split-mode first. Third-party figures labeled. Our lab 35B and Flash. Q3 is a labeled estimate. Canonical: https://amdai.net/serving/ ### Can my Radeon run it? Pick a cited Radeon sheet and a cited /models page. Q6/Q5/Q4 fit plus a labeled Q3 estimate. No invented speeds. Canonical: https://amdai.net/can-run/ ### Harnesses What to plug a local Qwen into after tokens work — agents, chat UIs, and ComfyUI. llama.cpp first. Canonical: https://amdai.net/harnesses/ ### Recipes First-party Radeon launch recipes: one model on one card with ROCm metadata, a sourced proof, and the exact launch command. llama.cpp first. No invented tok/s. Canonical: https://amdai.net/recipes/ ## FAQ ### What is AMDai? AMDai is an independent consumer notebook for local AI on AMD Radeon and ROCm. It is not an AMD publication, not a store, and not a scraped leaderboard. Official numbers come from amdai-bench/run@1 cards on the local desktop R9700 rig unless a cell says estimate or cited. Who runs it and how we test live on About and Methodology. ### How do you label measured vs estimate? A measured cell cites an official amdai-bench/run@1 card. Calculator totals, Q3 35B rows, and Windows paths stay labeled estimate unless a card says measured. Empty stays not measured. We do not invent tok/s or prices to fill a hole. The full rule set is on How we test. ### Do store buttons earn affiliate commission? Some Check-price buttons are Amazon Associates, Rakuten (Newegg), or B&H when those IDs are set. A commission does not change the price you pay and does not change the recommendation. We do not show a live or stale price. Full program notes are on the affiliate disclosure. ### Can vendors sponsor AMDai? Yes. AMDai sells labeled Sponsored slots for AMD and Radeon-relevant vendors: a homepage banner, featured listings on the GPU directory and inference-engines hub, and a newsletter mention. Rates are not listed — contact for rates. A booking does not change measured recommendations. ### How do I follow Lab Notes without an account? Subscribe to the Lab Notes RSS feed for posts, dated updates, and official run cards. The email digest is optional and email-only. We do not invent a social account. ### Which GPU should I start with? The measured house box is one AMD Radeon AI PRO R9700 (32 GB, gfx1201). There is no consumer RX 9700. A 16 GB RX 9070 XT is the same generation with half the VRAM — see the RX 9070 XT recipe sheet for calculator fit estimates, not local desktop tok/s. The non-XT RX 9070 is the 16 GB / 220 W gfx1201 sheet next to it; the 12 GB RX 9070 GRE is the 192-bit / 432 GB/s GRE sheet. The cheaper 16 GB RDNA 4 pick is the RX 9060 XT (gfx1200) — same estimate-sheet rules, different family package. The 24 GB RX 7900 XTX is the RDNA 3 gfx1100 used-card sheet — AMD cites 24 GB GDDR6 and 355 W typical board power; official on the cited ROCm 10.1 matrix. The 20 GB RX 7900 XT is the same gfx1100 family with AMD citing 315 W TBP — not the 24 GB XTX and not the 16 GB 7900 GRE. The 16 GB RX 7800 XT is the gfx1101 used-card sheet — AMD cites 16 GB GDDR6 and 263 W typical board power; official on that same 10.1 matrix. The 16 GB 7900 GRE sits between the 7900 XT and 7800 XT on the gfx1100 pin. The 12 GB 7700 XT is the next smaller gfx1101 sheet after the 7800 XT. The 16 GB 7600 XT is the gfx1102 budget 16 GB sheet — not the 8 GB RX 7600, which has its own estimate sheet. The 16 GB RX 6800 XT is the RDNA 2 gfx1030 estimate sheet — unofficial on the current Radeon ROCm row, Vulkan first. The shared RX 6950 XT / 6900 XT sheet is the other 16 GB gfx1030 consumer pair (335 W / 576 GB/s vs 300 W / 512 GB/s). The 12 GB RX 6700 XT is the RDNA 2 gfx1031 used-card sheet — not the 6800 XT and not the 10 GB non-XT 6700; AMD cites 12 GB GDDR6 and 230 W typical board power. Workstation PRO sheets: W7900 is 48 GB gfx1100 (864 GB/s, 295 W TBP); W7800 is 32 GB gfx1100 (576 GB/s, 260 W TBP) with a separate AMD-listed 48 GB SKU; W7700 is 16 GB gfx1101 (576 GB/s, 190 W TBP); W6800 is the official 32 GB gfx1030 RDNA 2 row (512 GB/s, 250 W TBP). None of those four are measured on the local desktop. Mini-PCs on Ryzen AI Max / Strix Halo (Radeon 8060S, gfx1151) are a shared-memory APU, not a discrete VRAM card — see that cited guide. Confirm the SKU on the R9700 measured page and the GPU directory, then size the file on the calculator before you buy a second card. ### How do I pick a budget tier? Setup by budget is four tiers from a ~16 GB card to the two-R9700 Pro rig. Size vs speed changes the pick. Buttons say Check price — we do not invent a street price. Consumer 7900 / 7800 sheets are estimates; the R9700 page is the measured north star. ### How do I size a PSU for a Radeon local-AI box? Read AMD’s published TBP on the product page. The R9700 is 300 W TBP with a 750 W minimum PSU recommendation (product page). Other directory cards stay not recorded until we cite that page. Two 300 W TBP cards plus 750 W minimum per card is not a dual-card PSU SKU we invent. 1000W-class is search guidance, not a SKU. The cited shopper guide is /power. llama.cpp first for tokens. Q3 stays a labeled estimate. Nothing below Q3. ### Radeon or NVIDIA for a local LLM? Radeon wins the VRAM class we shop — 32 GB on the R9700, 24 / 20 / 16 GB on the RX 7900 / 7800 sheets, two-card layouts we measured on the local desktop, and our cited llama.cpp numbers. NVIDIA wins CUDA day-one: vLLM and a longer tail of engines and models land there first. Specs cite the vendor page. NVIDIA tok/s stay “no cited number yet” unless a third-party figure is linked. Buttons say Check price — we do not print a figure. The shopper page is Radeon vs NVIDIA. ### Radeon or a Mac (Apple Silicon) for a local LLM? Radeon wins the path we measured: one R9700, llama.cpp Vulkan, house 35B. A 7900 XTX is the 24 GB discrete sibling. Two R9700s are 64 GB of VRAM we have split cards for. Strix Halo is AMD’s unified-memory option (max 128 GB). A Mac Studio lists larger soldered unified kits and llama.cpp Metal — Mac speed stays not measured here unless a third-party llama.cpp figure is linked. Buttons say Check price. The shopper page is Radeon vs Mac. ### Can I run a local Qwen on a Ryzen AI Max / Strix Halo mini-PC? Yes as a cited path, not a measured local desktop card. Ryzen AI Max+ 395 with Radeon 8060S is gfx1151 on AMD’s ROCm 10.1 matrix. It is an APU: the GPU uses shared system memory (we size 64 GB and 128 GB kits). llama.cpp Vulkan is the first token. HIP is second. The MES REMOVE_QUEUE hang is on the ROCm problems tracker. Size the file on the calculator. We do not invent tok/s for this box. ### Can I run a local Qwen on a Ryzen 780M or 890M iGPU? Yes as a cited path, not a measured local desktop card. Radeon 780M / 760M are gfx1103 (Phoenix / Hawk Point). Radeon 880M / 890M are gfx1150 (Strix Point). The iGPU uses system RAM — size a 32 / 64 / 96 GB kit on the calculator. llama.cpp Vulkan is first and does not need ROCm. Hawk Point and Strix Point are named on the ROCm 10.1 matrix; desktop 8700G is not named there. HSA_OVERRIDE_GFX_VERSION is unofficial. We do not invent tok/s. Strix Halo (8060S, gfx1151) is a different sheet. ### Ubuntu or Windows for a first local LLM box? Ubuntu LTS named on AMD’s current ROCm matrix is the path we write. First-run setup walks Ubuntu 24.04+ to the first token. Linux Mint, Pop!_OS, and Kubuntu are Ubuntu-based but are not on that 10.1 OS row — follow /setup only as unofficial, not verified on our rig. Cited Fedora and Arch guides live at /setup/fedora and /setup/arch. Docker or Podman is a cited isolated path at /setup/docker — official llama.cpp images, not a container tok/s card. LM Studio is a cited desktop llama.cpp wrapper at /setup/lm-studio — Vulkan runtime first on Radeon, local /v1 on :1234, not an LM Studio tok/s card. Lemonade Server is AMD’s own llama.cpp wrapper at /setup/lemonade — Vulkan first, local /api/v1 on :13305, not a Lemonade tok/s card. The Windows guide at /setup/windows is cited vendor docs, not measured on the local desktop. The WSL2 guide at /setup/wsl is the same preview AMD names in the 10.1 notes — KFD tools out of scope, AMD SMI monitoring-only, not a measured local desktop daily driver. Radeon Stack Lab follows that Ubuntu path — packages are not published yet. Vulkan llama.cpp is the honest Windows path for tokens. Full ROCm userspace is a Linux product. We do not have Windows, WSL, Fedora, Arch, container, LM Studio, or Lemonade tok/s cards. ### Can I run llama.cpp on Radeon in Docker or Podman? Yes as a cited path, not a measured local desktop card. Official ggml-org images are server-vulkan first and server-rocm second. Pass /dev/dri for Vulkan; add /dev/kfd and optional seccomp=unconfined for ROCm. Bind the published port to 127.0.0.1. Rootless Podman needs keep-groups; Fedora SELinux needs container_use_devices. Q6/Q5/Q4 first; Q3 stays a labeled estimate. We do not invent container tok/s or a speed delta versus bare metal. ### Can I run LM Studio on a Radeon GPU? Yes as a cited path, not a measured local desktop card. LM Studio is a desktop llama.cpp wrapper. AMD’s playbook documents Vulkan and ROCm runtimes; Vulkan is the first pick on RDNA 2 / 3 / 4 and the R9700. Official LM Studio docs do not publish a per-SKU Radeon list — if the ROCm runtime is missing or the GPU meter stays idle, stay Vulkan or move to /setup. Windows installer or Linux AppImage (Ubuntu 20.04+; versions newer than 22 are not well tested). GPU offload max, house file Qwen3.6-35B-A3B UD-Q5_K_XL at 262K. Q6/Q5/Q4 first; Q3 stays a labeled estimate. Local OpenAI-compatible /v1 is http://127.0.0.1:1234/v1 for harnesses like Hermes. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not measured inside LM Studio. We do not invent an LM Studio tok/s. ### Can I run Aider, Cline, or Continue on a Radeon GPU? Yes. Start llama.cpp llama-server first (Vulkan on Ubuntu 24.04+), then point the tool at http://127.0.0.1:8080/v1. House file is Qwen3.6-35B-A3B UD-Q5_K_XL at 262K on one R9700. Aider prefixes the model with openai/ and needs a non-empty API key. Cline uses the OpenAI Compatible provider — do not append /chat/completions to the base URL. Continue sets apiBase including /v1. LM Studio :1234 is a secondary path. Q6/Q5/Q4 first; Q3 stays a labeled estimate. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not Aider/Cline/Continue tok/s. We do not invent a harness speed. ### Can I run Ollama on a Radeon GPU? Yes as a cited path, not a measured local desktop card. Ollama is a packaged llama.cpp / ROCm (or Vulkan) runtime. Linux first: curl -fsSL https://ollama.com/install.sh | sh, then the AMD ROCm tarball. Official gpu.mdx lists R9700 / 9070 XT on Linux ROCm v7; the Windows ROCm table does not name those cards. Prove the GPU with ollama ps. Default context is 4096 — set OLLAMA_CONTEXT_LENGTH. KV grows with OLLAMA_NUM_PARALLEL. House family is Qwen3.6-35B-A3B; official library Q4 is qwen3.6:35b-a3b-q4_K_M; import the house UD-Q5 GGUF for the file we measure. Q3 stays a labeled estimate. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not Ollama tok/s. llama.cpp you compile remains the daily driver. gfx1201 tensiles and Windows R9700 CPU fallback live on /rocm-problems. ### Can I run Lemonade Server on a Radeon GPU? Yes as a cited path, not a measured local desktop card. Lemonade is AMD’s own local AI server. The primary LLM recipe wraps llama.cpp. Ubuntu 24.04+ first: sudo add-apt-repository ppa:lemonade-team/stable then sudo apt install lemonade-server (docs checked 2026-10-14; latest GitHub tag v2026.41.1, 2026-10-07). Windows is the MSI from lemonade-server.ai. Install llamacpp:vulkan first on this site; llamacpp:rocm is second and only on the gfx their docs name (RX 6000/7000 RDNA2/RDNA3/RDNA4, Ryzen AI iGPUs; nightly bins gfx1150 / gfx1151 / gfx103X / gfx110X / gfx120X). OpenAI-compatible base is http://127.0.0.1:13305/api/v1 — note /api/v1, not /v1. House family is Qwen3.6-35B-A3B; pull shape unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q5_K_XL. Q6/Q5/Q4 first; Q3 stays a labeled estimate. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not Lemonade tok/s. llama.cpp you compile remains the daily driver. AMDai runs Lemonade on the local desktop; there is still no Lemonade speed card. ### Can I run Open WebUI on a Radeon GPU? Yes as a cited path, not a measured local desktop card. Open WebUI is a browser chat UI. It does not run the GGUF. Start llama.cpp llama-server first (Vulkan on Ubuntu 24.04+), then Docker or pip. Official Docker maps 3000:8080; pip defaults to :8080 so use --port 3000 or it collides with house llama-server. Connections → OpenAI URL http://127.0.0.1:8080/v1 (host.docker.internal from a container). House file is Qwen3.6-35B-A3B UD-Q5_K_XL. Q6/Q5/Q4 first; Q3 stays a labeled estimate. Do not pull the :cuda image on a Radeon. Decode numbers on this site are Ubuntu llama.cpp llama-bench, not Open WebUI. Ollama is a secondary connection. ### Can I run local AI on a Radeon under Windows WSL2? Yes as a cited technical preview, not a measured local desktop card. ROCm 10.1.0 lists WSL2 in TheRock artifacts: Ubuntu 26.04 / 24.04 / 22.04 guests, host Adrenalin then guest ROCm via /dev/dxg. Profiling, debugging, and KFD-dependent tools are out of scope; AMD SMI is monitoring-only. Native Linux at /setup is still the measured path. Native Windows first token is llama.cpp Vulkan at /setup/windows. Inside the guest, llama.cpp HIP (GGML_HIP=ON) is the documented GPU build — we do not invent Vulkan-in-WSL. Q6/Q5/Q4 first; Q3 stays a labeled estimate. Nothing on /setup/wsl was measured on the local desktop. ### Can I run Stable Diffusion or Flux on a Radeon, not just a local LLM? Yes. ComfyUI is the AMDai first image-gen path — HIP/ROCm PyTorch, not llama.cpp Vulkan, and it does not speak llama-server /v1. First-run install is /setup/comfyui. Fit tables stay on /image-gen. Linux Ubuntu 24.04+ first. Image-gen speeds are not yet measured by AMDai. llama.cpp stays first for LLMs. Q3 stays a labeled estimate. Nothing below Q3. ### How do I install ComfyUI on a Radeon GPU? Cited walk is /setup/comfyui: Ubuntu 24.04+ git clone, a ROCm PyTorch venv, pip install -r requirements.txt, then python ComfyUI/main.py at http://127.0.0.1:8188. Fit and family tables stay on /image-gen. ComfyUI does not speak llama-server /v1. Image-gen speeds are not yet measured by AMDai. Q3 stays a labeled estimate on any LLM that shares the box. Nothing below Q3. ### How do I run llama.cpp across two Radeon GPUs? Cited walk is /setup/multi-gpu. llama.cpp first: --list-devices, then --device or GGML_VK_VISIBLE_DEVICES on Vulkan and HIP_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES on HIP. Official default is --split-mode layer; --split-mode row is deprecated; --split-mode tensor is experimental. Two patterns: one model split across both cards, or two llama-server copies pinned one per card (house ports :8080 and :8081). Overnight 35B UD-Q5 at 4K is 102.39 tok/s on one R9700 and 82.78 on a layer split. Combo-preset tok/s is not measured — use amdai-bench. Q3 stays a labeled estimate. Nothing below Q3. ### How do I keep llama-server running after I log out? Cited walk is /setup/server. A systemd Type=simple unit with enable --now. House bind is 127.0.0.1:8080. Confirm GET /health and /v1/models before you attach a harness. Dual-card units stay on /setup/multi-gpu. Shared clients use /concurrency-ab for -np. Q6/Q5/Q4 first; Q3 stays a labeled estimate. We do not invent a service tok/s. ### How do I pick and vet a used Radeon for local AI? Cited shopper guide is /guides/buying-used-radeon. Buy VRAM first (12 / 16 / 20 / 24 / 32 GB) using the existing /gpus sheets and calculator — Q6/Q5/Q4 plus a labeled UD-Q3_K_XL estimate. llama.cpp Vulkan is the always-works path. Official ROCm follows the 10.1 matrix; consumer RDNA 2 and gfx1031 HSA_OVERRIDE_GFX_VERSION stay unofficial. Confirm lspci, rocminfo, rocm-smi or amd-smi, and vulkaninfo before you pay. No invented prices, tok/s, or failure rates. Both RAM lines print where the file spills. ### Can I run speech-to-text or TTS on a Radeon, not just a local LLM? Yes. whisper.cpp Vulkan is the AMDai first STT path — ggml family, not llama-server /v1. Piper is the first TTS path (pip install piper-tts). The cited shopper guide is /speech. Linux Ubuntu 24.04+ first. Memory cells copy the official whisper.cpp README table. Speech speeds are not yet measured by AMDai. llama.cpp stays first for LLMs. Q3 stays a labeled estimate. Nothing below Q3. ### Can I chat with my own PDFs and notes on a Radeon, not just a local LLM? Yes. Local RAG on AMDai is chunk → embed → retrieve → answer. The first embed path is llama.cpp --embeddings and /v1/embeddings with a dedicated embedding GGUF. The chat sidecar stays llama.cpp Vulkan. The cited shopper guide is /rag. Linux Ubuntu 24.04+ first. Official nomic-embed-text-v1.5 Q6_K is 108 MiB; Q5_K_M 95 MiB; Q4_K_M 81 MiB. Q3_K_M (65 MiB) is a labeled estimate. Speeds and retrieval quality are not yet measured by AMDai. There is no house Stack Lab recipe. Nothing below Q3. ### Can I pass a picture to a local LLM on a Radeon? Yes. Local vision on AMDai is llama.cpp multimodal: a text GGUF plus a matching mmproj GGUF, served by llama-mtmd-cli or llama-server --mmproj. That is image in, text out — not ComfyUI image gen. The cited shopper guide is /vision. Linux Ubuntu 24.04+ first. Speeds are not yet measured by AMDai. The house 35B daily driver is text-only; we do not invent an mmproj for that file. Q3 stays a labeled estimate. Nothing below Q3. ### Vulkan or ROCm HIP? Consumer default on AMDai is llama.cpp Vulkan for the daily-driver 35B. HIP is the ROCm compile when you need it, and it is also where gfx1201 “no binary for GPU” reports show up. If HIP configure is a mess, try Vulkan first and read the cited problem row. The shopper compare is /compare/vulkan-vs-rocm. HIP decode on the local desktop stays pending. Engine notes live on the backends pages — llama.cpp first. ### Will ROCm work for a local LLM on Ubuntu in 2026? Yes for single-GPU inference if you pin a current AMD Core SDK (10.1 line as of the weekly card) and a matching runtime. It is still not CUDA. Packaging is the hard part. Longer answers stay on What is ROCm and the ROCm FAQ — this page does not repeat those guides. ### Can I apt install rocm from Ubuntu? You can. You should not, if you want Core SDK 10.0/10.1, gfx1201, or a current vLLM ROCm image. Canonical’s archive has lagged. Add AMD’s 10.1 repo or use ROCm CLI v0.2.0 — never both plus archive rocm. Walkthrough: Install ROCm. ### How do I upgrade ROCm on a Radeon GPU without breaking Linux? Cited shopper guide is /rocm/upgrade. Inventory first: rocminfo, amd-smi or rocm-smi, dpkg, dkms, /opt/rocm. Radeon Software / amdgpu-install does not support in-place upgrades — uninstall, then install. The ROCm 10.1 apt path documents install-and-update; if you still have 7.2.4 or older, uninstall first. apt-mark hold is unofficial (Ubuntu man-page, not an AMD pin). llama.cpp Vulkan usually survives a userspace swap; rebuild HIP and matching PyTorch ROCm wheels. No invented tok/s. ### Is “R9700 Pro” a real SKU? The product is the AMD Radeon AI PRO R9700. There is no consumer RX 9700 in AMD’s current lineup. The gaming sibling is the RX 9070 series. Names and gfx IDs: GPU directory. ### Should I start with vLLM? No. Consumers start on llama.cpp. vLLM is the multi-client serve try. AMD’s 10.0 supported combo is still 0.27 + PyTorch 2.13. Community / ROCm 0.31 is this week’s serve try. Default pip install vllm is CUDA. We do not invent a local desktop 0.31 tok/s. ### Can my Radeon run this model? Use /can-run. Each cited GPU sheet × cited /models page has a calculator verdict: fits fully on GPU, fits with RAM offload, needs 2 GPUs, or does not fit. Q6 / Q5 / Q4 plus a labeled Q3 estimate. Measured decode is only the official local desktop 35B llama.cpp Vulkan sweep on R9700. Everything else stays not measured yet. llama.cpp first. ### How do I speed up llama.cpp on Radeon? Fit the file first, then turn one cited knob at a time: -ngl, --threads, --threads-batch, -b / -ub, -fa, KV type, --split-mode, a Q4+ draft, or a vendor power cap. The shopper guide is /tuning. Flash Attention has its own page at /flash-attention. Prefill / batch has its own page at /prefill. Measured decode is the official local desktop 35B llama.cpp Vulkan sweep. Everything else stays not measured yet. Q3 is a labeled estimate. Draft models stay Q4 and up. llama.cpp first. ### Does speculative decoding speed up llama.cpp on a Radeon? AMDai has not measured a draft pair on the local desktop. A Q4+ same-family draft can help when acceptance is high (code, repetitive text) and both files fit. The house 35B-A3B already decodes fast on one R9700, so a draft can also waste VRAM. Draft speed stays not measured yet — run the Stack Lab speculative decoding A/B. Drafts stay Q4 and up. Q3 is a labeled estimate for the main file. The shopper page is /speculative-decoding. llama.cpp first. ### How do I force llama.cpp to emit valid JSON on a Radeon? Use constrained decoding: a GBNF grammar or a JSON Schema that llama.cpp converts to GBNF. Official flags are --grammar / --grammar-file, -j / --json-schema, -jf / --json-schema-file, plus response_format on /v1/chat/completions. Describe the shape in the prompt — the schema is not injected except during tool calling. Grammar decode versus free-form is not measured on the local desktop; run Stack Lab grammar-ab/1 and record an amdai-bench card. Q3 is a labeled estimate. llama.cpp first. ### How do I get OpenAI-style tool calls from llama-server on a Radeon? Start llama-server with --jinja and send a tools array on /v1/chat/completions. The shopper page is /guides/tool-calling. Official docs are llama.cpp function-calling.md and the server README. Native handlers exist for several families; unrecognized templates fall back to Generic. Structured JSON uses response_format, a JSON Schema, or GBNF — measure grammar cost on Stack Lab grammar-ab/1. Tool-call success is not measured on the local desktop. Q3 is a labeled estimate. llama.cpp first. ### How do I serve several clients from one llama-server on a Radeon? Cited path is llama-server -np / --parallel. Context per slot is total ctx / np unless you raise -c or set --kv-unified-per-slot. AMDai has not measured -np 2/4/8 on one R9700 — run Stack Lab concurrency-ab/1. The shopper page is /concurrency-ab. Two GPU copies stay on /qwen36-dual. Several users on 1–2 cards, including --split-mode and when to try vLLM or Radiance, live on /serving. Q3 is a labeled estimate. llama.cpp first. ### How do I serve several users on one or two Radeon cards? llama.cpp first. On one GPU use llama-server -np / --parallel. On two GPUs use --split-mode layer, or two pinned servers when the file fits one card. --split-mode tensor is experimental. Try vLLM TP=2 or Radiance only when you need tensor-parallel throughput and can live with gfx1201 bugs or a convert step. The shopper page is /serving. Third-party tok/s stay labeled. Our lab 35B and Flash cells are one-client or Flash-pair cards. Q3 is a labeled estimate. ### Should I turn Flash Attention (-fa) on for llama.cpp on a Radeon? Overnight 35B Vulkan llama-bench recorded -fa 1. Official values are on|off|auto (default auto). llama.cpp multi-gpu.md requires FA for tensor split and/or a quantized V cache. llama-server FA on versus off is not measured on the local desktop — run Stack Lab Flash Attention A/B. The shopper page is /flash-attention. House recipes stay Q4 and up. Q3 is a labeled estimate. llama.cpp first. ### How do I speed up llama.cpp prompt processing (prefill) on a Radeon? Official flags are -b / --batch-size (logical, default 2048) and -ub / --ubatch-size (physical, default 512). House llama-server does not record a pin. Overnight llama-bench prefill cells are not a llama-server -b / -ub A/B. AMDai has not measured server batch on the local desktop — run Stack Lab prefill-ab/1 (amdai-bench ab-prefill). The shopper page is /prefill. House recipes stay Q4 and up. Q3 is a labeled estimate. llama.cpp first. ### What does llama.cpp prompt / prefix cache do on a Radeon, and when does it help TTFT? Official llama-server reuses processed prefix tokens so a later request only prefills the new suffix. --cache-prompt is on by default. --cache-reuse N is KV-shift reuse (default 0 = off). --cache-ram is the MiB ceiling (default 8192). --slot-save-path enables POST /slots save/restore. AMDai has not measured TTFT with versus without the cache on the local desktop — not yet measured. Run Stack Lab prompt-cache-ab/1 (amdai-bench ab-prompt-cache). The shopper page is /prompt-cache. House recipes stay Q4 and up. Q3 is a labeled estimate. llama.cpp first. ### What is llama.cpp mmap versus --no-mmap versus --mlock on a Radeon? Official --load-mode auto memory-maps the GGUF (mmap default). --load-mode none replaces deprecated --no-mmap. --load-mode mlock locks a non-mapped copy; mmap+mlock locks a mapped copy. AMDai has not measured load time or extra RSS on the local desktop — not yet measured. Run Stack Lab Model loading A/B. The shopper page is /mmap. House recipes stay Q4 and up. Q3 is a labeled estimate. llama.cpp first. ### What is a llama.cpp chat template, and do I need --jinja? The Jinja string that wraps messages, tools, and reasoning. Official default is GGUF tokenizer.chat_template. llama-server README (checked 2026-10-14): --jinja default enabled; --chat-template / --chat-template-file override the metadata. House recipes still pass --jinja. Inspect GET /props. The shopper page is /chat-templates. Q3 is a labeled estimate. llama.cpp first. ### Why does the model talk to itself or leak think tags? Usually a wrong or missing chat template, not the Radeon or the quant. Confirm --jinja, then GET /props and POST /apply-template. Leaked thoughts are --reasoning-format or enable_thinking — see /thinking. Tools ignored are /guides/tool-calling. Start from /troubleshoot/errors (jinja-template). The shopper page is /chat-templates. Q3 is a labeled estimate. llama.cpp first. ### What llama.cpp sampling flags should I set on a Radeon? Official llama-server defaults are --temp 0.80, --top-k 40, --top-p 0.95, --min-p 0.05. House recipes do not record a sampler pin. Copy a Qwen card row only when that card publishes one. Sampler speed stays not measured yet. The shopper page is /sampling. Q3 is a labeled estimate. llama.cpp first. ### How do I turn thinking / reasoning mode off on llama.cpp? Cited path: llama-server --reasoning off, and/or reasoning_effort none, and/or chat_template_kwargs {"enable_thinking": false}. Official Qwen3.6 card thinks by default and does not support /think or /nothink. The shopper page is /thinking. Thinking-token counts and wall-clock stay not measured on the local desktop. Q3 is a labeled estimate. llama.cpp first. ### Why does a 35B MoE fit one 32 GB R9700 when dense 27B and Flash look different? MoE activates a slice; dense runs every parameter. Official Qwen3.6-35B-A3B is 35B total / 3B active — that is why UD-Q5_K_XL fits one 32 GB R9700 (publisher 26.6 GB; measured 25.97 GB VRAM at 4K and 29.34 GB at 256K). Dense Qwen3.6 27B and Qwen3.8 27B are a different file. Flash is a different MoE (official 125B / 6B plus a 51B n-gram) and usually needs DDR5 expert offload. The shopper hub is /moe. llama.cpp first. Q3 is a labeled estimate. Nothing below Q3. ### 35B or Flash — which Qwen should I run first? Daily driver: Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL on one R9700, llama.cpp Vulkan, 262144 context, f16 KV, no RAM offload. Measured on our rig, 2026-10, llama-bench: 102.39 tok/s at 4K, 90.52 at 32K, 66.43 at 128K, 44.4 at 256K. Fast agents: Qwen 3.8 Flash UD-Q6_K_XL on Strata — cited pair ~32.6 tok/s versus llama.cpp ~16 tok/s at 262K. House models only. Recipe and cards: stacks, recipes, /qwen. ### Why llama.cpp first instead of Strata? AMDai’s consumer default is llama.cpp (Vulkan first) for the daily-driver 35B. Strata is the first-party Flash try on the house Pro rig — the cited pair is ~32.6 tok/s versus llama.cpp ~16 tok/s at 262K on Qwen 3.8 Flash UD-Q6_K_XL. Start on llama.cpp unless the job is that Flash pin. ### Where do I see new open-weight model releases? The /new-models table is a daily Hugging Face Hub scan of text-generation and image-text-to-text rows from known labs plus trending. It lists params, architecture, context, license, GGUF Q3–Q8 packs, a Runs on column from cited engine lists (llama.cpp convert arch or GGUF, vLLM supported-architectures, Ollama library, Strata Flash notes on /backends; LM Studio and Lemonade follow GGUF; SGLang only when its architecture is listed; unknown if we lack evidence; MLX stays off), and a labeled Q4_K_M fit estimate for 16/24/32/2×32 GB. Not llama-bench. Q1/Q2 stay off. ### Where do I read AMDai lab results and failures? The /lab-log page is a public results log of measured and cited findings, plus informative failures, from Dual R9700 speed — Q4/Q5/Q6 Flash-class models on two R9700s — and the hourly Idea lab. It is not a status board. Empty cells stay empty. No invented tok/s. Entries are rendered into the HTML at build time from the repo snapshot. The page also reads /data/lab-log.json in the browser so a server can refresh newer live rows without a rebuild. ### How do I make my own GGUF quant for a Radeon GPU? Cited how-to is /guides/quantize-gguf. Ubuntu 24.04+ first: convert_hf_to_gguf.py to f16 or bf16, optional llama-imatrix, then llama-quantize to Q6_K, Q5_K_M, or Q4_K_M. Unsloth UD-*_XL stays the house default — stock llama-quantize does not reproduce those files. Q3 is a labeled estimate, not a house-rig quantize command. Verify with llama-perplexity or KL-divergence, then an amdai-bench run. Both RAM lines print on the guide. ### Q6, Q5, or Q4 — what should I pick? Those letters are GGUF quant recipes, not a speed ranking we invent. The overnight local desktop sweep measured UD-Q5_K_XL, Q5_K_M, Q4_K_M, and Q6_K on llama.cpp Vulkan. Read the cells on /benchmarks and the side-by-side on /compare. Q6 on one card at 32K and up is partly on DDR5 — that row says so. Fit the file on the calculator. ### When is Q3 only an estimate? UD-Q3_K_XL may appear in comparison chrome as a labeled estimate. The cited file listing is Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB). We do not invent a Q3 tok/s. A measured Qwen 3.8 Flash Q3 run is the approved one-time exception and is labeled on that card. Q3 is not a measured 35B overnight row. ### Why don’t you list Q1 or Q2? AMDai does not offer or recommend Q1 or Q2 quants. The calculator and compare pages list Q3 and up. Q3 stays a labeled estimate unless a card is measured. We will not invent a missing smaller file’s tok/s to fill a table. ### How do I read /benchmarks and /results? llama.cpp llama-bench prints pp (prefill) and tg (decode). Decode is the chat tok/s. /benchmarks is the 33-card local desktop overnight matrix: UD-Q5, Q5, Q4, Q6 × 4K / 32K / 128K / 256K × one GPU and split. Each cell links to its amdai-bench/run@1 card on /results — launch command, VRAM per card, VRAM total, peak system RAM. Empty stays not measured. The overnight cards recorded llama-bench, not llama-server. ### Will two R9700s double decode speed? Not automatically. On the overnight cards, one R9700 UD-Q5_K_XL at 4K decodes 102.39 tok/s; the two-card split is 82.78 tok/s. A split pays an RCCL / topology tax. Two copies of the same server can serve two requests; that is not “double one chat.” Read /multi-gpu for the why and /setup/multi-gpu for the llama.cpp flags, then measure on /compare/one-vs-two-gpu and /qwen36-dual before you buy a second card for speed. ### What is a two-GPU Combo versus a split? A split is one model cut across two cards (llama.cpp -sm layer on the overnight rows). A Combo on this site is two models at once — Flash on one card and 35B on the other — not a promise of 2× decode. Hands-on flags, ports, and systemd-style pins are /setup/multi-gpu. Read Dual / multi-GPU before you buy a second card for speed. ### Why does Q6_K 1-GPU 4K show 82.62 and 103.23 tok/s? The 35B table cell is 103.23 tok/s from the 23:55 card — the 4K point in the complete Q6_K single-GPU series. The earlier 23:26 card at 82.62 tok/s stays on /results and is not averaged into the cell. The whole sweep is measured, pending re-verify after the DDR5-5200 oops. Read both cards on /results and the caveat on /benchmarks. ### How much system RAM do I need at short context vs full context? The calculator prints two lines: “Minimum system RAM needed” (the load at the context you picked) and “RAM needed at full context” (262144 for the house 35B). Overnight UD-Q5_K_XL llama-bench on one R9700 recorded peak system RAM 37.32 GB at 4K and 42.9 GB at 256K (Linux GiB labeled GB). Those are measured peaks, pending re-verify after the RAM fix — not a kit you must buy. Change context on the calculator and read the pair. ### What is RAM offload and when do I use it? When the GGUF plus KV will not stay in VRAM, layers or experts spill to DDR5 and decode falls. Q6 35B on one R9700 at 32K and up is partly offloaded on the overnight cards. Qwen 3.8 Flash does not fit in one 32 GB card — hot experts stay in VRAM, the rest on DDR5. Kit sizes and measured DIMM-speed rows live on Flash RAM offload. We do not invent an offload tok/s when the card omitted one. ### Can I run 128K–262K locally on a Radeon? Yes on the measured house path: Qwen3.6-35B-A3B on llama.cpp Vulkan, f16 KV, one R9700 at 262144. Official UD-Q5_K_XL decode is 66.43 tok/s at 128K and 44.4 tok/s at 256K. The shopper guide is /long-context. Find the real ceiling with /context-ceiling (Stack Lab ctx-ceiling/1 — not yet ingested). Empty cells stay not measured yet. Q3 is a labeled estimate. We do not publish a planted-token recall score. llama.cpp first. ### What's the biggest context I can run on a Radeon GPU with llama.cpp? Native max is the model-card window. The real ceiling is the largest -c that still loads and serves. Stack Lab ctx-ceiling/1 (amdai-bench ab-ctx-ceiling) climbs a lean ladder on one GPU and on all GPUs. AMDai has not ingested a winner on the local desktop — not yet measured. House overnight llama-bench did run 262144 on UD-Q5, one R9700. The shopper page is /context-ceiling. Q3 is a labeled estimate. llama.cpp first. ### How much VRAM does the KV cache cost on a Radeon, and should I switch to q8_0? The shopper guide is /kv-cache. House daily driver stays f16 KV at 262144 on one R9700 — measured UD-Q5 peak is 25.97 GB at 4K and 29.34 GB at 256K. q8_0 and q4_0 shrink the estimated cache. Quality and tok/s for those types are not measured yet. llama.cpp multi-gpu.md requires flash attention for a quantized V cache — see /flash-attention. Print both RAM lines when the window spills. ### What is Radeon Stack Lab? An Ubuntu 24.04+ companion (Origin repo / local install — packages not published yet, no public download URL) with the same catalog as this site’s YAML. Jobs: check my rig, setup doctor, model manager, apply recipes, start a profile, live dashboard, amdai-bench, then share cards back. Profile qwen-reliable is the daily-driver 35B Q5. qwen-fast-agent is Flash on Strata. Lab Notes is the notebook; Stack Lab is the local app that records runs. ### How does Open in Stack Lab work? Allowlisted Lab Notes buttons use the documented amdai-stack-lab: screens (recipes, doctor, planner, library, support, profile). If the app is not installed, they fall back to Stack Lab install notes — packages are not published yet, and there is no public download URL. If Stack Lab is already running on this machine, the local app is 127.0.0.1:47321. House profiles qwen-reliable and qwen-fast-agent are the measured R9700 daily and Flash pins. They do not apply to untested consumer cards. ### How do recipes and amdai-bench connect? A recipe is a pinned launch on a named GPU — model, quant, engine, context, gates. Stack Lab’s amdai-bench emitter writes one official test-run card per line (schema amdai-bench/run@1). A shareable sweep is pack lab-notes-report/1 — Lab Notes ingests that pack onto data/runs/ and /results. The 35B matrix on /benchmarks is built only from those cards. Do not invent tok/s in a recipe; attach a log. ### Do I need a harness? No. You need tokens on the GPU first. Hermes or OpenClaw come after a loopback /v1 works. Local Hermes on the local desktop points at Strata :8083 and llama-server. Snippets: connect. Catalog: harnesses. A slow harness is usually a backend problem, not a card-reputation problem. ### llama.cpp says out of memory / ErrorOutOfDeviceMemory / hipErrorOutOfMemory — what do I change first? The shopper ladder is /out-of-vram. Cheapest first: lower -c, then q8_0 KV with -fa on, then -fa on, then a smaller -ub/-b, then MoE --n-cpu-moe / -ot if the file is MoE (do not invent N or the regex), then partial -ngl, then -ts / --split-mode, then close other VRAM users, then Q6 → Q5 → Q4. Cited strings come from llama.cpp #14836 / #9271 / #16725 / #8618 and Ollama #8734. LM Studio wraps llama.cpp — AMD shoppers still see the Vulkan or HIP strings. House recipes stay Q4 and up. Q3 is a labeled estimate. The existing /troubleshoot#oom-context row points here. ### The GPU is not seen or HIP configure is a mess — where do I start? Start from a symptom on /troubleshoot, or paste the log in the error matcher. Cited rows live on /rocm-problems: HIP no binary for GPU, Ubuntu archive packages shadowing AMD’s install, MODE1 reset / MES hang on gfx12, unofficial HSA overrides. Do not treat a community HSA_OVERRIDE as a measured fix. Walkthroughs: verify ROCm, then troubleshooting. ### Which BIOS settings matter for local AI? Cited chipset notes: Above 4G, ReBAR/SAM, CSM off, and matching GPU lane widths (x8/x8 or x16/x16 — not a leftover x4). The per-board table is empty until submissions. We do not invent a motherboard for local desktop. Read /bios before you file “ROCm is broken” on a firmware miss. ### Can I fine-tune like a 4090 blog post? PyTorch ROCm exists. The cited shopper guide is /fine-tuning — LoRA / QLoRA on consumer Radeon, then export to llama.cpp. ROCm 10.0 has a known issue: training / fine-tune GPU resets on R9700 — try TORCH_BLAS_PREFER_HIPBLASLT=0. That row is cited, not a measured tokens-per-second. Community training writeups often use older ROCm; keep inference on the pin you actually run. Training speed is not yet measured on our R9700s. Q3 stays a labeled estimate. ### What do ROCm, GGUF, and tok/s mean? Plain-English definitions live in the glossary — A–Z, searchable, same jargon the rest of AMDai uses. ROCm-stack citations stay on the ROCm-guide glossary. This FAQ does not duplicate those entries. ### Do you cite other people’s home labs? Yes, labeled Cited, not measured by AMDai. Shannon’s How I run my Lab is the first community write-up — Herdr, Coolify, a secure-lab layout, and an amdai-bench idea. We do not invent tok/s from those notes. House cards stay on /results. llama.cpp first. Q3 stays a labeled estimate. Nothing below Q3. ## Glossary ### Abliterated A weight edit that weakens a model’s refusal behavior. The file still looks like a GGUF; the change is inside the tensors, not a different architecture. ### Adapter The small trained matrices from LoRA or QLoRA. The base model stays frozen. After training you merge the adapter into the base (PEFT merge_and_unload) before converting to GGUF for llama.cpp. ### AMDai The display wordmark for this notebook (AMDai · R9700 Lab Notes). Domain, URLs, package names, and amdai-bench stay lowercase. Open Graph site_name is AMDai. ### APU A chip that combines CPU and GPU and shares one memory pool. There is no discrete VRAM sticker. The shopper box on AMDai is Ryzen AI Max+ 395 with Radeon 8060S (gfx1151) — Strix Halo in mini-PCs and laptops. ### amdai-bench The first-party measurement schema this site uses. A validating card is tagged amdai-bench/run@1 and records hardware, software pins, the launch command, and tok/s. The product name stays lowercase. ### Backend The program that loads weights and emits tokens — llama.cpp, Strata, or vLLM on this site. A harness (Hermes, a chat UI) sits above a backend; it is not a substitute for one. ### Context length How many tokens of prompt plus conversation the model can keep in the KV cache. House daily driver on one R9700 is 262144 (262K). Tables on this site also show 4K, 32K, 128K, and 256K because speed and RAM change with length. ### Context ceiling The largest llama.cpp -c that still loads and serves on a given box. Native max is the model-card window. The ceiling is what VRAM and RAM actually hold. ### Constrained decoding llama.cpp grammar sampling so every next token still matches a GBNF grammar or a JSON Schema converted to GBNF. Used for tool args, calculator output, and recipe export. ### Container (Docker / Podman) An isolated process that runs an official llama.cpp image instead of a host build. On AMDai that is ggml-org’s GHCR server-vulkan first, server-rocm second. The host still owns /dev/dri and, for HIP, /dev/kfd. ### ComfyUI A node-graph UI for local diffusion checkpoints on HIP/ROCm PyTorch. On AMDai it is the first image-gen path. It does not speak llama-server /v1. ### Chat template The Jinja string llama-server uses to wrap messages, tools, and reasoning. It usually ships in the GGUF metadata. --jinja turns the engine on; --chat-template and --chat-template-file override the file. ### Jinja The template engine llama-server uses to wrap chat messages. Official --jinja / --no-jinja (default enabled). House recipes still pass --jinja. ### Special tokens Control tokens the chat template inserts for roles, stops, and thinking blocks. They live in the GGUF tokenizer, not in the GGUF weights. ### Daily driver The stack AMDai tells a newcomer to run first: Qwen3.6-35B-A3B Unsloth UD-Q5_K_XL on one R9700, llama.cpp Vulkan, 262K context, f16 KV, no RAM offload. ### Device passthrough Giving a container the host GPU nodes it needs. Vulkan llama.cpp uses /dev/dri. ROCm / HIP also needs /dev/kfd. The login (or --group-add) must match render and video. ### Dual-GPU Two R9700s can run two full copies of a model (two clients) or one split / tensor-parallel copy (one larger or shared request). They are different layouts; AMDai does not treat “two GPUs” as free 2× speed. ### Expert offload A llama.cpp MoE trick: keep attention on the GPU and park expert tensors in system RAM. The flags are -ot (tensor override regex) and --n-cpu-moe (how many experts stay on CPU). ### Fast agent The house speed path after the daily driver works: Qwen 3.8 Flash Unsloth UD-Q6_K_XL on Strata v0.1.42, usually with RAM offload, pointed at a harness. ### Qwen 3.8 Flash The large fast-agent GGUF on this site (Unsloth UD-Q6_K_XL listing ~169 GB). It is a different model from the 35B daily driver and usually needs DDR5 expert offload on a pair of 32 GB cards. ### gfx1201 The LLVM / ISA target for RDNA 4 Radeon cards — Radeon AI PRO R9700 and the RX 9070 series on current ROCm tables. rocminfo and llama.cpp --list-devices should show it before you download 20 GB. ### GBNF GGML Backus–Naur Form — the grammar language llama.cpp uses to constrain decode. A JSON Schema is converted to GBNF. Samples live in llama.cpp grammars/. ### GGUF The llama.cpp weight-file format: one (or a few) files that already include a quant, tokenizer, and metadata. It is not a PyTorch checkpoint and not what vLLM wants by default. ### GTT The path where the driver maps more of the same system-memory kit to the iGPU after boot, instead of living only inside a fixed BIOS carve-out. ### HIP AMD’s C++ GPU compute API. llama.cpp can compile a HIP backend when /opt/rocm is clean. HIP is not CUDA and not a drop-in replacement. ### House rules How AMDai picks a stack: llama.cpp first; prefer Q5 over Q4; Q3 listed as estimate or community unless measured; Q1/Q2 stay off; GGUFs on NVMe, not a spinning disk; one LLM per GPU unless you deliberately combine. ### Official instruct The publisher’s chat / instruct GGUF — official Qwen weights plus a Jinja template — not an abliterated merge or an unnamed “uncensored” file. ### JSON schema A JSON Schema object that llama-server can turn into a GBNF grammar. Send it as response_format or -j / --json-schema when the whole reply must be one JSON object. ### imatrix An importance matrix llama.cpp builds from a model and a calibration text, then feeds to llama-quantize to reduce quality loss versus a naive quant. ### KV cache Key/value attention memory that grows with context length. Separate from the weight file. House types: f16 (2 bytes, daily driver on our rig) and q8_0 (1 byte, smaller KV; quality trade not measured here). ### Parallel slots (-np) How many concurrent sequences one llama-server accepts. Official flag is -np / --parallel (default auto). Each slot needs KV. Usable context per client is total ctx / np unless you raise -c or set --kv-unified-per-slot. ### Flash Attention (-fa) A fused attention kernel in llama.cpp. Official flag is -fa / --flash-attn on|off|auto (default auto). Overnight house 35B Vulkan llama-bench recorded -fa 1. ### Prompt cache llama-server keeps processed prompt tokens so a later request that shares a prefix skips that prefill. Official --cache-prompt / --no-cache-prompt (default on; env LLAMA_ARG_CACHE_PROMPT). The completion API field cache_prompt does the same per request (default true). ### Prefix cache Another name for llama.cpp prompt cache: the shared leading tokens of two requests. Official cache_prompt skips that prefix and prefills only the new suffix. ### Cache-reuse (--cache-reuse) Official llama-server min chunk size for reuse via KV shifting. --cache-reuse N (default 0 = disabled; env LLAMA_ARG_CACHE_REUSE). Requires prompt caching to be on. n_cache_reuse on the API matches that flag. ### Slot save/restore Official --slot-save-path PATH enables saving a slot’s KV / prompt cache to disk (default disabled; env LLAMA_ARG_SLOT_SAVE_PATH). POST /slots/{id_slot}?action=save and action=restore take a filename under that directory. ### TTFT Time to first token — how long the user waits after send before the first decode token. Prompt / prefix cache aims to cut that wait when a long system prompt is resent. ### KL-divergence A llama.cpp perplexity metric that compares a quantized model’s token distribution to an f16 / bf16 baseline. Zero means the distributions match. ### llama.cpp The consumer default inference engine on AMDai: a C/C++ GGUF runner from ggml-org. You compile a backend, load a file, and optionally serve an OpenAI-shaped /v1 on llama-server (127.0.0.1:8080). ### LM Studio A desktop chat app that wraps llama.cpp. You download a Vulkan or ROCm llama.cpp runtime, load a GGUF, and optionally serve an OpenAI-compatible /v1 on 127.0.0.1:1234. ### Ollama A pull-and-run daemon that wraps a packaged llama.cpp / ROCm or Vulkan runtime. Local API on 127.0.0.1:11434, including an OpenAI-shaped /v1. ### Lemonade Server AMD’s own local AI server. The primary LLM recipe wraps llama.cpp (Vulkan or ROCm). OpenAI-compatible API on 127.0.0.1:13305/api/v1 — note /api/v1, not /v1. ### Open WebUI A self-hosted browser chat UI. It talks to an OpenAI-compatible /v1. On AMDai that server is llama.cpp llama-server on 127.0.0.1:8080. The UI itself does not run the GGUF. ### Coding assistant An editor or CLI agent that talks to a local OpenAI-compatible /v1. On AMDai that server is llama.cpp llama-server on 127.0.0.1:8080. Aider, Cline, and Continue are the cited first three. ### LoRA Parameter-efficient fine-tuning: freeze the base weights and train small low-rank matrices. Unsloth’s table treats LoRA as 16-bit. Hugging Face PEFT/TRL and AMD’s Radeon playbook document it. ### mmproj The multimodal projector GGUF that llama.cpp uses to encode an image into embeddings the text model can read. You typically need two files: the text GGUF and a matching mmproj. libmtmd is the current llama.cpp path. ### MoE vs dense A mixture-of-experts (MoE) model activates a subset of experts per token; a dense model runs every layer. Qwen3.6-35B-A3B is MoE (the A3B names the active slice). Qwen 3.6 27B and Qwen 3.8 27B on this site are dense. ### NVMe staging Putting the GGUF on a fast NVMe SSD before you run it. A spinning disk (HDD) can feed weights slower than the GPU can prefill, so the card sits idle while the file is paged. ### Offload Some weights or KV live in system RAM instead of VRAM. llama.cpp -ngl 99 means “offload what fits”; leftover layers spill to DDR5. Expert offload is the MoE-specific form (-ot, --n-cpu-moe). ### mmap Official llama.cpp default model load. --load-mode auto memory-maps the GGUF unless a device cannot (env LLAMA_ARG_LOAD_MODE). The kernel pages the file from NVMe instead of copying the whole weight blob into anonymous RAM. ### --no-mmap Deprecated llama.cpp flag that copies the GGUF into process RAM instead of mapping it. Official replacement is --load-mode none (discussion #26469). ### --mlock Deprecated llama.cpp flag that asks the OS to keep model pages in RAM. Official replacements: --load-mode mlock (no mmap + lock, old --no-mmap --mlock) and --load-mode mmap+mlock (map + lock). ### --load-mode Official llama.cpp model-loading switch. Values: auto (default), none, mmap, mlock, mmap+mlock, dio. Env LLAMA_ARG_LOAD_MODE. Replaces --mmap / --no-mmap / --mlock (PR #20834). ### Prefill vs decode Prefill (prompt processing) ingests the prompt in a burst — high tok/s, one shot. Decode (token generation) is the streaming rate people mean by “how fast is it.” AMDai cards report both. ### Batch (-b) Official llama.cpp logical maximum batch size. llama-server README: -b / --batch-size N (default 2048; env LLAMA_ARG_BATCH). How much prompt the engine plans for. The GPU still computes in -ub-sized chunks. ### Micro-batch (-ub) Official llama.cpp physical maximum batch size. llama-server README: -ub / --ubatch-size N (default 512; env LLAMA_ARG_UBATCH). The chunk that actually hits the GPU during prefill. ### Quant A stored-weight encoding that shrinks the GGUF. AMDai lists Q3 through BF16. Q6 / Q5 / Q4 are measured on the local desktop for the 35B file. Q3 rows are options labeled estimate or community unless a run card says measured. ### QLoRA LoRA plus a 4-bit quantized base. Unsloth’s table treats QLoRA as 4-bit. bitsandbytes is the usual loader. AMD’s Radeon playbook: QLoRA is Linux-only because bitsandbytes is not well supported on Windows. ### Qwen3.6-35B-A3B The house daily-driver model: a 35B-class MoE instruct GGUF. The file we measure is Unsloth UD-Q5_K_XL (publisher listing 26.6 GB). Sibling 35B points on the same sweep include Q5_K_M, Q4_K_M, and Q6_K. ### Radeon AI PRO R9700 The AMD workstation / local-AI card this notebook is grounded in: 32 GB, gfx1201, RDNA 4. Official product name — there is no consumer “RX 9700.” The gaming sibling is the RX 9070 series (also gfx1201, 16 GB on the XT). ### RDNA 4 The Radeon architecture family whose LLVM target is gfx1201 on current ROCm hardware tables. Covers the Radeon AI PRO R9700 and RX 9070-class cards — not RDNA 3 (gfx1100, 7900 XTX) and not Instinct CDNA. ### Resizable BAR A BIOS feature that lets the CPU see the GPU’s full VRAM window instead of a small aperture. AMD’s consumer name is Smart Access Memory. Menu labels vary by board. ### ROCm AMD’s open GPU compute platform: compilers, runtimes, and libraries for Linux (and a thinner Windows story). Historically Radeon Open Compute. It is not CUDA, and Canonical’s archive rocm is not the current Core SDK. ### Run card One official amdai-bench/run@1 measurement published on /results: who ran it, which GPU layout, the exact launch command, prefill/decode tok/s, VRAM, and peak system RAM when recorded. ### Speculative decoding A smaller draft GGUF guesses the next tokens; the target accepts or rejects them. llama.cpp documents -md / --model-draft on the server README. AMDai has not measured a draft pair on the local desktop. ### Stack Lab The Ubuntu 24.04+ companion that ships the same catalog as this site (Origin repo / local install — packages not published yet, no public download URL). Profile qwen-reliable is the daily driver; qwen-fast-agent is Flash on Strata. ### Strata A GGUF server AMDai uses for the Flash fast-agent try. The pin on our rig is v0.1.42 (dated 2026-10-10). It is not the daily-driver engine. ### System RAM Memory on the motherboard (DDR5 on the local desktop: 4×64 GB, 256 GB kit). Separate from VRAM. It holds the OS, offloaded tensors, and some KV when the file does not fit the GPU. ### tok/s Tokens per second. Decode tok/s is chat feel; prefill tok/s is prompt ingest. AMDai never fills a missing cell with a guessed rate. ### Thinking mode A chat-template behavior where the model writes a hidden thought block, then a final answer. llama.cpp documents --reasoning on|off|auto, --reasoning-effort, --reasoning-budget, and chat_template_kwargs.enable_thinking. ### Temperature (--temp) llama.cpp sampler that scales logits before the rest of the chain. Official --temp / --temperature default 0.80 (env LLAMA_ARG_TEMPERATURE). The /completion field is temperature (default 0.8). 0 is greedy. ### Top-p (--top-p) Nucleus sampling. Official llama.cpp --top-p default 0.95; 1.0 disables (env LLAMA_ARG_TOP_P). The /completion and OpenAI-shaped field is top_p. ### Min-p (--min-p) Drops tokens below a fraction of the most likely token. Official llama.cpp --min-p default 0.05; 0.0 disables (env LLAMA_ARG_MIN_P). The /completion field is min_p. ### Repeat penalty Official llama.cpp --repeat-penalty penalizes repeating token sequences (CLI default 1.00 = disabled; env LLAMA_ARG_REPEAT_PENALTY). --presence-penalty and --frequency-penalty default 0.00. Qwen cards write repetition_penalty; llama.cpp maps that to repeat_penalty. ### Tool calling An OpenAI-shaped /v1/chat/completions request that includes a tools array. The model returns tool_calls when --jinja is on and the chat template understands tools. ### Unsloth UD Unsloth’s dynamic GGUFs: a named bucket (UD-Q5_K_XL, UD-Q6_K_XL, UD-Q3_K_XL) that keeps some tensors at higher precision. When the publisher lists a file size, the calculator uses that size instead of params × bits. ### UMA A BIOS reserve that hands the iGPU a fixed slice of system memory at boot. The rest of the kit stays with the OS unless GTT / dynamic allocation grows into it. ### vLLM A high-throughput Python serving stack with continuous batching. Default pip install vllm is CUDA. ROCm builds are a separate pin — AMD’s conservative 10.0 combo is still 0.27; community 0.31 is this week’s serve try. ### VRAM vs system RAM VRAM is memory on the GPU; system RAM is DIMMs on the motherboard. A 32 GB R9700 holds weights + KV + overhead. Overflow goes to DDR5 and decode usually drops. ### Out of VRAM llama.cpp could not allocate GPU memory. Vulkan prints ErrorOutOfDeviceMemory or failed to allocate Vulkan0 buffer. HIP prints hipErrorOutOfMemory or failed to allocate ROCm0 buffer. Those strings are quoted from upstream issues. ### Used Radeon A second-hand consumer or workstation Radeon bought for local LLMs. AMDai sizes it by VRAM and gfx first — not by a street price — and vets it with lspci, rocminfo, rocm-smi or amd-smi, and vulkaninfo. ### Vulkan A Khronos graphics/compute API. llama.cpp can use it instead of HIP. On our rig that is Mesa RADV 25.2.8 talking to gfx1201 — the build-vk tree, not a consolation prize. ### WSL2 Windows Subsystem for Linux 2. ROCm 10.1.0 lists it as a technical preview: Ubuntu guest, Windows host Adrenalin, GPU via /dev/dxg. Profiling, debugging, and KFD-dependent tools are out of scope. AMD SMI is discovery and monitoring only. ### local desktop The house test host: Ryzen 9 9950X, 4×64 GB DDR5-5200, two Radeon AI PRO R9700 32 GB, Ubuntu 24.04.5. Official 35B cards credit AMDai and llama.cpp c060ca974 Vulkan, ROCm 7.2.0, Mesa RADV 25.2.8. ## Recipes - [Qwen3.6-35B-A3B on r9700](https://amdai.net/recipes/qwen36-35b-q4km.llamacpp-vulkan.4k/): Q4_K_M, llamacpp, short, 117.69 tok/s. - [Qwen3.6-35B-A3B on r9700](https://amdai.net/recipes/qwen36-35b-q5km.llamacpp-vulkan.4k/): Q5_K_M, llamacpp, short, 109.73 tok/s. - [Qwen3.6-35B-A3B on r9700](https://amdai.net/recipes/qwen36-35b.llamacpp-vulkan.262k/): UD-Q5_K_XL, llamacpp, 256K, 44.4 tok/s. - [Qwen3.6-35B-A3B on r9700](https://amdai.net/recipes/qwen36-35b.llamacpp-vulkan.32k/): UD-Q5_K_XL, llamacpp, 32K, 90.52 tok/s. - [Qwen3.6-35B-A3B on r9700](https://amdai.net/recipes/qwen36-35b.llamacpp-vulkan.4k/): UD-Q5_K_XL, llamacpp, short, 102.39 tok/s. - [Qwen 3.8 Flash on r9700](https://amdai.net/recipes/qwen38-flash.llamacpp.262k/): UD-Q6_K_XL, llamacpp, 262K, 16 tok/s. - [Qwen 3.8 Flash on r9700](https://amdai.net/recipes/qwen38-flash.strata.262k/): UD-Q6_K_XL, strata, 262K, 32.6 tok/s. ## Official run cards - Qwen3.6-35B-A3B UD-Q5_K_XL · single · 4096 context · 102.39 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · single · 32768 context · 90.52 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · single · 131072 context · 66.43 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · single · 262144 context · 44.4 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · split · 4096 context · 82.78 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · split · 32768 context · 74.8 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · split · 131072 context · 57.94 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · split · 262144 context · 42.03 tok/s - Qwen3.6-35B-A3B Q6_K · single · 4096 context · 82.62 tok/s - Qwen3.6-35B-A3B Q5_K_M · single · 4096 context · 109.73 tok/s - Qwen3.6-35B-A3B Q5_K_M · single · 32768 context · 96.21 tok/s - Qwen3.6-35B-A3B Q5_K_M · single · 131072 context · 69.64 tok/s - Qwen3.6-35B-A3B Q5_K_M · single · 262144 context · 40.54 tok/s - Qwen3.6-35B-A3B Q5_K_M · split · 4096 context · 88.19 tok/s - Qwen3.6-35B-A3B Q5_K_M · split · 32768 context · 78.91 tok/s - Qwen3.6-35B-A3B Q5_K_M · split · 131072 context · 59.86 tok/s - Qwen3.6-35B-A3B Q5_K_M · split · 262144 context · 42.33 tok/s - Qwen3.6-35B-A3B Q4_K_M · single · 4096 context · 117.69 tok/s - Qwen3.6-35B-A3B Q4_K_M · single · 32768 context · 101.88 tok/s - Qwen3.6-35B-A3B Q4_K_M · single · 131072 context · 72.48 tok/s - Qwen3.6-35B-A3B Q4_K_M · single · 262144 context · 47.77 tok/s - Qwen3.6-35B-A3B Q4_K_M · split · 4096 context · 93.28 tok/s - Qwen3.6-35B-A3B Q4_K_M · split · 32768 context · 82.57 tok/s - Qwen3.6-35B-A3B Q4_K_M · split · 131072 context · 62.12 tok/s - Qwen3.6-35B-A3B Q4_K_M · split · 262144 context · 42.88 tok/s - Qwen3.6-35B-A3B Q6_K · single · 4096 context · 103.23 tok/s - Qwen3.6-35B-A3B Q6_K · single · 32768 context · 71.19 tok/s - Qwen3.6-35B-A3B Q6_K · single · 131072 context · 54.51 tok/s - Qwen3.6-35B-A3B Q6_K · single · 262144 context · 37.78 tok/s - Qwen3.6-35B-A3B Q6_K · split · 4096 context · 83.5 tok/s - Qwen3.6-35B-A3B Q6_K · split · 32768 context · 75.43 tok/s - Qwen3.6-35B-A3B Q6_K · split · 131072 context · 58.04 tok/s - Qwen3.6-35B-A3B Q6_K · split · 262144 context · 42.2 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · dual-instance · 262144 context · 100 tok/s - Qwen3.6-35B-A3B UD-Q5_K_XL · dual-instance · 262144 context · not a measured decode - Qwen3.6-35B-A3B UD-Q5_K_XL · dual-instance · 262144 context · not a measured decode - Qwen 3.8 Flash UD-Q6_K_XL · single · 262144 context · 32.6 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 262144 context · 16 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 8192 context · 32.8 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 8192 context · 32 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 32768 context · 32.8 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 100000 context · 31.8 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 8192 context · 18.5 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 32768 context · 18.2 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 100000 context · 17.9 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 8192 context · 15.9 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 32768 context · 15.6 tok/s - Qwen 3.8 Flash UD-Q6_K_XL · single · 100000 context · 15.3 tok/s - Qwen3.6-35B-A3B INT4-W4A16 · single · 4096 context · not a measured decode ## Lab log 101 findings · as of 2026-10-28. # AMDai lab log > Measured and cited findings on consumer Radeon (R9700) local AI, plus informative failures. Newest first. 101 findings · as of 2026-10-28. Full HTML with anchors: https://amdai.net/lab-log/ ## CPU cooler for an R9700 build: Ryzen 9000X has none in the box, and a 360 AIO in the front or bottom of the case feeds warm air to the cards 2026-10-28 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-28-cpu-cooler-for-dual-r9700 AMD ships the Ryzen 5 9600X, Ryzen 9 9900X and Ryzen 9 9950X without a cooler; AMD recommends a premium air cooler for the 65 W 9600X and a liquid cooler for the 120 W 9900X and 170 W 9950X for best boost. for a local-AI box with blower R9700s, where the radiator goes matters: the cards pull their air from the case's front and bottom intake, so a 360 radiator should go on top as exhaust, and a case that only takes one in front or bottom is better paired with a large dual-tower air cooler. Case limits checked on maker pages: Fractal Torrent takes a 360 radiator only in front or bottom and air coolers up to 188 mm; Fractal Meshify 2 takes a 360 on top or front and air coolers up to 185 mm; Phanteks Enthoo Pro 2 Server Edition takes radiators on top, front or side and air coolers up to 195 mm; Thermalright Peerless Assassin 120 SE (155 mm) and Noctua NH-D15 G2 (168 mm) fit all three; ARCTIC's Liquid Freezer III Pro 360 needs 63 mm of radiator clearance, and ARCTIC warns that some boards' oversized M.2_1 heatsinks clash with it. with MoE expert offload, token generation is limited by memory and PCIe traffic rather than CPU compute, so AMD's liquid recommendation is about peak boost, not decode speed; we have not measured cooler choice on our own cards. - Stack: Spec review: amd.com product pages for 9600X/9900X/9950X; Fractal, Phanteks, Thermalright, Noctua and ARCTIC product pages; published llama.cpp MoE offload guides - AMD default TDP, 9600X / 9900X / 9950X: 65 / 120 / 170 W - Air cooler height limit, Torrent / Meshify 2 / Enthoo Pro 2 Server: 188 / 185 / 195 mm - Cooler height, Peerless Assassin 120 SE / NH-D15 G2: 155 / 168 mm Sources: [https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-5-9600x.html](https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-5-9600x.html), [https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9900x.html](https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9900x.html), [https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9950x.html](https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9950x.html), [https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/](https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/), [https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/](https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/), [https://phanteks.com/product/enthoo-pro-2-server-edition-tg/](https://phanteks.com/product/enthoo-pro-2-server-edition-tg/), [https://www.thermalright.com/product/peerless-assassin-120-se/](https://www.thermalright.com/product/peerless-assassin-120-se/), [https://noctua.at/en/nh-d15-g2/specification](https://noctua.at/en/nh-d15-g2/specification), [https://www.arctic.de/en/Liquid-Freezer-III-Pro-360/ACFRE00180A](https://www.arctic.de/en/Liquid-Freezer-III-Pro-360/ACFRE00180A), [https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0](https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0) Next: Add a CPU-cooler line to the amdai.net builder parts list. ## Case for two R9700s: a standard 7-slot ATX case is enough on 4-slot-spacing x8/x8 boards, and case fans should follow the GPUs, not the CPU 2026-10-28 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-28-case-for-two-r9700s Every R9700 board-partner card is a 2-slot blower no longer than 271 mm. on the ASRock X870 Taichi Creator and Gigabyte B850 AI TOP, owners report 4-slot spacing between the two CPU x8/x8 slots, so the bottom card ends by expansion slot 7 and leaves a 2-slot gap above it. that means any 7-slot ATX case with at least 271 mm of card length fits two cards; no 8-slot case is needed unless you plan a third card or a riser. blowers push their heat out the back, so the case's job is fresh intake air for the card area. In a published 4x R9700 build, case fans on the stock CPU-following curve let the card area reach its 68 C warning and pause work; full-speed case fans cooled the card area from 67 C to 57 C but changed GPU junction temperature by only 1-4 C, because each card's own blower cools its GPU. tuning the case fans to follow card power and temperature kept the same GPU temperatures at 26% lower fan speed. owners stacking R9700 blowers with no gap report it works, but one owner found the ASUS Turbo's fan is not recessed and may choke without a slot gap, so use matching cards. slot spacing for the ASUS ProArt X870E/B850 boards and the ASRock X870E Taichi was not verified here; check the manual layout or choose an 8+ slot case. checked case specs: Fractal Torrent (7 slots, 2x 180 mm front + 3x 140 mm bottom fans included, PSU up to 230 mm), Fractal Meshify 2 (7 slots, E-ATX, PSU up to 250 mm), Phanteks Enthoo Pro 2 Server Edition (11 slots, PSU up to 240 mm). temperatures are published by others, not measured on our cards. - Stack: Spec review: R9700 board-partner spec sheets, ASRock/Gigabyte board pages and owner reports, Fractal and Phanteks case pages, one published 4x R9700 build guide - Longest R9700 board-partner card: 271 mm - Slot spacing, X870 Taichi Creator / B850 AI TOP (owner reports): 4 slots - Card-area temp, CPU-following vs full-speed case fans (4x R9700, published): 67 vs 57 C - GPU junction change from full-speed case fans (published): 1-4 C Sources: [https://forum.level1techs.com/t/asrock-x870-taichi-creator-dual-gpu/243897](https://forum.level1techs.com/t/asrock-x870-taichi-creator-dual-gpu/243897), [https://pg.asrock.com/general/news.us.asp?id=5643](https://pg.asrock.com/general/news.us.asp?id=5643), [https://www.reddit.com/r/LocalLLaMA/comments/1nqi0di/are_there_any_consumer_mobos_that_can_fit_2x/](https://www.reddit.com/r/LocalLLaMA/comments/1nqi0di/are_there_any_consumer_mobos_that_can_fit_2x/), [https://nicholasreese.com/guides/quad-r9700/quad-r9700-build-guide.md](https://nicholasreese.com/guides/quad-r9700/quad-r9700-build-guide.md), [https://www.reddit.com/r/LocalLLM/comments/1vyaesz/blower_r9700_cards_stacked_with_very_little_gap/](https://www.reddit.com/r/LocalLLM/comments/1vyaesz/blower_r9700_cards_stacked_with_very_little_gap/), [https://forum.level1techs.com/t/radeon-pro-ai-r9700-fan-noise-which-model-to-buy/247552?page=2](https://forum.level1techs.com/t/radeon-pro-ai-r9700-fan-noise-which-model-to-buy/247552?page=2), [https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/](https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/), [https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/](https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/), [https://phanteks.com/product/enthoo-pro-2-server-edition-tg/](https://phanteks.com/product/enthoo-pro-2-server-edition-tg/) Next: Add a case step to the amdai.net builder parts list, and verify slot spacing for the ASUS ProArt boards from their manuals. ## How much NVMe a Radeon local-AI box needs: 2 TB holds every Q4-Q6 quant of three model families, and Gen5 barely speeds up loading 2026-10-28 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-28-nvme-for-local-models Every Q4 to Q6 quant of Qwen3.6-35B-A3B (20.6 to 29.7 GiB), Qwen3.8-Flash-Next (103.7 to 157.6 GiB) and GLM-5.3-Flash (186.0 to 271.8 GiB) adds up to about 1,587 GiB once you keep one extra file of room for the next download and 150 GiB for the OS. That fits a 2 TB drive (1,863 GiB) with 10% left free; a fourth Flash-class family such as MiniMax-M3.1-Flash (UD-Q4 232 GiB) needs 4 TB. On llama.cpp's default memory-mapped loader, loading is usually limited by page faults and one CPU core rather than the drive, so a Gen5 SSD helps less than its doubled rating suggests. A llama.cpp user saw about 2.4 GB/s buffered loads from NVMe rated over 7 GB/s, with one core at 100%. llama.cpp's direct I/O loader reads at disk speed: its PR reports GPT-OSS-120B loading in about 10.5 s versus about 110 s on the first mmap load. One third-party test found a 40.5 GB Q4_K_M file loading in 15.2 s on a Gen4 990 PRO versus 13.8 s on a Gen5 T705 (about 9%). So capacity and a Gen4 TLC drive with DRAM is the better buy for most builds. Sizes are summed GGUF shard bytes from Hugging Face; load figures are published, not measured on our cards. - Stack: Spec and file-size review: Samsung 990 PRO 2/4 TB, WD_BLACK SN850X 4/8 TB, Samsung 9100 PRO 4/8 TB; unsloth GGUF repos - All Q4-Q6 quants of 3 families + headroom: 1587 GiB - Usable space on a 2 TB drive: 1863 GiB - Gen4 vs Gen5 cold load, 40.5 GB Q4_K_M (third-party): 15.2 vs 13.8 s - GPT-OSS-120B load, mmap vs direct I/O (llama.cpp PR): 110 vs 10.5 s Sources: [https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF), [https://github.com/ggml-org/llama.cpp/pull/18012](https://github.com/ggml-org/llama.cpp/pull/18012), [https://github.com/ggml-org/llama.cpp/discussions/12223](https://github.com/ggml-org/llama.cpp/discussions/12223), [https://craftrigs.com/guides/nvme-speed-llm-load-times-benchmarks/](https://craftrigs.com/guides/nvme-speed-llm-load-times-benchmarks/), [https://download.semiconductor.samsung.com/resources/data-sheet/samsung_nvme_ssd_990_pro_datasheet_rev.2.0.pdf](https://download.semiconductor.samsung.com/resources/data-sheet/samsung_nvme_ssd_990_pro_datasheet_rev.2.0.pdf), [https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_9100_PRO_Datasheet_Rev.2.0.pdf](https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_9100_PRO_Datasheet_Rev.2.0.pdf), [https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/product/internal-drives/wd-black-ssd/data-sheet-wd-black-sn850x-nvme-ssd.pdf](https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/product/internal-drives/wd-black-ssd/data-sheet-wd-black-sn850x-nvme-ssd.pdf) Next: Time a cold Flash-Next load on R9700s with the mmap loader versus direct I/O to get our own number. ## SGLang TP2 on 2x R9700 is slower than llama.cpp on one card for Qwen3.6-35B-A3B, and has no DDR5 path for Flash models 2026-10-27 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-27-sglang-tp2-vs-llamacpp-r9700 The most complete public 2x R9700 SGLang rig (v0.5.15 plus RDNA4 patches, TP=2) measured Qwen3.6-35B-A3B AWQ int4 at 67.0 tok/s short and 22.0 tok/s at 197K input. llama.cpp on a single R9700 does 165.2 tok/s (Vulkan, Q4_K_M, tg128) and our own Q5 run does 100.8 short and about 50 at 256K. Upstream SGLang still blocks gfx1201 builds, and its only CPU-expert path (KTransformers) is built around Intel AMX, so Flash-class models stay on llama.cpp or Strata. Numbers come from different hosts and harnesses, so treat ratios as rough: 1.5x to 2.5x in llama.cpp's favour short, about 2.3x at depth, with SGLang using both cards. Where SGLang does stand out is a dense 27B FP8 model with a speculator: 34.3 / 44.5 / 41.6 / 30.1 tok/s at 24 / 7.3K / 58K / 197K input, the only speculative lane shown holding near 200K on R9700 (llama.cpp Vulkan tensor split does 45.0 tg128 on the same model at Q4_K_M). Lessons that carry over to any TP2 engine on R9700: the kernel needs CONFIG_HSA_AMD_P2P and CONFIG_PCI_P2PDMA plus iommu=pt (HSA_FORCE_FINE_GRAIN_PCIE=1 changed nothing); a hard-killed run can leave RCCL shared-memory segments that crash the next launch; and aborting a deep TP2 prefill mid-collective can hang a GPU until reset. AMD's ROCm docs say to disable AITER and use the Triton attention backend on Radeon. - Model: Qwen3.6-35B-A3B; Qwen3.8-27B (dense) - Quant: AWQ int4 / FP8 (SGLang); Q4_K_M and UD-Q5_K_XL (llama.cpp) - Engine: SGLang (patched RDNA4 fork) vs llama.cpp - Engine version: SGLang v0.5.15-v0.5.22; llama.cpp PR #25051 build - GPUs: 1-2x Radeon AI PRO R9700 - Context: short to 197K input - Flags: SGLang TP=2, Triton attention, HIP graphs; llama.cpp Vulkan 1 GPU / layer / tensor - SGLang TP2 35B-A3B AWQ, short: 67 tok/s - SGLang TP2 35B-A3B AWQ, 197K input: 22 tok/s - llama.cpp Vulkan 1 card 35B-A3B Q4_K_M tg128: 165.2 tok/s - llama.cpp 1 card 35B-A3B UD-Q5_K_XL, short: 100.8 tok/s - SGLang TP2 27B FP8 + speculator, 197K input: 30.1 tok/s Sources: [https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference](https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference), [https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference/blob/main/benchmarks/qwen3.6-35b-moe-awq/results.json](https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference/blob/main/benchmarks/qwen3.6-35b-moe-awq/results.json), [https://github.com/ggml-org/llama.cpp/pull/25051#issuecomment-6382846940](https://github.com/ggml-org/llama.cpp/pull/25051#issuecomment-6382846940), [https://github.com/sgl-project/sglang/issues/30599](https://github.com/sgl-project/sglang/issues/30599), [https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/inference/sglang.html](https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/inference/sglang.html), [https://www.lmsys.org/blog/2025-10-22-KTransformers/](https://www.lmsys.org/blog/2025-10-22-KTransformers/) Next: Keep llama.cpp and Strata for the Flash lanes. Check the TP2 kernel and IOMMU prerequisites before the next vLLM tensor-parallel test. ## rocm-smi is gone in ROCm 10.x: use amd-smi --rocm-smi, or the driver's own counters on Vulkan-only setups 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-rocm-smi-gone-in-rocm-10 AMD's ROCm 10.1 transition guide lists ROCm SMI as removed from the Core SDK (replaced by AMD SMI, package amdrocm-amdsmi); it was dropped from TheRock builds in 10.0. On a fresh 10.x install, 'rocm-smi' prints command not found even though ROCm works. 'amd-smi --rocm-smi' prints the familiar concise table, and 'amd-smi monitor --gfx --vram-usage --watch 1' replaces 'watch -n1 rocm-smi'. Our own scan of amdai.net found 23 command lines across 15 pages still calling rocm-smi with no fallback, including the verify tutorial's pass/fail checklist; we are fixing those. Sourced, not a new measurement. Mapping from the amd-smi 10.1 CLI docs: rocm-smi --showproductname --showmeminfo vram becomes amd-smi static --asic --vram; --showuse / --showmemuse become amd-smi metric --usage --mem-usage. --rocm-smi is a top-level flag and cannot be combined with subcommands. Readers on llama.cpp Vulkan have neither tool installed; the amdgpu kernel driver exposes mem_info_vram_total, mem_info_vram_used and gpu_busy_percent under /sys/class/drm/cardN/device/, which is enough to prove a model is on the card. ROCm SMI can still be built from source, and older legacy installs (7.2 and earlier) may still have it. - Stack: ROCm Core SDK 10.1.0 (released 2026-10-05) documentation - Site Scan: 706 sitemap pages, 2026-10-27 - Site pages mentioning rocm-smi: 27 pages - Command lines calling rocm-smi with no amd-smi fallback: 23 lines Sources: [https://rocmdocs.amd.com/en/latest/about/transition-guide-TheRock.html](https://rocmdocs.amd.com/en/latest/about/transition-guide-TheRock.html), [https://rocm.docs.amd.com/projects/rocm_smi_lib/en/docs-10.1.0/index.html](https://rocm.docs.amd.com/projects/rocm_smi_lib/en/docs-10.1.0/index.html), [https://rocm.docs.amd.com/projects/amdsmi/en/docs-10.1.0/how-to/amdsmi-cli-tool.html](https://rocm.docs.amd.com/projects/amdsmi/en/docs-10.1.0/how-to/amdsmi-cli-tool.html), [https://docs.kernel.org/gpu/amdgpu/driver-misc.html](https://docs.kernel.org/gpu/amdgpu/driver-misc.html), [https://docs.kernel.org/gpu/amdgpu/thermal.html](https://docs.kernel.org/gpu/amdgpu/thermal.html) Next: Confirm the replacement commands on our dual-R9700 box, then update the verify, troubleshoot, serve and run tutorials to amd-smi with a sysfs fallback for Vulkan readers. ## Our KV-cache calculator vs llama.cpp's real allocator: GLM-5.3-Flash overstated 2.3x, Qwen3.8-Flash-Next 1.6 GiB short at 262K 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-kv-estimate-audit-llamacpp We checked the KV estimate for all 19 model presets in the amdai.net calculator against the model configs and llama.cpp master's memory code (Oct 27). 13 match within 10%. GLM-5.3-Flash was the big miss: only 11 of its 45 layers keep a growing cache (34 are linear-attention layers with a fixed state), so 262K context needs about 4.95 GiB at f16, not the 11.25 GiB we showed. Qwen3.8-Flash-Next needs about 7.6 GiB at 262K, not 6.0, because its sparse-attention indexer keys also live in the cache. Gemma 4's sliding-window layers were under-counted by 0.4-0.8 GiB. Derived from source, not yet a measured allocation. GLM-5.3-Flash: config.json layer_types lists 34 linear_attention + 11 deepseek_sparse_attention layers, and the GGUF head_count_kv mask agrees. llama.cpp keeps a K-only MLA cache (kv_lora_rank 512, no RoPE part) plus an indexer cache of 3 x 128 elements per token on those 11 layers, and a fixed f32 recurrent state (~146 MiB per sequence) on the 34 linear layers. Qwen3.8-Flash-Next: 12 full-attention layers at 2 KV heads x 256 plus a 2 x 128-element indexer row per token (raw + pooled key). Gemma 4 31B/12B: global layers use 4x512 / 1x512 heads but sliding layers use 16x256 / 8x256, and llama.cpp sizes the sliding cache at window + micro-batch (1,536 cells at the default 512). Dense models (Llama 3.x, Qwen3) and Qwen3.5/3.6 hybrids matched; hybrids also hold a 50-150 MiB fixed state per slot. All figures: f16 cache, one sequence, GiB. - Engine: llama.cpp master 19ff817f (2026-10-27), source review - Cache: f16 K/V, 1 sequence, -ub 512 - Presets within 10% of llama.cpp: 13 of 19 - GLM-5.3-Flash KV at 262K (llama.cpp): 4.95 GiB - GLM-5.3-Flash KV at 262K (old site estimate): 11.25 GiB - Qwen3.8-Flash-Next KV at 262K (llama.cpp): 7.61 GiB - Qwen3.8-Flash-Next KV at 262K (old site estimate): 6 GiB Sources: [https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json](https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json), [https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json), [https://huggingface.co/google/gemma-4-31b/blob/main/config.json](https://huggingface.co/google/gemma-4-31b/blob/main/config.json), [https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-model.cpp](https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-model.cpp), [https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/models/glm5-next.cpp](https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/models/glm5-next.cpp), [https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-kv-cache-iswa.cpp](https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-kv-cache-iswa.cpp), [https://amdai.net/calculator/](https://amdai.net/calculator/) Next: Confirm against llama-server startup logs on the 2x R9700 lanes, then fix the calculator presets (sparse-layer count, indexer cache, Gemma 4 window heads, fixed state per slot). ## R9700 street price is now about $2,000: cheapest new card $1,949.99, 50% over MSRP and $250 up in a day 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-r9700-street-price-noon As of Oct 27 at noon CT, every new in-stock Radeon AI PRO R9700 32 GB we found sold by Micro Center costs $1,949.99 to $1,999.99, and Newegg and B&H list $1,999.99 (ASUS Turbo). That is up $250-300 from yesterday's $1,699.99-$1,799.99. The cheapest is the Gigabyte AI TOP at $1,949.99 (Micro Center, in-store pickup only), 50% above AMD's $1,299 MSRP. Two cards (64 GB VRAM) now cost $3,899.98 before tax. Checked on the retailer pages and on vga.live's card pages around 12:00 CT, before tax. Micro Center: Gigabyte AI TOP $1,949.99 (was $1,999.99, in-store pickup only); ASRock Creator, ASUS Turbo, Sapphire and XFX $1,999.99. Newegg: ASUS Turbo $1,999.99 sold by Newegg, but Newegg's own page showed it out of stock; Newegg marketplace sellers list the Gigabyte AI TOP from $2,079.97. B&H lists the ASUS Turbo at $1,999.99 and the Gigabyte AI TOP at $1,899.95, but vga.live shows the B&H Gigabyte listing as unavailable. Amazon: third-party sellers from $1,943.99 (Sapphire), Gigabyte $2,199.99, ASUS $2,495.00, PowerColor $2,664.51. The lowest price seen since Sep 29 is $1,599.99, so today's cheapest card is $350 above that floor. At $1,949.99 the R9700 costs $60.9 per GB of VRAM. Prices move daily; check before buying. - GPUs: Radeon AI PRO R9700 32 GB (6 board-partner models) - Region: US, new condition - Cheapest new R9700 in stock: 1949.99 USD - Premium over $1,299 MSRP: 50.1 % - Change since Oct 26 snapshot: 250 USD - Newegg (ASUS Turbo, sold by Newegg): 1999.99 USD - Two cards today, before tax: 3899.98 USD - Lowest price since Sep 29: 1599.99 USD Sources: [https://www.microcenter.com/product/699521/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation-graphics-card](https://www.microcenter.com/product/699521/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation-graphics-card), [https://www.newegg.com/asus-turbo-ai-pro-r9700-32g-32gb-graphics-card/p/N82E16814126826](https://www.newegg.com/asus-turbo-ai-pro-r9700-32g-32gb-graphics-card/p/N82E16814126826), [https://www.newegg.com/p/pl?d=radeon+ai+pro+r9700](https://www.newegg.com/p/pl?d=radeon+ai+pro+r9700), [https://www.bhphotovideo.com/c/search?q=radeon%20ai%20pro%20r9700](https://www.bhphotovideo.com/c/search?q=radeon%20ai%20pro%20r9700), [https://www.amazon.com/dp/B0G75W64R3](https://www.amazon.com/dp/B0G75W64R3), [https://vga.live/gpus/radeon-ai-pro-r9700](https://vga.live/gpus/radeon-ai-pro-r9700), [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html) Next: Keep the daily price snapshot; flag when the cheapest new card drops back to $1,599.99. ## llama.cpp removed --no-mmap, --mlock and llama-bench -mmp: old commands now stop at startup, and LLAMA_ARG_NO_MMAP is silently ignored 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-llama-cpp-removed-load-flags Since release b10875 (2026-09-09, PR #28334) llama-server and llama-cli exit with 'invalid argument' on --no-mmap, --mmap, --mlock and --direct-io, and llama-bench rejects -mmp and -dio. Use --load-mode none (old --no-mmap), mmap+mlock (old --mlock), mlock (old --no-mmap --mlock) or dio; in llama-bench, -lm none. The env vars LLAMA_ARG_NO_MMAP, LLAMA_ARG_MMAP and LLAMA_ARG_MLOCK are no longer read at all, so Docker or systemd setups that used them quietly fall back to the default load mode (auto, which memory-maps). Checked against llama.cpp master 27b7a8a009de (2026-10-27) and six dated snapshots of common/arg.cpp back to 2025-04-27. The load flags were folded into --load-mode on 2026-07-23 (#20834) and warned as deprecated until the 2026-09-09 removal. llama.cpp reads environment variables only for options that still exist, so a removed option's env var produces no warning. Other flags that now fail: --draft / --draft-max / --draft-min (removed in the speculative-decoding argument refactor, #22397 and #22964, April-May 2026; use --spec-draft-n-max / --spec-draft-n-min or --spec-ngram-mod-n-max), -cd / --ctx-size-draft, -cpent / --checkpoint-every-n-tokens and --spec-replace (gone between the 2026-04-27 and 2026-07-27 snapshots). -dt / --defrag-thold is still accepted but does nothing. -fa needs a value (on, off or auto); a bare -fa followed by another flag fails. We wrote a small linter for commands, scripts, systemd units and env files and ran it on every llama.cpp command shown on amdai.net (130 distinct command lines across 704 pages): none use a removed or unknown flag. - Engine: llama.cpp - Engine version: master 27b7a8a009de (2026-10-27); removal in b10875 - Method: source audit of common/arg.cpp and tools/llama-bench/llama-bench.cpp - First release without --no-mmap / --mlock / -mmp: b10875 - llama.cpp option entries, 2025-04-27: 237 - llama.cpp option entries, 2026-10-27: 361 - amdai.net command lines with a removed or unknown flag: 0 of 130 Sources: [https://github.com/ggml-org/llama.cpp/pull/28334](https://github.com/ggml-org/llama.cpp/pull/28334), [https://github.com/ggml-org/llama.cpp/commit/14a9d09f75](https://github.com/ggml-org/llama.cpp/commit/14a9d09f75), [https://github.com/ggml-org/llama.cpp/pull/20834](https://github.com/ggml-org/llama.cpp/pull/20834), [https://github.com/ggml-org/llama.cpp/pull/26135](https://github.com/ggml-org/llama.cpp/pull/26135), [https://github.com/ggml-org/llama.cpp/pull/22397](https://github.com/ggml-org/llama.cpp/pull/22397), [https://github.com/ggml-org/llama.cpp/pull/22964](https://github.com/ggml-org/llama.cpp/pull/22964), [https://github.com/ggml-org/llama.cpp/releases/tag/b10875](https://github.com/ggml-org/llama.cpp/releases/tag/b10875), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) Next: Before rebuilding llama.cpp past b10875, check every launcher, systemd unit and env file for the removed flags and env vars; switch to --load-mode / LLAMA_ARG_LOAD_MODE. The site's mmap page and glossary get updated wording (removed, not deprecated). ## vLLM 4-bit on the R9700: recent RDNA int4 speedups skip gfx1201, GPTQ crashes, and the default attention backend can return wrong output 2026-10-27 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-27-vllm-int4-r9700-gating Reading vLLM main (e7fa26d5c8, 2026-10-27): the Oct 13 W4A16 decode tuning (+9-21% GEMM, up to -66% on speculative shapes) is gated to Strix Halo (gfx115x); the new int4 MoE decode kernel (+44% on a 35B-A3B, Strix Halo) and MXFP4 W4A8 GEMV are gfx11-only; upstream has no tuned int4 MoE configs for the R9700. On R9700, GPTQ checkpoints crash at startup unless you pass --linear-backend TRITON, and the default ROCM_ATTN backend gave wrong outputs or GPU faults outside head 128 / KV block 16, while TRITON_ATTN was clean. Int4 MoE on gfx12 runs vLLM's Triton WNA16 path, which reached about 18% of memory bandwidth at batch 1 on Strix Halo. The faster W4A8 MoE kernel (open PR #62480) hard-checks for gfx11, but its core int8 dot instruction is the one llama.cpp already uses on RDNA4, so a gfx12 port looks plausible (untested). An open tuned-config PR for E=256,N=512 on the R9700 cut fused-MoE kernel time 19% at batch 1 and raised end-to-end output 3.9% at 1 request and 17.7% at 4. Int4 checkpoints usually keep lm_head in bf16; open PR #62708 quantizes it at load on gfx11 and gfx12 (1.12-1.18x on Strix Halo). Our estimate (640 GB/s spec bandwidth at 75% efficiency, not measured): about 1.5 ms per token saved on a 35B-A3B on one R9700, about 0.9 ms per card on a Flash-class model at TP2. GGUF support moved out of vLLM core in June into vllm-gguf-plugin; vLLM's docs table still lists it as AMD-supported, the plugin's PyPI wheel links the CUDA runtime (libcudart.so.13), so ROCm needs a source build, and its kernels were copied from llama.cpp b2899. For GGUF on Radeon, llama.cpp remains the path. Also open: on R9700, vLLM main runs 13.5% below v0.29 on gpt-oss-20b because of the MoE combine step (fix PR #64055 restores +17.6%). - Quant: W4A16 / GPTQ / AWQ / MXFP4 / GGUF (4-bit class) - Engine: vLLM (source reading) - Engine version: main e7fa26d5c8 (2026-10-27) - GPUs: Radeon AI PRO R9700 (gfx1201) - Int4 MoE decode kernel, 35B-A3B (Strix Halo, gfx11-only): 44 % faster - R9700 tuned int4 MoE config, kernel time at batch 1 (open PR): -19 % - ROCM_ATTN wrong outputs on R9700 (of 12,389 batches): 635 batches - ROCM_ATTN GPU faults on R9700 (of 12,389 batches): 119 batches - R9700 int4_w4a16 MoE configs in vLLM main: 0 files Sources: [https://github.com/vllm-project/vllm/pull/57199](https://github.com/vllm-project/vllm/pull/57199), [https://github.com/vllm-project/vllm/pull/62480](https://github.com/vllm-project/vllm/pull/62480), [https://github.com/vllm-project/vllm/pull/60413](https://github.com/vllm-project/vllm/pull/60413), [https://github.com/vllm-project/vllm/pull/62708](https://github.com/vllm-project/vllm/pull/62708), [https://github.com/vllm-project/vllm/pull/64045](https://github.com/vllm-project/vllm/pull/64045), [https://github.com/vllm-project/vllm/issues/63597](https://github.com/vllm-project/vllm/issues/63597), [https://github.com/vllm-project/vllm/pull/52637](https://github.com/vllm-project/vllm/pull/52637), [https://github.com/vllm-project/vllm/issues/62697](https://github.com/vllm-project/vllm/issues/62697), [https://github.com/vllm-project/vllm/pull/64055](https://github.com/vllm-project/vllm/pull/64055), [https://github.com/vllm-project/vllm/pull/39612](https://github.com/vllm-project/vllm/pull/39612), [https://github.com/vllm-project/vllm-gguf-plugin/issues/142](https://github.com/vllm-project/vllm-gguf-plugin/issues/142), [https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/common.cuh](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/common.cuh) Next: Measure on two R9700s: attention backend correctness, the gfx115x tuning enabled on gfx12, a tuned int4 MoE file, int4 lm_head, then try the W4A8 MoE kernel on gfx12. ## Monitoring local LLM speed: llama-server's tok/s gauges read 0 mid-generation; use the counters, and compare by context depth 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-llama-server-metrics-gauges In current llama.cpp, the /metrics gauges predicted_tokens_seconds and prompt_tokens_seconds are reset by every scrape and only updated when a request finishes, so dashboards show 0 while a long generation runs and two scrapers steal each other's windows. Rates from the cumulative counters are reliable. Strata's /metrics already reports a live 2-second tok/s plus per-request speeds. Read at llama.cpp master 0c8c59ef3d (2026-10-27): each /metrics request sets metrics_reset_bucket, and generation stats reach the bucket only in the slot reset callback at request end. The 0-reading bug was filed as #27436 and auto-closed as stale; fix PRs #27760 and #30543 are still open. Use tokens_predicted_total / tokens_predicted_seconds_total and prompt_tokens_total / prompt_seconds_total deltas instead; prompt_tokens_cached_total (cache reuse) exists but is missing from the README table. The gauge and per-request timings count n_gen-1 decode steps while the counter counts n_gen, so counter-derived speed reads about 5% high on 20-token tool-call replies; speculative/MTP tokens count in both. Separately, average tok/s falls as conversations get longer, so a speed alert should compare like depths: we bin requests by context depth (0-8K, 8-32K, 32-64K, 64-128K, 128K+) using the 'stop processing: n_tokens' and 'eval time' log lines and flag a bin only when its median drops more than 5% and more than 3 robust standard errors. Strata's JSON /metrics keeps the last 500 requests with prompt size, decode tok/s and the share of experts read over PCIe, which feeds the same check. - Engine: llama.cpp llama-server / Strata (source read) - Engine version: llama.cpp 0c8c59ef3d; Strata main 2026-10-25 - GPUs: 2x Radeon AI PRO R9700 (target) - predicted_tokens_seconds while a request is generating: 0 tok/s - Counter-derived speed bias on a 20-token reply: 5 % - Strata live rate window: 2 s Sources: [https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-common.h](https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-common.h), [https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-context.cpp](https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-context.cpp), [https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-task.cpp](https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-task.cpp), [https://github.com/ggml-org/llama.cpp/issues/27436](https://github.com/ggml-org/llama.cpp/issues/27436), [https://github.com/ggml-org/llama.cpp/pull/27760](https://github.com/ggml-org/llama.cpp/pull/27760), [https://github.com/ggml-org/llama.cpp/pull/30543](https://github.com/ggml-org/llama.cpp/pull/30543), [https://github.com/Niko1221/Strata/blob/main/serve/server.py](https://github.com/Niko1221/Strata/blob/main/serve/server.py), [https://docs.kernel.org/gpu/amdgpu/thermal.html](https://docs.kernel.org/gpu/amdgpu/thermal.html) Next: Build 7-day per-depth speed baselines for the always-on dual-R9700 lanes and check them after every driver or engine update. ## VRAM per dollar: a new R9700 is now at parity with a used RTX 3090, and wins on watts and slots 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-vram-per-dollar With the cheapest new R9700 at $1,949.99 (up $250 since yesterday) it costs $60.9 per GB of VRAM, against $58.9-63.7 per GB for a used RTX 3090 at eBay sold medians of $1,413-$1,528. Per watt it wins clearly (9.4 W/GB vs 14.6), and 64 GB takes two R9700s at 600 W instead of three 3090s at 1,050 W. For Flash-class MoE models that don't fit in VRAM, gigabytes per card decide how many experts stay on the GPU, so $/GB, W/GB and card count matter more than per-card bandwidth. At yesterday's $1,699.99 the R9700 was 10-17% cheaper per GB than a used 3090; at its $1,599.99 low since Sep 29, 15-21% cheaper. Break-even against a $1,528 used 3090 is about $2,037. The 3090 still has about 1.5x the bandwidth per card (936 vs 640 GB/s, vendor spec), which matters when the whole model fits in VRAM. Builds (GPUs only): 64 GB = 2x R9700 $3,900, 600 W vs 3x 3090 (72 GB) $4,584, 1,050 W; 128 GB = 4x R9700 $7,800, 1,200 W vs 6x 3090 (144 GB) $9,168, 2,100 W. Cheapest per GB among used cards: RX 7900 XTX ($1,004, $41.8/GB) and Quadro RTX 8000 48 GB ($2,080, $43.3/GB, 5.4 W/GB). Used RTX 4090 ($3,037) and new RTX 5090 ($5,999.99) cost $126-188/GB. Used prices carry no warranty; one tracker (getpcparts) shows a much lower $1,093 3090 average and was treated as an outlier. Prices are before tax, US. - R9700 cheapest new in stock: 1949.99 USD - Used RTX 3090 eBay sold median (gpusold, 334 sales): 1528 USD - R9700 cost per GB VRAM: 60.9 USD/GB - Used RTX 3090 cost per GB VRAM: 63.7 USD/GB - R9700 board power per GB: 9.4 W/GB - RTX 3090 board power per GB: 14.6 W/GB - R9700 price for per-GB parity with a $1,528 3090: 2037 USD Sources: [https://gpusold.com/](https://gpusold.com/), [https://compsniper.com/worth/rtx-3090](https://compsniper.com/worth/rtx-3090), [https://www.pcprice.watch/search/RTX3090/US/GPU](https://www.pcprice.watch/search/RTX3090/US/GPU), [https://vga.live/gpus/radeon-ai-pro-r9700](https://vga.live/gpus/radeon-ai-pro-r9700), [https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation](https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation), [https://rtx.live/gpus/geforce-rtx-5090](https://rtx.live/gpus/geforce-rtx-5090), [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html) Next: Add a VRAM-per-dollar table to the buying pages, refreshed with the price data; for a two-card purchase, wait for the R9700 to return to about $1,600. ## R9700 upstream bug check: 15 RDNA4 fixes landed in 30 days, but the ROCm llama.cpp hang that needs a reboot still fails on the newest driver stack 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-gfx1201-upstream-bugs We searched 10 GitHub repos Radeon local-AI users rely on for issues naming gfx1201 or R9700. 15 RDNA4-titled issues were closed as fixed in the last 30 days, including two Tensile GEMM page faults and a ~12% Vulkan prefill regression in llama.cpp. Still open: a ROCm-only llama.cpp generation hang that resets the GPU (Vulkan passes), a cross-stream wait penalty whose size depends on MES firmware, a GFXOFF-related idle power cost, a vLLM TP=2 deadlock, and a vLLM 0.28 FP8 kernel choice that slows RDNA4 decode. TheRock #9279: ROCm llama.cpp hang leading to MES REMOVE_QUEUE failure, GPU reset and PSP error; on 2026-10-22 a reporter showed it still fails on a TheRock 10.2 nightly with amdgpu-dkms and firmware 31.60, including a pure dual-R9700 layer-split run, while Vulkan passes. TheRock #9297: tiny kernels slow down when two or more other streams hold pending waits; reported penalty by MES firmware is +7 to +39 us at 0x89, +5 to +17 us at 0x91 and about +9 us at 0x93, and GPU_MAX_HW_QUEUES=2 or 1 removes it. TheRock #9245: an AMD contributor traced the extra idle draw to the kernel disabling GFXOFF on GC 12 once a compute queue exists (Linux commit 90505894c4ed); the reporter measured about 5-6 W more per card than Vulkan. vLLM #57838: from v0.28.0 a RowWise FP8 kernel is selected on RDNA4 and the report measures 5-24% slower decode than v0.27.1's choice. llama.cpp #28093: with an AMD iGPU enabled next to an R9700, Vulkan llama-server reserved most host RAM at idle; it was closed as stale, not fixed. Housekeeping: the ROCm/ROCm repo now redirects to ROCm/legacy-rocm-build and GitHub issue search on the old name returns an error, so new ROCm reports go to ROCm/TheRock. - RDNA4-titled issues closed as fixed, last 30 days: 15 issues - Open issues mentioning gfx1201/R9700 in ROCm/TheRock: 104 issues - Open in llama.cpp: 35 issues - Open in vLLM: 40 issues - Cross-stream wait penalty at MES 0x93 (reported): 9 us per small kernel Sources: [https://github.com/ROCm/TheRock/issues/9279](https://github.com/ROCm/TheRock/issues/9279), [https://github.com/ROCm/TheRock/issues/9297](https://github.com/ROCm/TheRock/issues/9297), [https://github.com/ROCm/TheRock/issues/9245](https://github.com/ROCm/TheRock/issues/9245), [https://github.com/ROCm/TheRock/issues/9228](https://github.com/ROCm/TheRock/issues/9228), [https://github.com/vllm-project/vllm/issues/40980](https://github.com/vllm-project/vllm/issues/40980), [https://github.com/vllm-project/vllm/issues/57838](https://github.com/vllm-project/vllm/issues/57838), [https://github.com/ggml-org/llama.cpp/issues/28093](https://github.com/ggml-org/llama.cpp/issues/28093), [https://github.com/ggml-org/llama.cpp/issues/29892](https://github.com/ggml-org/llama.cpp/issues/29892), [https://github.com/ROCm/rocm-libraries/issues/12346](https://github.com/ROCm/rocm-libraries/issues/12346), [https://github.com/ROCm/rocm-libraries/issues/7992](https://github.com/ROCm/rocm-libraries/issues/7992) Next: Check our own cards' MES firmware, launcher environment and iGPU state against this list, rerun the search weekly, and add an open-upstream-bugs panel to /rocm-problems. ## Read the LICENSE, not the tag: 1 in 4 new open-weight models had no usable license label, and Qwen3.8-Flash-Next's terms are stricter than its 2.4T sibling's 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-new-model-licenses Of 88 open-weight models released in the last 90 days, 20 were tagged only 'other' or nothing on Hugging Face. After reading each LICENSE: 62 open, 7 open with conditions, 5 commercial only under a revenue cap, 9 non-commercial or research only, 1 not for EU use, 4 still unclear. Personal and home use is fine for every Flash-class model we run. Qwen Community License 1.0 (Qwen3.8-Flash-Next): free for personal and internal use, but any company running a Model-as-a-Service or coding/office AI-assistant business needs a separate Qwen license for commercial use, with no revenue floor; the Qwen3.8-Max license (2.4T-A95B) only requires that above US$50M revenue in 12 months. GLM-5.3: MIT-style, only Model-as-a-Service firms above US$10B revenue need a Z.AI security review. MiniMax-M3.1-Flash: the base grant is non-commercial; commercial use needs a 'Built with MiniMax M3.1-Flash' notice and a one-time email, and written approval above US$20M/year. LiquidAI's LFM Open License 1.0 (5 models): commercial use only for entities under US$10M annual revenue. Tencent ContextPilot 8B/14B/E4B ship Apache 2.0 text with an added clause limiting use to scientific research. Youtu-Parsing-Omni is 'not intended for use within the European Union'. Also, 36 of 61 repos tagged apache-2.0, mit or llama3 had no LICENSE file at the repo root; none of the 25 that did added extra terms. Plain-language reading, not legal advice. - Models with license 'other' or none: 20 of 88 - Still unclear after reading LICENSE files: 4 of 88 - Non-commercial or research only: 9 of 88 - Commercial only under a revenue cap: 5 of 88 - Permissive-tagged repos with no LICENSE file: 36 of 61 Sources: [https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE), [https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE](https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE), [https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE](https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE), [https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/LICENSE](https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/LICENSE), [https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/main/LICENSE](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/main/LICENSE), [https://huggingface.co/tencent/ContextPilot-8B/blob/main/LICENSE](https://huggingface.co/tencent/ContextPilot-8B/blob/main/LICENSE), [https://huggingface.co/tencent/Youtu-Parsing-Omni/blob/main/LICENSE](https://huggingface.co/tencent/Youtu-Parsing-Omni/blob/main/LICENSE) Next: Add a filterable 'Use' column (open, conditions, revenue cap, non-commercial) with a LICENSE link to /new-models. ## Coding CLIs on a local llama-server: 0.7K to 19K tokens before your first word, and one background call that forces a full re-read 2026-10-27 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-27-coding-cli-prompt-tax We captured what five coding CLIs send to a local endpoint. First-turn prompts: Claude Code 19,384 tokens, Qwen Code 15,091 (plus a 7,084-token memory call every turn), Codex CLI 8,343, OpenCode 7,158, Aider 749. Replayed on a real one-slot llama-server, Qwen Code's memory call evicted the session when the RAM prompt cache was off: turn 2 re-read 15,602 tokens instead of 21. Method: each CLI pointed at a recording mock of /v1/chat/completions, /v1/messages and /v1/responses in a 3-file git repo, prompt 'Say done.'; tokens counted with the Qwen3 tokenizer over system text, tool schemas and messages (chat-template tokens excluded). Claude Code's 19,384 tokens are mostly tool schemas (16,465 for 24 tools). Two fresh runs of every CLI produced identical prompts; Claude Code 2.1.197's attribution block was stable, unlike older builds that changed it per request. The captured request bodies were then replayed in order on llama-server b11912 (CPU, Qwen3.5-0.8B Q8_0, a hybrid model, -np 1). Qwen Code turn 2: 21 tokens re-read (0.29 s) with the default --cache-ram, 15,602 tokens (31.0 s on this CPU) with --cache-ram 0, and 21 tokens with --cache-ram 0 when its setting memory.enableManagedAutoMemory is false. OpenCode's title call runs only on the first turn, before the main request, so turn 2 re-read 21 tokens either way. Prefill seconds on Radeon will differ; the token counts are what carry over. Also note llama-server's default port changed from 8080 to 9931 (merged 2026-10-09), so guides that assume 8080 need --port. - Model: Qwen3.5-0.8B - Quant: Q8_0 - Engine: llama-server - Engine version: b11912 (CPU build) - Context: 65536 - Flags: -np 1, --reasoning off; arms: default --cache-ram, --cache-ram 0 - Clients: Claude Code 2.1.197, Qwen Code 0.15.10, Codex CLI 0.164.0, OpenCode 1.19.4, Aider 0.86.2 - Claude Code first-turn prompt: 19384 tokens - Qwen Code first-turn prompt: 15091 tokens - Codex CLI first-turn prompt: 8343 tokens - OpenCode first-turn prompt: 7158 tokens - Aider first-turn prompt: 749 tokens - Qwen Code turn 2 re-read, --cache-ram 0: 15602 tokens - Qwen Code turn 2 re-read, default cache or memory off: 21 tokens Sources: [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [https://github.com/ggml-org/llama.cpp/pull/30159](https://github.com/ggml-org/llama.cpp/pull/30159), [https://github.com/ggml-org/llama.cpp/issues/20003](https://github.com/ggml-org/llama.cpp/issues/20003), [https://code.claude.com/docs/en/llm-gateway-protocol](https://code.claude.com/docs/en/llm-gateway-protocol), [https://opencode.ai/docs/providers/](https://opencode.ai/docs/providers/) Next: Repeat the turn-by-turn count on the dual-R9700 Flash and single-R9700 35B lanes, and publish ready-to-use configs for each CLI. ## llama.cpp on 2x R9700: build with RCCL off for batch-1 tensor split, and three build flags that no longer do anything 2026-10-26 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-26-hip-build-flags-rccl Since llama.cpp PR #27825 (merged 2026-09-15), a HIP build with GGML_HIP_RCCL=OFF (the default) uses a built-in two-GPU AllReduce for --split-mode tensor. On 2x R9700 it beat the old fallback by +17.5% prefill and +8.3% decode, and in published RCCL-on vs off tests decode was equal or slightly faster without RCCL while RCCL only helped long prefills. Cited results: 2x R9700 (PCIe 4.0 x8/x8, ROCm 7.14), gemma4 31B Q6_K, -sm tensor, pp2048/tg512 at depth 4096: 959.9/30.0 t/s before vs 1127.5/32.5 t/s after #27825; that tester also reported frequent GPU hangs with RCCL enabled. 2x R9700, Qwen3.8-27B UD-Q4_K_XL server: RCCL on vs off gave +3.6% to +10.1% prefill at 60K-120K and identical decode at 1-3 parallel slots. 2x 7900 XTX, Qwen3.8-27B Q8_0: RCCL ~9% faster prefill, built-in AllReduce ~2.5-2.8% faster decode, also at 100K depth. Layer split (-sm layer) has no AllReduce and is unaffected. Dead or changed flags on master (2026-10-26): GGML_HIP_ROCWMMA_FATTN was removed 2026-07-24 and is silently ignored; -ffast-math and -funsafe-math were dropped from HIP builds by mid-August 2026 (unsafe math broke MTP output identity), so builds from July differ numerically; GGML_CUDA_FA_ALL_QUANTS became GGML_CUDA_FA_QUANTS, whose default covers only f16, bf16, q8_0 and q4_0 matched pairs. A mixed KV cache such as -ctk q8_0 -ctv f16 then converts K and V to f16 on every FlashAttention call, with a single log warning. Head-dim-256 models still lack the MMA FlashAttention path on RDNA4 (PR #26419 open; +21.8% prefill at 127K depth on one R9700 in its tests). - Engine: llama.cpp (HIP/ROCm) - Engine version: master 7aca0d2 (2026-10-26) source read; results from cited testers - GPUs: 2x Radeon AI PRO R9700 (and 2x RX 7900 XTX for the RCCL decode comparison) - Flags: -sm tensor, -fa on, GGML_HIP_RCCL OFF vs ON - Prefill gain, built-in AllReduce vs old fallback (2x R9700, 31B Q6_K): 17.5 % - Decode gain, built-in AllReduce vs old fallback (2x R9700, 31B Q6_K): 8.3 % - Decode, RCCL on vs off (2x R9700, 27B Q4, 60K-120K): 0 % - Max prefill gain, RCCL on vs off (2x R9700, 27B Q4, 60K-120K): 10.1 % Sources: [https://github.com/ggml-org/llama.cpp/pull/27825](https://github.com/ggml-org/llama.cpp/pull/27825), [https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5535620413](https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5535620413), [https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5699240655](https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5699240655), [https://github.com/ggml-org/llama.cpp/pull/26046](https://github.com/ggml-org/llama.cpp/pull/26046), [https://github.com/ggml-org/llama.cpp/pull/26696](https://github.com/ggml-org/llama.cpp/pull/26696), [https://github.com/ggml-org/llama.cpp/pull/28079](https://github.com/ggml-org/llama.cpp/pull/28079), [https://github.com/ggml-org/llama.cpp/pull/26419](https://github.com/ggml-org/llama.cpp/pull/26419) Next: Audit the build cache of each local lane, and A/B RCCL off vs on only if a tensor-split lane is used. ## Hot DDR5 can quietly cost hybrid GPU+RAM decode up to ~10%, and non-ECC sticks never report errors 2026-10-26 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-26-ddr5-dimm-heat-hybrid-decode When a big MoE model keeps its experts in DDR5 next to two R9700s, the DIMMs work hard for hours. DDR5 doubles its refresh rate above 85 C (JEDEC refresh interval 3.9 us drops to 1.95 us). For 64 GB sticks built from 32 Gb chips, that takes the time a rank spends refreshing from about 10.5% to about 21%, so bandwidth-bound work can slow by up to ~13%. For hybrid decode we estimate up to ~7-11% fewer tokens per second once sticks pass 85 C, depending on how much of each token's time is spent reading DDR5 (an upper-bound estimate). Linux can read each stick's temperature with the spd5118 driver, so this is easy to check. Reading temps: the spd5118 hwmon driver (Linux 6.11 and later) exposes temp1_input, temp1_crit and alarm files per DIMM; on AMD boards the SMBus driver registers the SPD hubs automatically, and an AM5 example shows sensors named spd5118-i2c-1-51 and -53 with a default 'high' of 55 C (not a problem by itself) and 'crit' of 85 C. Field reports: one owner measured DDR5-6400 RDIMMs at 95 C and above, cooled them under 60 C with shrouds and fans, and saw about 10% faster inference; a dual-3090 hybrid build reported decode falling from 17 to 8-12 tok/s after minutes from DIMM heat. Stability: consumer AM5 sticks are non-ECC, and DDR5's on-die ECC fixes bits inside the chip without telling the OS, so EDAC/rasdaemon show nothing; a marginal four-stick speed appears only as rare crashes, kernel oopses, machine-check events or wrong output. A published 4x64 GB AM5 setup was 12 h y-cruncher-stable at DDR5-4800 (36-40-40-78, about 56 GB/s), and the same settings failed after about 7 h on an older AGESA. On Linux, y-cruncher's VT3, FFTv4 and N63 components plus stressapptest are the practical tests; TestMem5 and Karhu are Windows tools. - DDR5 refresh interval below 85 C: 3.9 us - DDR5 refresh interval at 85-95 C: 1.95 us - Time refreshing, 32 Gb chips, below 85 C (calc): 10.5 % - Time refreshing, 32 Gb chips, 85-95 C (calc): 21 % - Hybrid decode loss above 85 C, upper-bound estimate: 7-11 % - Inference gain after cooling RDIMMs from 95 C+ (field report): 10 % Sources: [https://docs.kernel.org/hwmon/spd5118.html](https://docs.kernel.org/hwmon/spd5118.html), [https://cateee.net/lkddb/web-lkddb/SENSORS_SPD5118.html](https://cateee.net/lkddb/web-lkddb/SENSORS_SPD5118.html), [https://lkml.indiana.edu/2405.3/07725.html](https://lkml.indiana.edu/2405.3/07725.html), [https://www.phoronix.com/forums/forum/hardware/motherboards-chipsets/1512681-linux-6-14-to-add-sensor-monitoring-for-a-~-180-asrock-am5-motherboard/page2](https://www.phoronix.com/forums/forum/hardware/motherboards-chipsets/1512681-linux-6-14-to-add-sensor-monitoring-for-a-~-180-asrock-am5-motherboard/page2), [https://hivebook.wiki/wiki/dram-row-buffers-and-bank-conflicts-activate-rate-tfaw-and-refresh](https://hivebook.wiki/wiki/dram-row-buffers-and-bank-conflicts-activate-rate-tfaw-and-refresh), [https://semiengineering.com/dram-thermal-issues-reach-crisis-point/](https://semiengineering.com/dram-thermal-issues-reach-crisis-point/), [https://www.reddit.com/r/LocalLLaMA/comments/1p89j2t/today_i_learned_that_ddr5_can_throttle_itself_at/](https://www.reddit.com/r/LocalLLaMA/comments/1p89j2t/today_i_learned_that_ddr5_can_throttle_itself_at/), [https://forum.level1techs.com/t/ddr5-4-dimms-on-am5-whats-working-whats-not/197153?page=14](https://forum.level1techs.com/t/ddr5-4-dimms-on-am5-whats-working-whats-not/197153?page=14), [https://pakt.cc/en/ddr5/testing/](https://pakt.cc/en/ddr5/testing/) Next: Log DIMM temperatures next to tok/s during a 30-minute hybrid decode on our dual-R9700 desktop with four 64 GB sticks, and test whether better airflow changes the speed. ## Power budget for four R9700s in two desktops: one 15 A circuit is not enough, even at the lowest power cap 2026-10-26 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-26-four-r9700-two-desktops-power-budget Before buying a second desktop for two more R9700s, we built a power budget from published measurements. Under full compute an R9700 has been measured at about 390-400 W at the card (spec TBP 300 W), while decode-bound llama.cpp is much lower: about 340-353 W for two cards together in layer split. With all four cards at 300 W, the two desktops draw an estimated ~1,780 W at the wall; even at the lowest allowed cap (210 W) it is ~1,640 W. A 15 A, 120 V circuit may carry only 1,440 W continuously (the 80% rule for loads over 3 hours), so the two boxes need separate circuits, or one 20 A circuit with cards capped at 260 W or less. Card numbers: Igor's Lab (compute/AI full load ~390-400 W), PCGH (only ~1.5 W from the PCIe slot, so nearly all power rides the single 12V-2x6 connector), Puget Systems (two cards, sysfs power: llama.cpp Q4_K_M layer split 339 W at 1 request, 353 W at 8; vLLM TP2 FP8 310-315 W at 1 request, up to 510 W at 4-8). CPU: Ryzen 9 9950X measured at its 200 W PPT, 210-235 W with PBO. Power cap results from a published llama.cpp sweep on one R9700: 260 W cost 0.0% decode and 4.0% prefill; 210 W cost 1.9% decode and 11.2% prefill. Rest of each system (board, four DDR5 sticks, NVMe, fans, NIC) is our estimate of 60-100 W, and wall power assumes an 80 PLUS Platinum supply at 92%. Per box: two cards at 400 W plus a PBO CPU is ~1,135 W DC, which is 76% of a 1,500 W supply, 95% of 1,200 W and over 100% of 1,000 W. A UPS should be sized on input watts times 1.25 and be pure sine wave for active-PFC supplies; one desktop in hybrid Flash decode needs at least ~925 W / ~1,030 VA by that rule. - 4 cards at 300 W TBP, both desktops, wall (estimate): 1782 W - 4 cards capped at 210 W, worst case, wall (estimate): 1642 W - 15 A / 120 V circuit, continuous limit: 1440 W - R9700 full compute load, measured at card (Igor's Lab): 400 W - Two R9700s, llama.cpp layer split, 1 request (Puget): 339 W - Decode loss at 260 W cap (published sweep): 0 % Sources: [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html), [https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/](https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/), [https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/5/](https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/5/), [https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/](https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/), [https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/](https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/), [https://www.tomshardware.com/pc-components/cpus/amd-ryzen-9-9950x-cpu-review/4](https://www.tomshardware.com/pc-components/cpus/amd-ryzen-9-9950x-cpu-review/4), [https://www.infobytes.de/pages/atx3.1powerrev2.1.pdf](https://www.infobytes.de/pages/atx3.1powerrev2.1.pdf), [https://electricaltoolbox.com/continuous-load-calculator/](https://electricaltoolbox.com/continuous-load-calculator/), [https://www.se.com/us/en/faqs/FA158939/](https://www.se.com/us/en/faqs/FA158939/) Next: Measure real CPU, GPU and rest-of-system draw per workload on our dual-R9700 desktop to replace the estimates, and pick the second desktop's supply (1,200-1,300 W with two native 12V-2x6 cables, or 1,000 W only with cards capped at 260 W). ## llama-bench A/A test: identical runs differed by up to 20%, and one run's mean +- stddev called identical runs 'different' 9 times in 15 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-llama-bench-noise-aa-test Before trusting a 3-5% win between two configs, we ran the same llama-bench command 6 times in a row. Inside one run, repetitions varied about 5%, but the medians of whole runs varied 6-7%, and the first run after loading was 12-16% slower than the rest. Comparing any two of the identical runs, a 95% interval built from one run's repetitions wrongly flagged a difference in 9 of 15 pairs. llama-bench's own tools don't guard against this: the generation warmup is a single token, repetitions run back-to-back by default, and compare-llama-bench.py prints a plain ratio of averages with no spread. Measured on a shared 8-vCPU cloud VM with a CPU build, not a Radeon card, so the size of the noise here is a worst case; the pattern (repetitions within one invocation share warm caches and clock state, so their spread understates run-to-run noise) is general. Source facts are from llama.cpp master 7aca0d2: llama-bench defaults to -r 5 and --delay 0; its warmup runs one full prompt for pp tests but test_gen with 1 token for tg tests; compare-llama-bench.py compares AVG(avg_ts) only. What we do now: an A/A run of every config before an A/B sweep to measure its noise floor, at least 3 invocations per arm interleaved A B A B, one throwaway invocation after each model load, and a hierarchical bootstrap of the median ratio (resample invocations, then repetitions) with a minimum effect size, instead of mean +- stddev from one run. With only 3 vs 3 identical runs, one split still showed +9.9% (interval 1.003-1.187) because the slow first run landed on one side, which is why the throwaway run matters. - Within-run CV, tg64 / pp512: 5.2 / 5.1 % - Between-run CV of medians, tg64 / pp512: 6.1 / 7.3 % - Largest gap between identical runs, tg64 / pp512: 20.4 / 19.2 % - First run vs later runs, pp512: 357 vs 412-426 tok/s - Identical pairs flagged 'different' by a one-run 95% CI: 9 of 15 pairs Sources: [https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md), [https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp), [https://github.com/ggml-org/llama.cpp/blob/master/scripts/compare-llama-bench.py](https://github.com/ggml-org/llama.cpp/blob/master/scripts/compare-llama-bench.py) Next: Run the same A/A test on dual R9700s for the Flash lane and on one R9700 for the 35B lane to get the real Radeon noise floor, then use it as the minimum effect size for the Q4/Q5/Q6 dual-GPU sweep. ## Open WebUI in front of llama-server: with --cache-ram 0, its background calls force a full re-prefill of the chat on every turn 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-openwebui-background-tasks-prompt-cache Open WebUI 0.12.4 sends extra model calls per chat turn (follow-up suggestions every turn, plus title and tags on the first), each a fresh one-message prompt that shares no prefix with the conversation. On a one-slot llama-server they replace the slot's state. With the default host prompt cache (--cache-ram 8192 MiB) the conversation is saved and restored and each later turn re-processes only about 70 tokens; with --cache-ram 0, or a cache smaller than one conversation's saved state, every turn re-processed the whole 11K-token chat (about 40x the prefill time in this test). Method: Open WebUI driven through its API against a logging mock backend to capture exactly what it sends, then those task prompts replayed against a real llama-server (llama.cpp b11912 CPU build, Qwen3.5-0.8B Q8_0, a hybrid Gated DeltaNet model like the Flash-class models) for a 5-turn, ~11.1K-token chat. Measured on an 8-vCPU cloud Xeon, not a Radeon card; the re-processed token counts do not depend on hardware, the seconds do. Captured calls: task prompts start with '### Task:' and carry a truncated history (follow-ups: first user message plus last 4, about 1,000 characters each); title max_tokens is 1000, follow-up and tags have no limit, and by default all run on the chat's own model. llama.cpp's cache entry for the conversation was 150.6 MiB of state, 208.4 MiB with its 3 context checkpoints, and saving plus restoring took 209 ms; each task prompt adds its own 27-30 MiB entry to the same cache. Auto slots (4 slots, unified KV, the llama-server default when -np is not set) and -np 2 also kept reuse at about 70 tokens; -np 2 without unified KV halves each slot's context. - Tokens re-processed per later turn, default --cache-ram: 68-70 tokens - Tokens re-processed per later turn, --cache-ram 0: 11,201-11,396 tokens - Prefill per later turn, default --cache-ram (CPU test box): 0.52-0.56 s - Prefill per later turn, --cache-ram 0 (CPU test box): 19.8-22.9 s - Prompt cache entry for the 11.2K-token chat incl. checkpoints: 208.4 MiB - Extra model calls per later turn (Open WebUI defaults): 1 calls Sources: [https://github.com/open-webui/docs/blob/main/docs/features/administration/task-models.mdx](https://github.com/open-webui/docs/blob/main/docs/features/administration/task-models.mdx), [https://github.com/open-webui/open-webui/issues/638](https://github.com/open-webui/open-webui/issues/638), [https://github.com/ggml-org/llama.cpp/pull/16391](https://github.com/ggml-org/llama.cpp/pull/16391), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp), [https://github.com/ggml-org/llama.cpp/releases/tag/b11912](https://github.com/ggml-org/llama.cpp/releases/tag/b11912) Next: Keep --cache-ram on and larger than one long conversation plus its checkpoints (check the server log's 'total state size' line), point Open WebUI's task model at a small non-reasoning model or turn follow-ups and tags off, and cap task max_tokens. Next: measure the cache entry size and restore time on a Radeon R9700 lane at 60K and 200K tokens. ## GLM-5.3-Flash GGUF chat template is two upstream fixes behind: up to 17 s of template rendering per request with many parallel tool results in llama.cpp 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-glm-flash-chat-template-render The chat template embedded in the current Unsloth GLM-5.3-Flash GGUF matches the upstream 2026-08-27 version and misses the 2026-10-15 upstream rewrite of tool-result reordering. Rendered with llama.cpp's own Jinja engine, one block of 100 parallel tool results takes 1.14 s with the embedded template vs 31 ms with the new one (400 results: 17.0 s vs 0.19 s), with byte-identical output. Fix without re-downloading: pass the upstream template with --chat-template-file. Method: templates read from the GGUF headers and from upstream at each commit, rendered with llama.cpp's Jinja engine (common/jinja at master 4244ad7) and with Python Jinja2 3.1.6. Single-threaded, measured on an 8-vCPU cloud Xeon, not the R9700 desktop, so absolute times will differ; the ratio is the point. Long agent sessions: 60 turns x 4 calls 176 vs 81 ms; 30 x 16 910 vs 127 ms; 10 x 64 4,356 vs 175 ms. llama.cpp's engine was about 5x slower than Jinja2 on the same template. The 2026-09-04 upstream 'early break' change alone gave no speedup. 200 randomized conversations (shuffled, missing and duplicate tool ids, multi-turn) rendered byte-identical. Side finding on Qwen templates: the Unsloth GGUF templates for Qwen3.6-35B-A3B and Qwen3.8-Flash-Next accept a developer role, two leading system messages and a conversation with no user message, where upstream raises an error; but a system message after the first turn makes the Flash-Next GGUF template raise and the 35B GGUF template silently drop it. Keep the GGUF Qwen templates and avoid late system messages. - Model: GLM-5.3-Flash (also Qwen3.6-35B-A3B, Qwen3.8-Flash-Next templates) - Quant: UD-Q6_K_XL header (template is quant-independent) - Engine: llama.cpp Jinja engine + Python Jinja2 - Engine version: llama.cpp master 4244ad7; Jinja2 3.1.6 - GPUs: none (CPU-only template rendering) - Context: n/a - Flags: --chat-template-file - Render, 100 parallel tool results, GGUF template: 1140 ms - Render, 100 parallel tool results, upstream 2026-10-15 template: 31 ms - Render, 400 parallel tool results, GGUF template: 17001 ms - Render, 400 parallel tool results, upstream 2026-10-15 template: 188 ms Sources: [https://huggingface.co/zai-org/GLM-5.3-Flash/discussions/63](https://huggingface.co/zai-org/GLM-5.3-Flash/discussions/63), [https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF), [https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/chat_template.jinja](https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/chat_template.jinja), [https://github.com/ggml-org/llama.cpp/tree/4244ad7acd98328c1f04f1c2201d571f643370bb/common/jinja](https://github.com/ggml-org/llama.cpp/tree/4244ad7acd98328c1f04f1c2201d571f643370bb/common/jinja) Next: Time /apply-template on the R9700 desktop with both templates and fold --chat-template-file into the GLM preset if it saves time at a normal tool fan-out. ## R9700 street price today: $1,699.99, 31% over MSRP, and only in store 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-r9700-street-price Across the six US R9700 32 GB models, the cheapest new card in stock is $1,699.99 (Gigabyte AI TOP and XFX at Micro Center, in-store pickup), 31% above AMD's $1,299 MSRP. The lowest price seen since Sep 29 is $1,599.99. Amazon's in-stock new offers run $2,399.99 to $2,680.81. Snapshot taken Oct 26, 2026 around 4 PM CT from vga.live's public card pages, before tax. ASRock Creator, ASUS Turbo and Sapphire are $1,799.99 at Micro Center; PowerColor is only in stock on Amazon at $2,680.81. Newegg (ASRock, $1,699.99) and B&H (Gigabyte, $1,899.95) showed out of stock. A two-card setup (64 GB VRAM) costs $3,399.98 before tax today, $200 above the recent low. Prices move daily; check before buying. - GPUs: Radeon AI PRO R9700 32 GB (6 board-partner models) - Region: US, new condition, in stock - Cheapest new R9700 in stock: 1699.99 USD - Premium over $1,299 MSRP: 30.9 % - Lowest price since Sep 29: 1599.99 USD - Cheapest Amazon in-stock new offer: 2399.99 USD - Two cards today, before tax: 3399.98 USD Sources: [https://vga.live/gpus/radeon-ai-pro-r9700](https://vga.live/gpus/radeon-ai-pro-r9700), [https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation](https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation), [https://vga.live/cards/xfx-amd-radeon-ai-pro-r9700-single-fan-32gb-gddr6](https://vga.live/cards/xfx-amd-radeon-ai-pro-r9700-single-fan-32gb-gddr6), [https://vga.live/cards/asrock-amd-radeon-ai-pro-r9700-creator-single-fan-ai-workstation](https://vga.live/cards/asrock-amd-radeon-ai-pro-r9700-creator-single-fan-ai-workstation), [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html) Next: Track the cheapest new price daily and flag when it returns to the $1,599.99 floor. ## Strata's MTP draft layer only loads Q2_0 experts, so a Q4+-only R9700 setup loses drafts and the RDNA4 pipeline-windows split gain 2026-10-26 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-26-strata-mtp-q2-only-second-card In Strata main (2026-10-25), the runtime drafter sizes its 512 experts as kernels::cpu::BLOB, which is the Q2_0 layout (1,382,400 bytes per expert, 707,788,800 for the file). mtp_pack.py can write Q4_0 or Q8_0 draft experts, but nothing loads them (open issue #3728), and --mtp-q4 only requantizes the dense head, not the experts. Anyone who keeps every file at 4-bit or more therefore runs Strata with no draft layer. On a layer split that also turns off pipeline windows, the RDNA4 default that gave +21% code decode on 2x R9700 (103 to 125 tok/s) in the maintainer's table. Cited, not measured here: the maintainer's R9700 numbers (one card at 225 W, EPYC 7452 with 8-channel DDR4 at 106 GB/s, every expert in RAM, 40K context, MTP on) are UD-Q4 63-64 story / 85-88 code tok/s, UD-Q5 57-60 / 81-83, UD-Q6 54 / 76. Adding a second card by layer split moved UD-Q4 63 to 64 story and 88 to 96 code, UD-Q5 57 to 60 and 83 to 94; 32K prompts gain more at 3-4 cards (UD-Q4 1,110 to 2,381 tok/s on four). No two-R9700 number without a draft layer has been published. Second-card options that don't need a draft layer: layer split with per-card expert caches, STRATA_SPLIT_OVERLAP=1 (2x 7900 XTX code 53 to 70.1, about one card's speed), STRATA_STAGE_TRIM=1, helper expert caches with --remote-expert-opt (+63-132% over the plain helper path, measured only on 2x RTX 4090) and --peer-device. Also worth a deliberate choice: the default STRATA_ROUTE_TAIL_SKIP=7 moves UD-Q4's KL vs Q8_0 from 0.056 to 0.086 for +4-10% decode. Our own Strata Flash lane (UD-Q6_K_XL, one R9700, no draft layer, two-channel DDR5, engine 0.1.42) measured 32.6 tok/s; upstream is now at engine 0.1.50 with RDNA4 matrix-core prompt kernels. - Q2_0 draft expert size (per expert): 1,382,400 bytes - Pipeline windows, 2x R9700 code decode (with MTP, cited): 103 -> 125 tok/s - UD-Q5 1 -> 2 R9700 code decode (with MTP, cited): 83 -> 94 tok/s - UD-Q6 1 R9700 story / code (with MTP, 8ch DDR4, cited): 54 / 76 tok/s - Our UD-Q6 lane, 1 R9700, no draft layer (measured): 32.6 tok/s Sources: [https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/src/core/mtp.cpp](https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/src/core/mtp.cpp), [https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/include/strata/kernels/cpu/expert.hpp](https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/include/strata/kernels/cpu/expert.hpp), [https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/MULTI_GPU.md](https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/MULTI_GPU.md), [https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/SECOND_GPU.md](https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/SECOND_GPU.md), [https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/QUANTS.md), [https://github.com/Niko1221/Strata/issues/3728](https://github.com/Niko1221/Strata/issues/3728) Next: Test on the local desktop: engine 0.1.42 vs 0.1.50 on one card, then layer split, split overlap and helper-cache arms with no draft layer, gated by a correctness check. A Q8_0 or Q4_0 draft-expert runtime (#3728) would unlock drafts under a Q4+ rule. ## vLLM TP2 on two R9700s: every all-reduce goes through RCCL, and batch-1 decode pays it ~80-96 times per token 2026-10-26 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-26-vllm-tp2-allreduce-rccl-only In current vLLM source, the custom all-reduce, QuickReduce and AITER all-reduce paths are enabled only on MI300-class GPUs; on gfx1201 vLLM forces disable_custom_all_reduce on, so every tensor-parallel all-reduce is an RCCL call (this is why --disable-custom-all-reduce 'has no effect' on R9700). With two all-reduces per layer, Qwen3.6-35B-A3B makes 80 per token (4 KiB each) and Qwen3.8-Flash-Next 96 (5 KiB). The only published R9700 rccl-tests table (4 cards, P2P off) is flat at 31-36 us up to 32 KiB, which is about 2.9 and 3.4 ms of communication per token, before any compute. Estimate, not a measurement: if vLLM matched llama.cpp's ~100.8 tok/s single-card decode for the 35B, an 80-100% efficient TP2 split would land near 110-128 tok/s, so TP2 is mainly a capacity tool, not a big speed win, for models that already fit on one card. RCCL's low-latency LL protocol (auto-picked for AllReduce up to 16 KiB per rank on gfx120x) is what deadlocks on RDNA4 (ROCm rocm-systems #5480, open); the NCCL_PROTO=Simple workaround avoids it but likely costs small-message latency. AMD's patch to force Simple by default was closed unmerged on 2026-10-20 after its 4-GPU hang turned out to be a hipBLASLt issue. Published R9700 TP numbers (e.g. Qwen3.6-35B-A3B-FP8 TP2 17.4 tok/s) were run with --enforce-eager. Two platform traps: with iommu=pt, cross-process peer IPC on PCIe RDNA4 returns success but reads 0xFF and drops writes; and RCCL collectives currently fail on an R9700 attached through the chipset ('AQL dispatch failed!', rocm-systems #13616, open). - Model: Qwen3.6-35B-A3B / Qwen3.8-Flash-Next - Quant: n/a (communication analysis) - Engine: vLLM (main, 2026-10-26 source) + RCCL - Engine version: vLLM main 7f56f45 - GPUs: 2x Radeon AI PRO R9700 - Context: batch 1 decode - Flags: tensor-parallel-size 2, NCCL_PROTO=Simple - All-reduces per token, Qwen3.6-35B-A3B: 80 - All-reduces per token, Qwen3.8-Flash-Next: 96 - Published R9700 RCCL all-reduce latency, 8 B-32 KiB: 31-36 us - Comms time per token, 35B (from published table): 2.9 ms - Comms time per token, Flash-Next (from published table): 3.4 ms - Estimated TP2 decode ceiling, 35B (estimate): 110-128 tok/s Sources: [https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/platforms/rocm.py](https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/platforms/rocm.py), [https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/config/parallel.py](https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/config/parallel.py), [https://github.com/vllm-project/vllm/pull/19744](https://github.com/vllm-project/vllm/pull/19744), [https://github.com/ROCm/rccl-tests/issues/162](https://github.com/ROCm/rccl-tests/issues/162), [https://github.com/ROCm/rocm-systems/issues/5480](https://github.com/ROCm/rocm-systems/issues/5480), [https://github.com/ROCm/rocm-systems/pull/13151](https://github.com/ROCm/rocm-systems/pull/13151), [https://github.com/ROCm/ROCR-Runtime/issues/355](https://github.com/ROCm/ROCR-Runtime/issues/355), [https://github.com/ROCm/rocm-systems/issues/13616](https://github.com/ROCm/rocm-systems/issues/13616), [https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504](https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504) Next: Measure 2-card RCCL all-reduce latency (Simple vs shared-memory path vs a time-limited LL run) and vLLM TP2 decode with HIP graphs vs eager on the local desktop. ## Stale GGUFs after a re-upload: GLM-5.3-Flash needs one 9 MiB shard, not a 272 GiB re-download, and old Flash-Next MTP heads carry a wrong value 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-gguf-reupload-drift On 2026-10-05 Unsloth re-uploaded files in its GLM-5.3-Flash and Qwen3.8-Flash-Next GGUF repos 'due to upstream changes'. We diffed the old and new files from the Hugging Face history. For GLM-5.3-Flash only shard 1 of each quant changed, and it holds metadata only (9.0 MiB, no tensors): the architecture name went from glm5next to glm5-next, the name mainline llama.cpp merged on 2026-09-30, and one key was added (attention.indexer.index_share_mtp = true). A copy downloaded before then can be fixed by replacing that one shard instead of re-downloading 186-272 GiB. For Qwen3.8-Flash-Next the UD-Q4/Q5/Q6 model shards did not change; the three non-shared MTP head files did, by one value: the MTP layer's attention compress ratio went from 0 to 4. Method: listed every GGUF with its LFS sha256 at the commit and at its parent via the Hugging Face API, then parsed the GGUF headers of both versions. GLM-5.3-Flash UD-Q6_K_XL shard 1: 40 architecture keys renamed with the architecture, 1 key added, tensor data shards unchanged; the pre-change file's sha256 matches revisions from 2026-08-29 to 2026-09-06. The old glm5next name came from llama.cpp PR #27752, which was closed unmerged; PR #27773 merged support as glm5-next. Flash-Next MTP heads (BF16, Q4_K_M, Q8_0; same size and same 34 tensors): qwen4exp.attention.compress_ratios entry 48 changed 0 -> 4. We have not measured what the old value does to MTP acceptance. Qwen3.6-35B-A3B GGUFs were last re-uploaded 2026-04-20. We wrote a read-only checker that compares a local GGUF's size and header with the current Hugging Face file (or its full sha256 with --full) and labels it current, stale or corrupt; it correctly flagged an old GLM shard, a current shard and a shard with one flipped byte. - Model: GLM-5.3-Flash, Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (Unsloth GGUF) - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header diff on UD-Q6_K_XL shard 1 and MTP Q8_0) - Engine: llama.cpp (arch names from master) - Engine version: master, 2026-10-26 - GPUs: n/a (file analysis) - GLM-5.3-Flash shard that changed (metadata only): 9 MiB - GLM-5.3-Flash UD-Q6_K_XL full set: 271.8 GiB - GLM-5.3-Flash UD-Q4_K_XL full set: 186 GiB - Metadata keys changed in GLM shard 1: 82 keys (40 renamed pairs + arch + 1 new) - Flash-Next MTP heads changed: 3 files Sources: [https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/commit/a38483c8cd5df544f53d70fb281afe97369d5ab6](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/commit/a38483c8cd5df544f53d70fb281afe97369d5ab6), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/commit/efc50b7c3195a10eebd8d3e32f83b27b2aeb239d](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/commit/efc50b7c3195a10eebd8d3e32f83b27b2aeb239d), [https://github.com/ggml-org/llama.cpp/pull/27773](https://github.com/ggml-org/llama.cpp/pull/27773), [https://github.com/ggml-org/llama.cpp/pull/27752](https://github.com/ggml-org/llama.cpp/pull/27752), [https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/commits/main](https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/commits/main) Next: Check our own GLM-5.3-Flash and Flash-Next MTP copies with the checker; if stale, swap the 9 MiB shard and A/B MTP acceptance with the old vs new head. ## Faster Qwen3.6-35B-A3B on one R9700: use the built-in MTP head, not a draft model, and watch the 262K fit 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-35b-a3b-mtp-vs-draft-r9700 Published tests show a separate small draft model makes Qwen3.6-35B-A3B slower (one RTX 3090: 135.7 tok/s baseline vs 115.9-121.3 mean with drafts), while the model's own MTP head helps one user at a time: on an R9700 (Q4_K_M, Vulkan) one slot went from about 108 to about 133 tok/s, but MTP made a 3-slot job 1.58x slower. We read the headers of Unsloth's MTP GGUFs: the extra MTP block is Q8_0 attention with Q5_K/Q6_K experts in UD-Q5_K_XL, nothing below Q4. Size cost from the Hugging Face file listings: the MTP files are 470 MiB (UD-Q4_K_XL), 540 MiB (UD-Q5_K_XL) and 732 MiB (UD-Q6_K_XL) larger than the plain files. The MTP block has its own attention layer, so its KV cache adds about 512 MiB at f16 or 272 MiB at q8_0 for 262K context (K/V width 512 from the header). Our estimate for UD-Q5_K_XL with MTP at 262K and f16 KV is about 30.9 GiB before compute buffers, which is tight or over on a 32 GiB card with ECC on (about 29.9 GiB visible); quantizing only the draft cache or capping context near 128K should fit (estimate, not measured). Other data points: a 7900 XTX on ROCm gained 1.70x at 16K falling to 1.35x at 196K and ran out of memory at 245K; Unsloth reports 1.15-1.2x on average for this MoE at 2 draft tokens, with acceptance falling from 83% to 50% at 4; the community DSpark drafter beat MTP on code (+12.7%) but lost on prose (-4.8%) on non-Radeon hardware. A later re-run of the 3090 study found the slowdown tracks low real acceptance plus draft overhead. - Model: Qwen3.6-35B-A3B (MTP GGUF) - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header check); Q4_K_M and UD-Q4_K_XL in cited speed tests - Engine: llama.cpp - Engine version: per source - GPUs: 1x R9700 (cited); RTX 3090 and 7900 XTX for comparison - Context: short to 245K - Flags: --spec-type draft-mtp --spec-draft-n-max 2, -np 1 - R9700 Q4_K_M Vulkan, 1 slot, no MTP: 108 tok/s - R9700 Q4_K_M Vulkan, 1 slot, MTP: 133 tok/s - R9700, 3 slots, MTP slowdown: 1.58 x slower - RTX 3090 baseline vs best draft-model mean: 135.7 vs 121.1 tok/s - MTP file size increase, UD-Q5_K_XL: 540 MiB - MTP draft KV at 262K, f16: 512 MiB Sources: [https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/](https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/), [https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/discussions/14](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/discussions/14), [https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090](https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090), [https://gist.github.com/AshyIsMe/ef9dc0e57d7af5626f3935f55c38543a](https://gist.github.com/AshyIsMe/ef9dc0e57d7af5626f3935f55c38543a), [https://unsloth.ai/docs/models/tutorials/qwen3.6](https://unsloth.ai/docs/models/tutorials/qwen3.6), [https://github.com/ggml-org/llama.cpp/discussions/26230](https://github.com/ggml-org/llama.cpp/discussions/26230), [https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF) Next: Measure MTP at 1, 2 and 3 draft tokens on our single-R9700 UD-Q5_K_XL lane at 128K, then test whether 262K fits with a q8_0 draft cache. ## Two Radeon cards, wrong answers at full speed: which split setups corrupt text, what fixed it, and a correctness check 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-multi-gpu-silent-corruption Several 2026 Radeon multi-GPU setups on ROCm/HIP produced fluent gibberish or blank output at normal speed with no error, and llama-bench still reported healthy numbers. The Windows layer-split case was fixed in ROCm's runtime (PR #10553, merged 2026-09-02); Windows -sm tensor on ROCm is still open; Linux cards with no peer-to-peer path need a no-peer-copy build. Always check text, not just tok/s, after a stack change. Windows HIP, -sm layer (2x R9700, 7900 XTX + 9070 XT and others): a staged GPU-to-GPU copy raced in-flight compute; fixed in ROCm CLR PR #10553 and confirmed by AMD on a current Lemonade build, but older Windows ROCm releases (e.g. b10767) still corrupt. Building llama.cpp with -DGGML_CUDA_NO_PEER_COPY=ON (a cmake option; an environment variable does nothing) also fixes it; one mixed-card test went from 0/7 to 7/7 exact answers at about -4.5% prefill and -7.7% decode. Windows HIP -sm tensor: open (Lemonade llamacpp-rocm #148); root cause is hipMemcpy2DAsync from memory-mapped model pages, and --load-mode none avoids it. Even when correct, Windows exposes no GPU peer access, so a second R9700 adds memory, not speed (open driver request ROCm/rocm-systems #13715). Linux, 2x RX 9060 XT on separate root ports (ROCm 7.2.4): HSA_ENABLE_PEER_SDMA=0 did not fix it; the no-peer-copy build did. Quick check: ask a few one-answer questions at temperature 0 on one card and then split across both; answers should match exactly. - Model: Qwen3.8-27B Q4_K_M (Windows speed test); various (corruption reports) - Quant: >=Q4 in all cited tests - Engine: llama.cpp (HIP and Vulkan), Strata - Engine version: per source - GPUs: 2x R9700; 7900 XTX + 9070 XT; 2x 9060 XT; others - Context: short - Flags: -sm layer / -sm tensor, -DGGML_CUDA_NO_PEER_COPY=ON, --load-mode none - Windows 2x R9700 Vulkan, one card (-sm none): 32.1 tok/s - Windows 2x R9700 Vulkan, -sm layer: 31.3 tok/s - Windows 2x R9700 Vulkan, -sm tensor: 14.9 tok/s - No-peer-copy build, decode cost (one mixed-card test): -7.7 % Sources: [https://github.com/ggml-org/llama.cpp/issues/28676](https://github.com/ggml-org/llama.cpp/issues/28676), [https://github.com/lemonade-sdk/llamacpp-rocm/issues/75](https://github.com/lemonade-sdk/llamacpp-rocm/issues/75), [https://github.com/ROCm/rocm-systems/pull/10553](https://github.com/ROCm/rocm-systems/pull/10553), [https://github.com/lemonade-sdk/llamacpp-rocm/issues/148](https://github.com/lemonade-sdk/llamacpp-rocm/issues/148), [https://github.com/ROCm/rocm-systems/issues/13715](https://github.com/ROCm/rocm-systems/issues/13715), [https://github.com/ROCm/ROCm/issues/6342](https://github.com/ROCm/ROCm/issues/6342) Next: Run the same exact-answer check on our dual-R9700 Linux launchers (Vulkan and HIP, layer and tensor) and make it a required step before every benchmark. ## llama.cpp slot save/restore now works for hybrid Qwen models: resume a long agent context after a restart 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-slot-restore-hybrid-checkpoints Before llama.cpp b11507, restoring a saved slot on hybrid (Gated DeltaNet) models like Qwen3.5/3.6/3.8 reported success but re-processed the whole prompt, because context checkpoints were not saved. PR #26004 now appends them to the slot file. A published AMD Vulkan test (Radeon 8050S, Qwen3.5-35B-A3B UD-Q4_K_XL, 58K tokens) cut the first request after restore from 181.9 s to 4.7 s. Source read at llama.cpp master c6fb3b5: the slot file carries the KV payload, then a checkpoint appendix, then (since PR #30574, b11669) the draft/MTP state, so speculative decoding keeps its acceptance after a restore (37% to 49% in the PR's Qwen3.8-27B test). Older files still load; mismatched draft data is dropped instead of crashing. Testing trap: the RAM prompt cache (--cache-ram) can hide a broken disk restore, so test with --cache-ram 0. Estimate from the Qwen3.6-35B-A3B config (not measured): about 20 KiB per token of f16 KV plus about 63 MiB of recurrent state, so roughly 1.2 GiB at 58K tokens and 5.1 GiB at 262K. A test on dual R9700 is next. - Model: Qwen3.5-35B-A3B (published test); Qwen3.6-35B-A3B (size estimate) - Quant: UD-Q4_K_XL (published test) - Engine: llama.cpp llama-server - Engine version: >= b11507 (checkpoints), >= b11669 (draft state) - GPUs: Radeon 8050S (published test) - Context: 58K tokens - Flags: --slot-save-path DIR, --ctx-checkpoints, --cache-ram 0 for testing - First request after restore, before fix: 181.9 s - First request after restore, with fix: 4.7 s - Tokens re-processed after restore, with fix: 516 tokens - Est. slot file, 35B-A3B at 262K, f16 KV: 5.1 GiB Sources: [https://github.com/ggml-org/llama.cpp/pull/26004](https://github.com/ggml-org/llama.cpp/pull/26004), [https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5055464245](https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5055464245), [https://github.com/ggml-org/llama.cpp/pull/30574](https://github.com/ggml-org/llama.cpp/pull/30574), [https://github.com/ggml-org/llama.cpp/issues/28194](https://github.com/ggml-org/llama.cpp/issues/28194), [https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json) Next: Measure save, restart and restore times on dual R9700 for the 35B and the Flash llama.cpp lane, then wire save-on-stop and restore-on-start into the server units. ## Capping thinking on a slow Flash model: the stop message matters more than the token count 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-thinking-budget-latency At about 33 tok/s (a Flash-class MoE on 2x R9700 with experts in DDR5), every 1,000 thinking tokens is roughly 31 seconds before the first visible word. llama.cpp can cap this per request with thinking_budget_tokens plus reasoning_budget_message, and since June 2026 a client can end thinking mid-stream with POST /v1/chat/completions/control, which allows a wall-clock deadline instead of a token count. A bare cap without a stop message hurts: HumanEval on a Qwen3.5 9B fell to ~79% vs ~89% with a message (~93% uncapped), and on Qwen3.6-35B-A3B a bare 4,096-token cap ran 170 s as reasoning spilled into the answer, vs 78 s with Qwen's own stop sentence. In llama.cpp master (2026-10-26) the per-request budget overrides the --reasoning-budget server flag unless it is -1, the sampler re-arms on every new think block, and nothing happens without --jinja because the think tags come from the chat template. Qwen's documented stop sentence is: 'Considering the limited time by the user, I have to give the solution based on the thinking directly now.' Arithmetic at measured decode speeds: 2,048 thinking tokens take about 63 s on the Flash model at 32.6 tok/s, about 128 s on a 16 tok/s fallback, and about 20 s on a 35B-A3B at 100.8 tok/s on one R9700. A 20-second deadline allows about 650 thinking tokens on the Flash model vs about 2,000 on the 35B. Whether the Strata engine honours these request fields is not yet checked. - Model: Qwen3.8 Flash (UD-Q6_K_XL) and Qwen3.6-35B-A3B (UD-Q5_K_XL) - Engine: llama.cpp server (source review), Strata (speed only) - Engine version: llama.cpp master 8b28c23 - GPUs: 2x Radeon AI PRO R9700 (Flash); 1x R9700 (35B) - Flash decode used for the arithmetic: 32.6 tok/s - Seconds per 1,000 thinking tokens (Flash): 31 s - HumanEval, budget 1000 with message (Qwen3.5 9B): 89 % - HumanEval, budget without message (Qwen3.5 9B): 79 % - Bare 4,096 cap, 35B-A3B, 3 prompts: 170 s - 4,096 cap with stop message, 35B-A3B: 78 s Sources: [https://github.com/ggml-org/llama.cpp/pull/20297](https://github.com/ggml-org/llama.cpp/pull/20297), [https://github.com/ggml-org/llama.cpp/pull/23971](https://github.com/ggml-org/llama.cpp/pull/23971), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder/commit/59aa5b59f6ecd7a7fa9cc191e1e40d064ad2b2e7](https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder/commit/59aa5b59f6ecd7a7fa9cc191e1e40d064ad2b2e7), [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388) Next: Measure natural thinking length and capped accuracy on real agent turns for both models, then pick a default cap or deadline. ## R9700 GPU hangs come in three kinds, and only one is fixed by restarting the server 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-r9700-hang-recovery-classes Public Radeon AI PRO R9700 (gfx1201) fault reports split into three recovery classes. (1) The server process dies and the driver's reset succeeds: a systemd Restart=on-failure is enough. (2) A silent wedge: the server stays up at 100% GPU busy and 0% memory busy but produces no tokens, or blocks at idle power with no error, so nothing restarts it and a /health check still answers 200. (3) A dead GPU: the reset ends with 'PSP resume failed' / ret -62 or a reset storm with a CPU soft lockup, and only a reboot helps; repeated auto-retries in that state have hard-hung an R9700. Practical rules for unattended serving: probe with a tiny real completion and a wall-clock timeout instead of trusting /health; scan the kernel log for PSP/IP-block resume failures and stop (don't restart) when they appear; cap restarts (for example 3 per 30 minutes) so a failing load can't loop. amdgpu.cwsr_enable=0 is circulating as a fix but one report saw it turn clean faults into 'GPU Hang' on 3 of 4 R9700s. In one controlled A/B the same llama.cpp workload that wedged on ROCm 7.14 passed on Vulkan RADV, so a Vulkan build is a reasonable fallback service. The hipBLASLt page fault behind many training-time resets is fixed by rocm-libraries PR 8909 (in ROCm 10.2 nightlies as of 2026-09-21). We wrote a stdlib watchdog that encodes these rules and passes 8 of 8 simulated cases; it has not yet been run against real hangs. - Engine: llama.cpp / vLLM (published reports) - GPUs: Radeon AI PRO R9700 (gfx1201) - No-output stall before timeout (ROCm 7.14, #6630): 180 s - R9700s that degraded to GPU Hang with cwsr_enable=0: 3 of 4 - Successful MODE2 resets recovered by a 10 s systemd restart (gfx1151 report): 4 Sources: [https://github.com/ROCm/legacy-rocm-build/issues/6630](https://github.com/ROCm/legacy-rocm-build/issues/6630), [https://github.com/ROCm/ROCm/issues/6413](https://github.com/ROCm/ROCm/issues/6413), [https://github.com/ROCm/TheRock/issues/9271](https://github.com/ROCm/TheRock/issues/9271), [https://github.com/lemonade-sdk/lemonade/issues/2609](https://github.com/lemonade-sdk/lemonade/issues/2609), [https://github.com/ROCm/ROCm/issues/6127](https://github.com/ROCm/ROCm/issues/6127), [https://docs.kernel.org/gpu/amdgpu/debugfs.html](https://docs.kernel.org/gpu/amdgpu/debugfs.html) Next: Run the watchdog in log-only mode for three nights on the dual-R9700 desktop, then turn on automatic restart, fallback and stop-and-alert. ## Securing a local LLM server: what llama-server's API key covers, 2026 CVEs, and why ggml RPC must stay on a private link 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-endpoint-security llama-server binds 127.0.0.1 by default, and --api-key protects every route except /health and the web-UI files. Without a key, /props, /slots and /v1/models are open, and the default CORS setting echoes any Origin. Builds from b8227 up to but not including b11393 have a 9.2-rated use-after-free on POST /completion (CVE-2026-107183), so update to b11393 or newer. The RPC backend used for multi-machine splits has no authentication and had a 9.8-rated remote code execution bug this year (fixed in b8492), so it belongs on a dedicated point-to-point link only. Sources: llama.cpp master server source (public endpoints list) and server/RPC READMEs; CVE records for CVE-2026-107183 (fixed b11393, PR #29942; Debian's sid package 0.7.0 still listed vulnerable), CVE-2026-43632 (tokenizer-endpoint use-after-free when --sleep-idle-seconds is set), CVE-2026-21869 (negative n_discard with --context-shift), CVE-2026-34159 (RPC GRAPH_COMPUTE RCE). vLLM's own security docs say --api-key guards only /v1, /v2, /inference and /cohere; /invocations, /pause, /abort_requests and /metrics stay open. A September 2026 internet census found 36,769 exposed self-hosted AI endpoints, of which 2.02% returned an auth challenge (llama.cpp 69 found, 0 authenticated; vLLM 4,880 found, 3 authenticated). Our posture for a dual-R9700 box: keep every server on localhost and reach it through one reverse proxy or VPN tunnel; if a LAN bind is needed use --api-key-file, set --cors-origins to the frontend, and turn off /slots; run rpc-server bound only to the inter-machine link address. We also wrote a read-only probe that checks which routes answer without a key and whether CORS reflects arbitrary origins; a bare TCP check on the RPC port can false-positive, since other services may use 50052. - Engine: llama.cpp llama-server, ggml RPC, vLLM - Engine version: llama.cpp master 9125a2e (release v0.7.0) - GPUs: 2x Radeon AI PRO R9700 (applies to any GPU) - CVE-2026-107183 fixed in: b11393 - CVE-2026-107183 CVSS 4.0: 9.2 - CVE-2026-34159 (RPC) CVSS 3.1: 9.8 - Exposed AI endpoints with auth challenge (Sept 2026 census): 2.02 % Sources: [https://www.cve.org/CVERecord?id=CVE-2026-107183](https://www.cve.org/CVERecord?id=CVE-2026-107183), [https://www.cve.org/CVERecord?id=CVE-2026-34159](https://www.cve.org/CVERecord?id=CVE-2026-34159), [https://github.com/ggml-org/llama.cpp/security/advisories?state=published](https://github.com/ggml-org/llama.cpp/security/advisories?state=published), [https://github.com/advisories/GHSA-3f2h-96vw-c84x](https://github.com/advisories/GHSA-3f2h-96vw-c84x), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), [https://docs.vllm.ai/en/latest/usage/security/](https://docs.vllm.ai/en/latest/usage/security/), [https://www.mysteriumvpn.com/blog/data-and-research/we-exposed-ai-supply-chain](https://www.mysteriumvpn.com/blog/data-and-research/we-exposed-ai-supply-chain) Next: Run the read-only probe against every listener on our own box and from a second LAN machine, then pin all llama.cpp builds at b11393 or newer before the two-box RPC test. ## Text-to-speech next to a local LLM on R9700: llama-server now serves Qwen3-TTS, and a CPU voice would compete with offloaded experts for DDR5 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-r9700-tts-llama-server llama.cpp merged a native POST /tts endpoint for llama-server on 2026-10-20 (release b11781), with streaming, speaker-reference upload and router-mode autoload, so a TTS model can sit behind the same server as the LLM. Published R9700 results for Qwen3-TTS: the PyTorch reference runs at RTF 1.90 (slower than real time), while a custom HIP port reaches RTF 0.55 for the 1.7B model and 0.41-0.54 for the 0.6B. No llama.cpp R9700 TTS number has been published yet. The 1.7B Base GGUFs are Q4_K_M 0.965 GiB, Q8_0 1.721 GiB and bf16 3.234 GiB, plus a 0.416-0.623 GiB mmproj, all at or above 4-bit. A review of the PR found that code2wav pads every decode batch to 72 frames, so shrinking batches for faster first audio detunes the voice (speaker similarity 0.817 vs 0.862); a not-yet-merged patch with a 12/24/48/72 frame ramp cut time to first audio from about 1.8 s to 0.4 s on Vulkan. Fixed seeds are not reproducible on the server yet. Estimate only (bandwidth roofline from the published per-frame weight reads): on an R9700 every quant is far faster than real time (RTF about 0.07-0.23), while on CPU with dual-channel DDR5 at a measured 55 GB/s Q8_0 lands near RTF 1.05 and Q4_K_M near 0.60, and that traffic competes with any MoE experts offloaded to the same DDR5. - Model: Qwen3-TTS-12Hz 1.7B and 0.6B - Engine: llama.cpp llama-server /tts (b11781+); custom HIP port; PyTorch reference (published) - GPUs: 1x Radeon AI PRO R9700 - Qwen3-TTS 1.7B, PyTorch FP32 reference, RTF: 1.9 x - Qwen3-TTS 1.7B, custom HIP port, RTF: 0.55 x - Qwen3-TTS 0.6B, custom HIP port streaming, RTF: 0.54 x - Qwen3-TTS 1.7B Q8_0 GGUF + bf16 mmproj: 2.34 GiB - Time to first audio with code2wav ramp patch (unmerged, Vulkan): 0.4 s - Estimated CPU RTF, Q4_K_M at 55 GB/s DDR5 (estimate): 0.6 x Sources: [https://github.com/ggml-org/llama.cpp/pull/26603](https://github.com/ggml-org/llama.cpp/pull/26603), [https://github.com/ggml-org/llama.cpp/pull/26254](https://github.com/ggml-org/llama.cpp/pull/26254), [https://github.com/ggml-org/llama.cpp/tree/master/tools/tts](https://github.com/ggml-org/llama.cpp/tree/master/tools/tts), [https://huggingface.co/ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF](https://huggingface.co/ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF), [https://flynnbody.com/blog/running-qwen3-tts-on-amd-r9700/](https://flynnbody.com/blog/running-qwen3-tts-on-amd-r9700/), [https://github.com/piedshag/qwen3-tts-hip](https://github.com/piedshag/qwen3-tts-hip) Next: Measure llama-server /tts on our own R9700s (Vulkan vs HIP vs CPU), then again while the Flash model decodes, to see what speaking costs the LLM. ## Image and video generation on one R9700: step-distill LoRAs give ~6x, the hipBLASLt/MIOpen flags slow video 5-7% 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-r9700-image-video-gen From published R9700 ComfyUI runs: SDXL 1024x1024 runs at 4.49-4.6 it/s. 4-step distillation LoRAs cut Qwen-Image-2512 from 85.9 s to 12.4 s and Wan 2.2 A14B 480p from 341.7 s to 56.6 s, far more than any runtime knob. In a same-box A/B, TORCH_BLAS_PREFER_HIPBLASLT=1 plus COMFYUI_ENABLE_MIOPEN=1 made all four video workflows 5.3-7.3% slower and left Qwen-Image within 2%. Wan 2.2 T2V A14B Q6_K GGUF at 720p (33 frames, 4 steps) peaks at 31.8 of 31.86 GiB on one card, 27.7 GiB at 480p, so video cannot share a card with any resident LLM layers; even SDXL peaks at 12.87 GiB. Video runs drew 500-568 W at 97-104 C junction vs 342-503 W for image runs (same source). Cold starts dominate short jobs: Qwen-Image 4-step 33.4 s cold vs 12.4 s warm. The flag A/B is one warm run per config, so differences under ~2% are noise; the video slowdown is consistent in sign across Wan 2.2 I2V/T2V and HunyuanVideo 1.5 I2V/T2V. Qwen-Image-Edit 20-step was 7.0% faster with the flags. All weights cited are 4-bit or higher (FP8, Q6_K, BF16). - Model: SDXL 1.0, Qwen-Image-2512 FP8, Wan 2.2 A14B (FP8 and Q6_K), HunyuanVideo 1.5 - Engine: ComfyUI (published) - Engine version: per source (ROCm 7.1-7.2.4) - GPUs: 1x Radeon AI PRO R9700 - SDXL 1024x1024 sampling: 4.49 it/s - Qwen-Image-2512 FP8, 20 steps, warm: 85.9 s - Qwen-Image-2512 FP8, 4-step LoRA, warm: 12.4 s - Wan 2.2 A14B Q6_K 480p 33 fr, 4-step LoRA: 56.6 s - Wan 2.2 A14B Q6_K 720p VRAM peak: 31.8 GiB - Wan 2.2 I2V with hipBLASLt+MIOpen flags: 5.3 % slower Sources: [https://kyuz0.github.io/amd-r9700-comfy/](https://kyuz0.github.io/amd-r9700-comfy/), [https://kyuz0.github.io/amd-r9700-comfy/benchmark_results.json](https://kyuz0.github.io/amd-r9700-comfy/benchmark_results.json), [https://brookesia.app/boards/ai-hardware/76](https://brookesia.app/boards/ai-hardware/76), [https://brookesia.app/boards/ai-hardware/78](https://brookesia.app/boards/ai-hardware/78), [https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html](https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html) Next: Measure an SDXL baseline, an ABBA test of the two flags, and two ComfyUI instances (one per card) on our own pair of R9700s. ## LoRA fine-tuning on the R9700: the hipBLASLt page-fault trap and what fixes it 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-r9700-lora-pagefault The R9700 (gfx1201) is officially supported for LoRA/QLoRA training (Unsloth lists RDNA4 as Full; TheRock lists gfx1201 Release Ready), but stock hipBLASLt GEMMs can fault in the LoRA backward pass at a fixed step. On ROCm 7.2.4 a reporter saw bf16 LoRA on Qwen3-8B fault at step 85 and QLoRA NF4 at step 799. The fix (rocm-libraries PR 8909, merged 2026-08-14) ran clean in independent validations at about 1.5% extra step time. Cause: RDNA4's transpose global load has no hardware bounds field, so some Tensile GEMMs read past the end of a tensor; PyTorch's caching allocator places tensors next to unmapped pages, so the over-read faults. The fault step depends on data order and resume point, so resuming from a checkpoint with the same order dies at the same step. On stock libraries, PYTORCH_HIP_ALLOC_CONF=roundup_power2_divisions:16 had no measurable cost and strongly reduced faults but did not remove them; disabling hipBLASLt (TORCH_BLAS_PREFER_HIPBLASLT=0, ROCBLAS_USE_HIPBLASLT=0, DISABLE_ADDMM_HIP_LT=1) was clean but about 2.5x slower. HSA_ENABLE_SDMA=0, HSA_XNACK=1 and max_split_size_mb did not help. Other reported tips: plain adamw_torch instead of paged or 8-bit optimizers, frequent checkpoints, change the data seed on resume, and reboot after a fault before relaunching. Multi-GPU (RCCL) training is Linux-only per Unsloth. We have not yet seen the fix named in a stable 7.2.x release note. - Model: Qwen3-8B, Qwen3.6-27B (reproducers) - Engine: PyTorch ROCm + PEFT / Unsloth - Engine version: ROCm 7.2.4, torch 2.13.0 (issue 6600) - GPUs: Radeon AI PRO R9700 (gfx1201), published reports - bf16 LoRA fault step, stock ROCm 7.2.4: 85 step - QLoRA NF4 fault step, stock ROCm 7.2.4: 799 step - 12-step run, stock vs PR 8909 build: 726 vs 737 s - hipBLASLt-off workaround step time: 2.5 x slower Sources: [https://github.com/ROCm/legacy-rocm-build/issues/6600](https://github.com/ROCm/legacy-rocm-build/issues/6600), [https://github.com/ROCm/rocm-libraries/pull/8909](https://github.com/ROCm/rocm-libraries/pull/8909), [https://unsloth.ai/docs/basics/amd](https://unsloth.ai/docs/basics/amd), [https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md](https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md) Next: Run a one-card LoRA smoke test on the local desktop comparing stock, allocator rounding and hipBLASLt-off, then QLoRA NF4. ## Local RAG for an R9700 agent: Qwen3-Embedding-4B Q5_K_M is the value point, 0.6B pair fits in ~1.2 GiB 2026-10-26 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-26-embed-rerank-pick From official GGUF sizes and vendor scores: Qwen3-Embedding-4B Q5_K_M (2.69 GiB) scores 69.45 on MTEB multilingual, +5.1 over the 0.6B Q8_0 (0.60 GiB, 64.33), while the 8B Q5_K_M (5.05 GiB) adds only +1.1 more. Qwen3-Reranker-0.6B (0.60 GiB at Q8_0) lifts retrieval to 65.80 MTEB-R from 61.82, well ahead of bge-reranker-v2-m3 at 57.03. llama.cpp serves both: --embedding --pooling last for Qwen3-Embedding and --reranking for /v1/rerank. Bidirectional rerankers such as bge-reranker-v2-m3 must fit the whole prompt in --ubatch-size; causal Qwen3 rerankers do not have that limit. Qwen reports that adding a query-side instruction ('Instruct: ...\nQuery: ...') improves retrieval by about 1-5%. Scores are vendor-reported (Qwen model cards), not measured here; reranker scores are on the top-100 candidates from Qwen3-Embedding-0.6B. Whether the small pair is fast enough on CPU, and how much a GPU embedder slows a co-resident Flash model, is still to be measured. - Model: Qwen3-Embedding 0.6B/4B/8B, Qwen3-Reranker-0.6B, bge-reranker-v2-m3 - Quant: Q8_0 / Q5_K_M - Engine: llama.cpp server (docs at master d854d98bae) - GPUs: Radeon AI PRO R9700 (planned placement) - Qwen3-Embedding-0.6B Q8_0 size: 0.595 GiB - Qwen3-Embedding-4B Q5_K_M size: 2.691 GiB - Qwen3-Embedding-8B Q5_K_M size: 5.05 GiB - MTEB multilingual mean, 0.6B / 4B / 8B: 64.33 / 69.45 / 70.58 - MTEB-R, Qwen3-Reranker-0.6B vs bge-reranker-v2-m3: 65.80 vs 57.03 Sources: [https://huggingface.co/Qwen/Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B), [https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF](https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF), [https://huggingface.co/Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B), [https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF](https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) Next: Measure CPU vs one-R9700 placement for the 0.6B pair and the 4B embedder, including the decode drop on a co-resident Flash model. ## Strix Halo vs Radeon AI PRO R9700 on the same benchmark harness: ~2x decode, 3-4x prefill when the model fits 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-strix-halo-vs-r9700 Joining kyuz0's two public llama-bench grids (same scripts, best backend per row, 8 overlapping Q4+ models), one or two R9700s lead a 128 GB Ryzen AI Max+ 395 by a median ~2.0x on decode (tg128) and ~2.9x on prompt processing (pp512), widening to ~3.6-4.1x prefill at 32K-64K depth. Strix Halo's edge is capacity: it runs 80-90 GiB Q4/Q5 MoE models entirely in unified memory. gpt-oss-120b MXFP4 (59 GiB, two R9700s): 98.0 vs 56.6 tok/s decode, 1956 vs 720 tok/s pp512. Qwen3.6-35B-A3B UD-Q4_K_XL (one R9700): 108.3 vs 60.4 tok/s decode; at 64K depth 82.4 vs 43.2 tok/s decode and 2048 vs 548 tok/s prefill. Strix Halo runs Qwen3.5-122B-A10B UD-Q5_K_XL (85.6 GiB) at 22.3 tok/s and Nemotron-3-Super-120B-A12B UD-Q4_K_XL (78 GiB) at 14.9 tok/s; those exceed 64 GB of R9700 VRAM and need DDR5 expert offload on a desktop. Caveats: different llama.cpp builds (b8990 vs b9187/b9193) and OS versions; best backend chosen separately per platform; Q1-Q3 rows excluded. - Model: 8 overlapping models (gpt-oss-120b, Qwen3.6-35B-A3B, Gemma 4, others) - Quant: Q4+ - Engine: llama.cpp llama-bench (published grids) - Engine version: b8990 (R9700), b9187/b9193 (Strix Halo) - GPUs: 1-2x Radeon AI PRO R9700 vs Ryzen AI Max+ 395 128 GB - Context: 0, 32K and 64K depth - Flags: fa on, ngl 99, best of ROCm and Vulkan backends - Median decode lead, R9700 vs Strix Halo (tg128): 2 x - Median prompt lead (pp512): 2.9 x - Median prefill lead at 64K depth: 4 x - gpt-oss-120b decode, 2x R9700: 98 tok/s - gpt-oss-120b decode, Strix Halo: 56.6 tok/s - Qwen3.5-122B-A10B UD-Q5_K_XL decode, Strix Halo: 22.3 tok/s Sources: [https://kyuz0.github.io/amd-r9700-ai-toolboxes/](https://kyuz0.github.io/amd-r9700-ai-toolboxes/), [https://kyuz0.github.io/amd-strix-halo-toolboxes/](https://kyuz0.github.io/amd-strix-halo-toolboxes/), [https://github.com/lhl/strix-halo-testing/tree/main/llm-bench](https://github.com/lhl/strix-halo-testing/tree/main/llm-bench) Next: Measure Qwen3.5-122B-A10B UD-Q5_K_XL on two R9700s with experts in DDR5, with the same tests, against Strix Halo's 22.3 tok/s. ## Swapping big models on one box: llama.cpp router mode evicts by count, not VRAM; llama-swap can budget per card 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-model-swap-router llama-server's router mode (start it with no model and a --models-preset INI) loads models on demand, but it decides what to unload only by counting: --models-max defaults to 4 and the scheduler never checks VRAM or RAM. On a dual-R9700 box where one Flash-class model fills both cards and DDR5, run it with --models-max 1, or a request naming a second big model will try to load it alongside the first. llama-swap's capacity router lets you declare per-card and RAM budgets instead, and can also host non-llama.cpp servers. Read from llama.cpp master d854d98bae (2026-10-25): server_lru_sched::has_capacity() compares the number of running models with models_max, and a sleeping model still counts as running, so it holds a slot after its memory is freed. --sleep-idle-seconds (PR #18228) unloads the model and KV cache after the idle time; GET /health, /props, /models and /metrics neither wake it nor reset the timer, so monitoring doesn't keep the cards busy. Presets accept any CLI flag as an INI key plus load-on-startup and stop-timeout (default 10 s). Router mode only runs llama.cpp and can't express rules like 'these two single-card models may coexist, nothing else may'. llama-swap v266 runs any OpenAI-compatible server as a child process and has three routing engines: group (one model at a time by default), matrix (expressions listing allowed combinations, evicting by evict_cost) and capacity (declared pools such as gpu0, gpu1 and ram, with a requirement per model). The capacity engine does no automatic measurement, so per-model footprints have to be measured first. Load, swap and wake times on R9700s are not measured yet. - Engine: llama.cpp master d854d98bae and llama-swap v266 (source and docs read) - Hardware: planned for 2x Radeon AI PRO R9700 + DDR5 - router --models-max default: 4 models - preset stop-timeout default: 10 s - llama-swap routing engines: 3 engines Sources: [https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/server-models.cpp](https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/server-models.cpp), [https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/README.md](https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/README.md), [https://github.com/ggml-org/llama.cpp/pull/18228](https://github.com/ggml-org/llama.cpp/pull/18228), [https://github.com/mostlygeek/llama-swap/blob/v266/docs/config.example.yaml](https://github.com/mostlygeek/llama-swap/blob/v266/docs/config.example.yaml), [https://github.com/mostlygeek/llama-swap/blob/v266/README.md](https://github.com/mostlygeek/llama-swap/blob/v266/README.md) Next: Measure each preset's per-card VRAM and DDR5 footprint, cold-load, swap-in and wake times on 2x R9700, then decide whether the always-on agent model goes behind the swap endpoint. ## Tool calls and JSON mode in llama.cpp: lazy grammars, rejection sampling, and why -bs switches off 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-llamacpp-grammar-sampling-cost In current llama.cpp, a request with tools and tool_choice "auto" uses a lazy grammar that stays idle until the model starts a tool call, while tool_choice "required" or a response_format JSON schema constrains every token from the start. Sampling checks only the chosen token against the grammar and computes the full-vocabulary mask only when that token is rejected. Experimental backend (GPU) sampling, -bs, is turned off for any request that carries a grammar, lazy tool grammars included, so agent requests always sample on the CPU. Read from llama.cpp master d854d98bae (2026-10-25): common/sampling.cpp logs "backend sampling is not compatible with grammar, disabling" and also disables it when a reasoning budget is active; rejection sampling was restored in PR #18137 because grammar sampling is widely used and slow. Published -bs results are CUDA-only: no change at 1 slot and +48% at 32 slots (issue #27050), about 8% with MTP speculative decoding on an RTX 5090 (PR #25532); PR #17004 recommends CUDA for now. In MaskBench (128K-token vocab, one core), llama.cpp's native grammar mask averaged 16.6 ms vs 64 us for LLGuidance, and llama.cpp handled 6,505 of 11,306 schemas vs 8,909. Building llama.cpp with LLGuidance helps JSON Schema requests only; tool-call grammars from the chat parser stay GBNF. Qwen3.6 has a 248,320-token vocab, about 1.9x the benchmark's. Not yet measured on R9700s; the cost should show most on fast models (~10 ms per token) and speculative lanes, least on DDR5-hybrid Flash. - Engine: llama.cpp master d854d98bae (source read) - Models: Qwen3.6 tokenizer (248,320 vocab) - llama.cpp native grammar mask, MaskBench avg: 16.6 ms - LLGuidance mask, MaskBench avg: 0.064 ms - -bs throughput gain at 32 slots (RTX 5090): 48 % - Qwen3.6 vocab size: 248320 tokens Sources: [https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/sampling.cpp](https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/sampling.cpp), [https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/chat-auto-parser-generator.cpp](https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/chat-auto-parser-generator.cpp), [https://github.com/ggml-org/llama.cpp/pull/18137](https://github.com/ggml-org/llama.cpp/pull/18137), [https://github.com/ggml-org/llama.cpp/pull/17004](https://github.com/ggml-org/llama.cpp/pull/17004), [https://github.com/ggml-org/llama.cpp/pull/25532](https://github.com/ggml-org/llama.cpp/pull/25532), [https://github.com/ggml-org/llama.cpp/issues/27050](https://github.com/ggml-org/llama.cpp/issues/27050), [https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md](https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md), [https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json](https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json) Next: Measure plain vs tools (auto and required) vs JSON-schema decode speed on a fast single-R9700 model and on dual-R9700 Flash, with and without -bs. ## Speech-to-text next to a local LLM on R9700s: use whisper.cpp, and mind the endpoint 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-whisper-stt-rdna4 For voice input on RDNA4 cards, whisper.cpp (HIP or Vulkan) is the working path: a test on an R9700 ran the 11 s JFK sample with tiny.en in 206 ms, and an RX 9070 XT on Vulkan transcribed a 19.3-minute talk with large-v3-turbo in 48 s (about 24x real time, 1.5% WER). One RDNA4 tester reports the CTranslate2 ROCm forks behind faster-whisper produce wrong transcripts, so avoid that stack. The turbo model is 0.81 GiB at q8_0. whisper-server only serves /inference unless you pass --inference-path /v1/audio/transcriptions, which OpenAI-style agent clients expect. Cited results, not yet measured here. Strix Halo (gfx1151) on an 11 s clip: large-v3-turbo 0.84 s on Vulkan vs 0.63 s on ROCm/HIP. A single RX 9070 tester ran whisper.cpp on Vulkan and a ROCm diarizer on the same card at once without meaningful contention, and saw large-v3-turbo crash with a UTF-8 decode error (large-v3 was fine). whisper-server (master 231638a) turns flash attention on by default, picks the card with -dev N or WHISPER_ARG_DEVICE, and has built-in VAD. llama.cpp's own /v1/audio/transcriptions route (PR #21863) is for audio-capable LLMs, not Whisper models. Model sizes: large-v3-turbo f16 1.513 GiB, q8_0 0.814, q5_0 0.535; large-v3 f16 2.882, q5_0 1.007. On a dual-R9700 Flash setup those GiB compete with experts kept in VRAM, so the CPU lane is worth testing first. - Engine: whisper.cpp master 231638a (docs/source), llama.cpp PR #21863 - Models: Whisper large-v3 / large-v3-turbo ggml (f16, q8_0, q5_0) - R9700 tiny.en, 11 s clip, total: 205.72 ms - RX 9070 XT Vulkan large-v3-turbo, 19.3 min talk: 48 s - RX 9070 XT large-v3-turbo WER (TED talk): 1.5 % - Strix Halo large-v3-turbo 11 s clip, Vulkan vs HIP: 0.84 / 0.63 s - large-v3-turbo q8_0 file: 0.814 GiB Sources: [https://github.com/ggml-org/whisper.cpp/pull/3823](https://github.com/ggml-org/whisper.cpp/pull/3823), [https://pypi.org/project/babelscribe/](https://pypi.org/project/babelscribe/), [https://github.com/collabora/WhisperLive/issues/520](https://github.com/collabora/WhisperLive/issues/520), [https://github.com/maherr/onnxruntime-migraphx-rdna4/blob/main/witness/README.md](https://github.com/maherr/onnxruntime-migraphx-rdna4/blob/main/witness/README.md), [https://github.com/ggml-org/whisper.cpp/blob/master/examples/server/server.cpp](https://github.com/ggml-org/whisper.cpp/blob/master/examples/server/server.cpp), [https://github.com/ggml-org/llama.cpp/pull/21863](https://github.com/ggml-org/llama.cpp/pull/21863), [https://huggingface.co/ggerganov/whisper.cpp](https://huggingface.co/ggerganov/whisper.cpp) Next: Measure HIP, Vulkan and CPU-only Whisper lanes on a dual-R9700 box, and how much a loaded speech model slows Flash decode. ## Vision on Qwen3.8-Flash-Next with R9700s: the encoder is cheap, the image tokens are not 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-flash-next-vision-cost Flash-Next reads images through a separate 0.84 GiB projector (27-layer ViT, patch 16, merge 2, so one image token per 32x32 pixels). On an R9700 the GPU encoder takes 57 ms for a 260-token picture and 0.40 s for 972 tokens (3.13 s and 14.1 s on a CPU encoder). With llama.cpp's default cap of 4,096 tokens per image, a 1920x1080 screenshot becomes 2,040 tokens, so on a hybrid setup that prefills around 200 tok/s the image tokens, not the encoder, dominate time to first token. Capping images at 1,024 tokens halves that. Estimate, not a measurement: at a 205 tok/s hybrid prefill a default-cap 1080p screenshot needs about 0.9 s of encoding plus about 10 s of prefill; at a 1,024-token cap about 5.3 s. llama.cpp allows 8 to 4,096 tokens per Qwen-VL image by default and warns that grounding tasks want at least 1,024; Strata caps pictures at 1,024 by default. Strata's GPU reader reserves about 1.4 GiB of VRAM (fewer cached experts, a few percent slower text) and can start on demand and stop after 120 s idle. llama.cpp can pin the encoder to one card with --mmproj-device or keep it on the CPU with --no-mmproj-offload. On a DGX Spark, GGUF plus projector scored 58/60 on a 60-image synthetic set, the same as NVFP4; both misses were output-length truncations. - Model: Qwen3.8-Flash-Next GGUF + mmproj-BF16/F16 (Unsloth) - Engine: llama.cpp master d854d98bae (mtmd), Strata v0.2.6 docs - Projector size (F16): 0.842 GiB - R9700 GPU encode, 972-token picture: 0.4 s - CPU encode, 972-token picture (EPYC): 14.1 s - 1920x1080 screenshot at llama.cpp default cap: 2040 image tokens - Strata GPU reader VRAM reserve: 1.4 GiB Sources: [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/clip.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/clip.cpp), [https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp](https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp), [https://github.com/Niko1221/Strata/blob/main/docs/AMD_HIP.md](https://github.com/Niko1221/Strata/blob/main/docs/AMD_HIP.md), [https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md](https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md), [https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md), [https://0xbakeer.com/flash-next/vision](https://0xbakeer.com/flash-next/vision) Next: Measure time to first token, VRAM per card and text decode with the encoder on and off on two R9700s at Q5 and Q6, at 1,024 and 4,096 image-token caps. ## R9700 reference band on llama.cpp's community scoreboards (Llama 2 7B Q4_0) 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-r9700-scoreboard-band Published single-card R9700 results for the standard llama-bench test: Vulkan tg128 126-146 t/s (FA off) and 131-153 (FA on), pp512 5,000-5,900; ROCm/HIP 112.7 / 121.6 tg128 at build 8ea2902. A card well under this band points to a driver, power or PCIe problem, not the hardware. The ROCm scoreboard's R9700 row is stale (93.84 tg128 at bd4ef13); newer builds are about 20% faster. On one rig and one build (8ea2902), Vulkan decode was ~8% faster than HIP on this dense model (125.96 vs 112.72 FA off). R9700 Vulkan decode is about 76-89% of an RTX 3090 on Vulkan (171.61 FA on). The same-die RX 9070 XT posted 142.00 FA on (Vulkan, 2026-10-25), so the extra VRAM costs no decode speed. Splitting a model that fits one card across two hurts: a 2x 7900 XTX owner saw tg128 drop from 170.69 to 149.04 with layer split and to 56.35 with Vulkan tensor split. - Model: llama-2-7b.Q4_0.gguf (3.56 GiB) - Engine: llama.cpp Vulkan and HIP, various builds - Command: llama-bench -ngl 99 -fa 0,1 -sm none (one GPU) - R9700 Vulkan tg128, FA on (best published): 152.7 tok/s - R9700 Vulkan tg128, FA on (8ea2902): 131.19 tok/s - R9700 ROCm tg128, FA on (8ea2902): 121.62 tok/s - R9700 ROCm scoreboard row tg128, FA off (stale bd4ef13): 93.84 tok/s - RTX 3090 Vulkan tg128, FA on: 171.61 tok/s Sources: [https://github.com/ggml-org/llama.cpp/discussions/10879](https://github.com/ggml-org/llama.cpp/discussions/10879), [https://github.com/ggml-org/llama.cpp/discussions/15021](https://github.com/ggml-org/llama.cpp/discussions/15021) Next: Measure both local R9700s with the same test (Vulkan and HIP, each card alone) and check them against this band before further dual-card Flash tuning. ## R9700 'regressions' that are really the driver stack: AMDVLK vs RADV, Mesa version, and a silent SMU hang 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-r9700-driver-stack-traps A review of public R9700 (gfx1201) reports found that the biggest swings come from the driver stack, not model flags. AMDVLK's Vulkan driver can silently take priority over RADV: one reported -39% MoE prefill regression was really AMDVLK, and on an R9700 AMDVLK's dense prefill was about 4x slower than RADV (207 vs 823 tok/s on a 27B). One tester saw Mesa 25.3.6 give +48% prefill on a 35B MoE over Ubuntu's 25.2.8. Two alarming R9700 Vulkan reports (an 8x decode drop tied to one llama.cpp PR, and a Qwen3.6-35B decode cliff after ~32 tokens) were never confirmed. The real safety gap: a hung power-management controller can leave a card unreadable by every sensor while it can still draw 300 W. Check the llama-bench device line: RADV prints '(RADV GFX1201) (radv)' with shared memory 65536, AMDVLK prints '(AMD open-source driver)' with 32768; pin AMD_VULKAN_ICD=RADV. The dmesg line 'SMU driver if version not matched' (driver 0x2E vs card 0x32/0x33) shows up on every R9700 and AMD staff call it harmless. AMD says MES firmware 0x8B fixes a stuck-busy idle state (100% busy, ~100 W at idle), but one dual-R9700 owner saw about 10.8% lower decode on that package. On a three-card host, one card's SMU hung while idle and only a reboot cleared it; health checks that skip unreadable sensors go blind, so treat an unreadable sensor as an alarm. Docked or external cards can pick the wrong runtime power method; amdgpu.runpm=0 fixed one case. Our own measured single-card Qwen3.6-35B Vulkan decode (~101 tok/s) shows the decode-cliff report does not apply to our current build. Mesa and kernel deltas are single-tester results. - Model: various (per source) - Quant: per source - Engine: llama.cpp Vulkan (RADV / AMDVLK), amdgpu kernel driver - Engine version: per source - GPUs: Radeon AI PRO R9700 (gfx1201) - AMDVLK dense 27B prefill on R9700: 207 tok/s - RADV dense 27B prefill on R9700: 823 tok/s - Mesa 25.3.6 vs 25.2.8, 35B MoE pp2048 (one tester): 48 % - Decode drop after MES 0x8B package (one dual-R9700 report): 10.8 % - Power a sensor-blind card can still draw: 300 W Sources: [https://github.com/ggml-org/llama.cpp/discussions/21043](https://github.com/ggml-org/llama.cpp/discussions/21043), [https://github.com/ggml-org/llama.cpp/issues/22375](https://github.com/ggml-org/llama.cpp/issues/22375), [https://github.com/ggml-org/llama.cpp/issues/26679](https://github.com/ggml-org/llama.cpp/issues/26679), [https://github.com/ggml-org/llama.cpp/issues/26795](https://github.com/ggml-org/llama.cpp/issues/26795), [https://github.com/ROCm/ROCm/issues/6101](https://github.com/ROCm/ROCm/issues/6101) Next: Fingerprint the stack (kernel, Mesa, ICD, firmware, llama.cpp build) on both desktops and record it with every benchmark; A/B RADV vs AMDVLK only if AMDVLK is installed. ## vLLM expert parallel on 2x R9700: no extra PCIe traffic, but ~25% routing imbalance at batch 1 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-ep-vs-tp-moe-imbalance On one box, vLLM's --enable-expert-parallel with TP=2 and DP=1 still uses all-reduce, not all-to-all, so it adds no PCIe traffic; it only changes how experts are placed. A routing simulation for Qwen3.8-Flash-Next (512 experts, top-10) shows the busier card reads 1.25x the routed-expert bytes of plain sharded TP at batch 1 (p95 1.60x), shrinking to 1.04x at 32 concurrent streams. On a 4-card setup it is 1.68x at batch 1. For single-stream chat, keep plain -tp 2; try EP only for many-slot batch serving, and never DP+EP over PCIe. Sharded TP gives each card half of every active expert, so the work is equal every token. EP gives each card full copies of the experts it owns, and the all-reduce waits for the card that drew more of the token's 10 experts. Analytic check: E[max(X,10-X)]/5 with X~Binomial(10,0.5) = 1.246. Round-robin placement and EPLB do not help at batch 1 because the imbalance is per-token chance, not a hot-expert bias; EPLB also costs VRAM. Estimate: routed experts are roughly 40% of Flash-Next's per-token weight reads by parameter count, so the single-stream penalty would be about 10% on 2 cards and about 27% on 4, unless the half-width sharded expert kernel is much slower on RDNA4. This is a model, not a measurement; a TP vs TP+EP benchmark is planned. - Model: Qwen3.8-Flash-Next (config), 256-expert top-8 class for comparison - Engine: vLLM (behavior per docs and AMD blog) - GPUs: 2x Radeon AI PRO R9700 (also modeled 4x) - Flags: -tp 2 vs -tp 2 --enable-expert-parallel - EP/TP slowest-card expert bytes, 2 GPUs, batch 1: 1.25 x - same, batch 32: 1.04 x - same, 4 GPUs, batch 1: 1.68 x - AMD MI300X: TP8+EP vs DP8+EP throughput at concurrency 64 (DeepSeek-R1): 52 % higher Sources: [https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html](https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html), [https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/), [https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py), [https://0xsero.github.io/blackwell-gpu-wiki/interconnect/moe-parallelism/](https://0xsero.github.io/blackwell-gpu-wiki/interconnect/moe-parallelism/) Next: Benchmark plain TP2 vs TP2+EP in vLLM at 1, 4, 16 and 32 concurrent streams on a 4-bit-or-better MoE that fits two R9700s. ## Estimating MoE active parameters and GGUF sizes from Hugging Face: the k/E shortcut is ~40% low, and MTP files fake a Q4 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-new-models-moe-active-params-audit An audit of the 84 models on our /new-models page found that the common shortcut active = total x (experts per token / experts) underestimates active parameters by 33-43% on four models with published counts, while subtracting only the routed expert weights lands within +1 to +25%. It also found that configs nesting MoE keys under text_config made 9 of 30 MoE models (including Qwen3.8-Flash-Next, GLM-5.3-Flash and DeepSeek-V4.1-Flash) display as fully active. Shape method: active = total - 3 x hidden x moe_intermediate x experts x MoE layers x (1 - k/E), counting MTP layers as MoE layers. Checks against official active counts: DeepSeek-V3 37B (shortcut 21.4, shape 40.1), Qwen3-30B-A3B 3.3B (1.91 vs 3.35), Qwen3-235B-A22B 22B (14.7 vs 22.2), GLM-4.5-Air 12B (6.9 vs 15.0; its official count appears to leave out MTP and embeddings). For file sizes, the Hugging Face model API returns per-file sizes only when called with ?blobs=true. Split GGUFs must have their shards summed per quant folder, and helper files must be skipped: Qwen3.8-Flash-Next's GGUF repo carries a 3.9 GiB MTP draft head named mtp-...-Q4_K_M.gguf that a filename match reads as the model's Q4_K_M (the real UD-Q4_K_XL is 103.7 GiB). Checkpoints with FP4-packed experts also undercount parameters in the safetensors metadata: MiniMax-M3.1-Flash estimates to 124.5 GiB at Q4 but the measured UD-Q4_K_XL is 232.3 GiB. With measured sizes and a rough rule (2x 32 GB VRAM less ~6 GiB for KV/compute, experts in DDR5 with 16 GB left for the OS), 5 Flash-class MoE models fit two R9700s plus 128 GB DDR5 as a hybrid and 5 more need 256 GB; dense models of that size are not worth running from DDR5. A corrected /new-models data script is prepared. - Engine: Hugging Face Hub API + config.json - GPUs: fit tiers computed for 2x Radeon AI PRO R9700 (32 GB each) + DDR5 - k/E shortcut error vs official active params (4 models): -33 to -43 % - Shape-method error vs official active params (4 models): +1 to +25 % - MoE models shown as fully active (nested text_config): 9 of 30 models - Qwen3.8-Flash-Next UD-Q4_K_XL, shards summed: 103.7 GiB - MTP draft head misread as Q4_K_M: 3.9 GiB - MiniMax-M3.1-Flash Q4: estimate vs measured: 124.5 vs 232.3 GiB Sources: [https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json](https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json), [https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/config.json](https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/config.json), [https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/config.json](https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/config.json), [https://huggingface.co/zai-org/GLM-4.5-Air/blob/main/config.json](https://huggingface.co/zai-org/GLM-4.5-Air/blob/main/config.json), [https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json), [https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF?blobs=true](https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF?blobs=true), [https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF](https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF) Next: Ship the corrected script, then show measured vs estimated sizes and two DDR5-hybrid fit columns (128 GB, 256 GB) on /new-models. ## Measured: Qwen 3.8 Flash quants on dual R9700, quality vs speed 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-measured-quant-quality-speed Measured on the local desktop (2x R9700, 262K f16 KV, reference Q8_0). Strata 0.2.3.1 UD-Q5_K_XL decodes 58 tok/s short / 52 at 36K with 2,192 tok/s prefill and 24/26 tool calls, so it stays the default. llama.cpp hybrid decode was only 17-22 tok/s (UD-Q5 17.6/17.1/14.2 at 1K/36K/163K; UD-Q4 21.2/19.9/16.0); our earlier bandwidth-model estimate of ~34-45 tok/s was far too optimistic. UD-Q5 drifts least from Q8_0 (KLD 0.036 text / 0.019 code vs 0.060/0.034 for UD-Q4). llama.cpp (decode ~1K/36K/163K; prefill 36K/163K; KLD text/code; tools): UD-Q5_K_XL 147 GB 17.6/17.1/14.2, 226/195, 0.036/0.019, top-1 92.2%/96.3%, 24/26. UD-Q4_K_XL 104 GB 21.2/19.9/16.0, 314/273, 0.060/0.034, top-1 90.1%/95.1%, 24/26. AP-Q5_K_M 104 GB 20.1/20.3/16.5, 330/301, 0.065/0.033, 24/26. AP-Q4_K_XL 94 GB 22.2/21.8/17.6, 416/359, 0.070/0.038, 24/26. Strata UD-Q5: 25/26 tools with thinking on; greedy exact match vs Q8_0 37.5% (llama.cpp UD-Q5 43.8%). vLLM community MXFP4: best tools 25/26 but lowest exact match 18.8%; KLD run failed, not speed-benched. KLD = llama-perplexity KL vs Q8_0 on held-out post-Oct-1 corpus (152K text / 51K code tokens); 26-case tool-call eval. Q3 estimate: not applicable for this project. ## Measured: Strata mmap-experts on vs off 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-strata-mmap-experts-ab On the local desktop with Strata UD-Q5_K_XL, mmap-experts=on left decode unchanged, halved load time (~27 s vs 48-61 s) and cut process RAM from 109 to 19 GB, but prefill dropped 37% (1,377 vs 2,189 tok/s at 36K). Use it when RAM or load time matters more than long-prompt ingest. ## llama.cpp --fit on dual R9700 + DDR5: what it changes silently, and how to A/B it 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-llamacpp-fit-autoplace llama.cpp's automatic placement (--fit) is on by default. It only adjusts arguments you left unset: set any of -ngl, -ts or -ot and placement is entirely yours. When a model does not fit it shrinks the context first (down to 4096 by default) before moving tensors to RAM, so a launcher without an explicit -c can quietly lose its long context. For MoE models it fills the last GPU with dense-only layers (experts in system RAM), then converts layers to full GPU layers front-to-back, letting parts of a layer overflow to the next GPU or RAM. Default margin is 1024 MiB per device and can be set per device (-fitt 512,1024). llama-bench now accepts -fitt and -fitc, so fitted and hand-written placements can be compared in one tool; llama-fit-params prints the chosen -c/-ngl/-ts/-ot. It is not available with -sm tensor. Integrated GPUs are eligible targets, so a desktop CPU's iGPU that shows up as a ROCm or Vulkan device should be excluded with --device. A user report on two Arc Pro B70s found an explicit -ts 1,1 silently spilling to one card plus RAM with fit left on (about 10x slower) until -fit off. Upstream's own CUDA table shows a second GPU can lower prefill while the model is still RAM-bound (Qwen3 Next q8_0 pp4096 904 to 574 t/s, tg 24.3 to 28.3). Fit measures memory before warm-up, so leave headroom for long prompts. - Engine: llama.cpp - Engine version: master 6050b467 (2026-10-25) - GPUs: 2x Radeon AI PRO R9700 (applies to any multi-GPU + RAM setup) - Default --fit-ctx floor: 4096 tokens - Default --fit-target margin per device: 1024 MiB - Upstream 2x vs 1x RTX 4090 pp4096, Qwen3 Next q8_0 (RAM-bound): 574 vs 904 t/s Sources: [https://github.com/ggml-org/llama.cpp/discussions/18049](https://github.com/ggml-org/llama.cpp/discussions/18049), [https://github.com/ggml-org/llama.cpp/pull/18679](https://github.com/ggml-org/llama.cpp/pull/18679), [https://github.com/ggml-org/llama.cpp/issues/22592](https://github.com/ggml-org/llama.cpp/issues/22592), [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md) Next: Measure fitted vs hand-tuned placement for Flash-class Q4/Q5/Q6 on two R9700s with DDR5 experts, at 0 and 32K depth. ## HIP graphs on R9700 in llama.cpp: the real off switch, the new knobs, and a build trap 2026-10-25 · Dual R9700 speed · Progress · https://amdai.net/lab-log/#2026-10-25-hip-graphs-r9700 llama.cpp has captured decode as HIP graphs by default since April 2026 (upstream cites +3-14% decode on other AMD GPUs). The working runtime off switch is GGML_CUDA_DISABLE_GRAPHS; a dual-gfx1201 bug report concluded env vars can't disable graphs, but it tried GGML_CUDA_NO_GRAPHS, which the code never reads. Two new knobs are worth an A/B: GGML_CUDA_GRAPH_CHUNK_SIZE and GGML_CUDA_GRAPH_OPT=1. From llama.cpp master source (2026-10-25): the only graph compatibility check left is the MUL_MAT_ID sync fallback, and batch-1 quantized MoE decode takes the graph-safe path. Each GPU split gets its own graph, which needs two identical calls to warm up and resets on any shape change, so with experts in DDR5 a GPU's work is cut into many small graphs and the gain there is unmeasured. gfx1201 has no VMM, so llama.cpp uses its legacy pool, and every pool flush forces all graphs to be captured again. GGML_CUDA_GRAPH_CHUNK_SIZE (default 512 nodes) landed 2026-10-23 with +1-2% decode on CUDA; GGML_CUDA_GRAPH_OPT=1 (concurrent streams) has been allowed per split on multi-GPU since 2026-09-03, +3.4-3.7% MoE decode on CUDA. GGML_CUDA_GRAPH_DEBUG=1 with verbose logging prints warm-up, reset and reuse events. Build trap: CUDA/HIP builds fail between the chunked-graphs merge and its fix (commit 9ff0d7e90f), so don't pin a commit in that window. Graphs only exist in the HIP/CUDA backend; Vulkan launchers are unaffected. No R9700 measurement yet; an A/B is planned. - Engine: llama.cpp (source read) - Engine version: master 6050b467 (2026-10-25) - GPUs: Radeon AI PRO R9700 (gfx1201) - Flags: GGML_HIP_GRAPHS=ON (default) - Upstream decode gain from HIP graphs, qwen3moe 30B-A3B Q4_K_M (gfx908): 12 % - Upstream decode gain from HIP graphs, gfx1100 range: 3-7 % - GRAPH_OPT=1 MoE decode gain, 2 GPUs (CUDA): 3.4-3.7 % - Graph chunking decode gain (CUDA): 1-2 % Sources: [https://github.com/ggml-org/llama.cpp/pull/22254](https://github.com/ggml-org/llama.cpp/pull/22254), [https://github.com/ggml-org/llama.cpp/issues/23579](https://github.com/ggml-org/llama.cpp/issues/23579), [https://github.com/ggml-org/llama.cpp/pull/30804](https://github.com/ggml-org/llama.cpp/pull/30804), [https://github.com/ggml-org/llama.cpp/pull/31053](https://github.com/ggml-org/llama.cpp/pull/31053), [https://github.com/ggml-org/llama.cpp/pull/28198](https://github.com/ggml-org/llama.cpp/pull/28198), [https://github.com/ggml-org/llama.cpp/pull/30327](https://github.com/ggml-org/llama.cpp/pull/30327), [https://github.com/ggml-org/llama.cpp/issues/25082](https://github.com/ggml-org/llama.cpp/issues/25082), [https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-cuda/ggml-cuda.cu](https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-cuda/ggml-cuda.cu) Next: A/B on R9700: default vs graphs off vs chunk 0/128 vs GRAPH_OPT=1, single-GPU MoE and the dual-R9700 Flash hybrid, with graph event counts. ## Ollama, LM Studio and Lemonade on two R9700s: what they do with the second card, and five traps 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-easy-apps-dual-r9700-traps From the apps' docs and bug trackers: Ollama puts a model that fits on one GPU onto one GPU only and defaults to a 4096-token context. A Ryzen desktop iGPU (gfx1036) can show up as an extra ROCm device, and llama.cpp-based apps that auto-split onto it segfault; list only the cards with --device ROCm0,ROCm1. Ollama MoE models crash on mixed R9700 + other Radeon pairs because the rocBLAS fix is not in any ROCm 7.2.x release, which Ollama bundles. Ollama's Linux ROCm v7 list includes the R9700, but its Windows ROCm list has no RDNA4 cards, so Windows R9700 users run on Ollama's Vulkan backend. The mixed-card MoE crash ('no kernel image is available') was reproduced on Ollama 0.32.14 through 0.34.1 with R9700 + RX 9060 XT and R9700 + RX 7900 XTX; two identical R9700s are not affected. The rocBLAS fix (rocm-libraries #4781) merged 2026-03-02. On kernels without the gfx12 MES idle-power fix (in 6.12+ stable), ROCm runtimes can keep a card 100% busy at up to ~300 W between prompts until the model unloads (reported on dual R9700 in LM Studio). LM Studio's ROCm runtime has an open report of a 27B dense model hanging at 99% load on an R9700 where the Vulkan runtime loads it. llama.cpp's own multi-GPU doc marks -sm row deprecated and -sm tensor experimental with no performance guarantees outside CUDA, so layer split remains the safe default. Not measured on our hardware yet. - Engine: Ollama / LM Studio / Lemonade (llama.cpp-based) - Engine version: per source - GPUs: 2x Radeon AI PRO R9700 - Ollama default context: 4096 tokens - Ollama default parallel requests per model: 1 - Kernel with gfx12 idle-power MES fix: 6.12+ - Reported idle draw with the bug (per card): 300 W Sources: [https://docs.ollama.com/gpu](https://docs.ollama.com/gpu), [https://docs.ollama.com/faq](https://docs.ollama.com/faq), [https://github.com/ollama/ollama/issues/18162](https://github.com/ollama/ollama/issues/18162), [https://github.com/lemonade-sdk/llamacpp-rocm/issues/96](https://github.com/lemonade-sdk/llamacpp-rocm/issues/96), [https://lemonade-server.ai/docs/dev/backends-reference/](https://lemonade-server.ai/docs/dev/backends-reference/), [https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1991](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1991), [https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1916](https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1916), [https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md) Next: Check whether the iGPU is visible as a device on our dual-R9700 desktop, and measure Ollama against raw llama.cpp on the same 35B Q5 file. ## Short-context KL can't settle Q4 vs Q5 vs Q6 for Flash models at 128K+ 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-longctx-quant-evidence Published studies show the 4-bit penalty grows with context. 8-bit stays near-lossless, but 4-bit methods lose up to 16% on RULER at 64K+ inputs and decline further past 75K. The public long-context evidence for Qwen3.8-Flash-Next at Q4 vs Q6 is only 4 items each, so our UD-Q5_K_XL sweet-spot pick is unproven past short context. arXiv 2505.20276 tested FP8, int8, AWQ/GPTQ int4 and nf4 on Llama 3.1 and Qwen 2.5. The average 8-bit drop was about 0.8%, the worst 4-bit cell dropped 59%, and the authors suggest RoPE rounding error accumulating over long context as a cause. These are not GGUF k-quants. Red Hat measured INT W4A16 accuracy recovery of 85-88% at 128K on RULER, against an already-low baseline. One 2026 Qwen3-8B study at 4K found Q8_0 through Q4_K_M within noise of BF16 and first damage at Q3_K_S. In the same study, distance hurt more than precision: the BF16 two-fact join fell from 0.538 to 0.066 between 4K and 23K. For Flash-Next, Unsloth's short-context mean KLD is 0.0469 (UD-Q4_K_XL) and 0.0304 (UD-Q5_K_XL), with top-1 agreement of 92.26%, 93.68% and 94.09% (Q6). One 2x RTX 3090 review at 131K scored Q4 and Q6 4/4 each on long-context items. Flash-Next mixes Gated DeltaNet layers with QSA sparse attention (a 2048-token budget), so results from dense models may not transfer. We built a paired 160-item probe (literal needle, near-duplicate keys, 3-hop chain, no-word-overlap join, count) at 16K-245K to measure it on 2x R9700. - Worst 4-bit drop on RULER/OneRULER, 64K+ inputs (published): 16 % - Average 8-bit accuracy drop, long-context study: 0.8 % - Flash-Next UD-Q4_K_XL mean KLD (Unsloth): 0.0469 - Flash-Next UD-Q5_K_XL mean KLD (Unsloth): 0.0304 Sources: [https://arxiv.org/html/2505.20276v3](https://arxiv.org/html/2505.20276v3), [https://developers.redhat.com/articles/2024/02/03/how-well-do-quantized-models-handle-long-context-tasks](https://developers.redhat.com/articles/2024/02/03/how-well-do-quantized-models-handle-long-context-tasks), [https://egordmitriev.dev/blog/2026-08-12-does-precision-loss-break-retrieval-or-does-distance](https://egordmitriev.dev/blog/2026-08-12-does-precision-loss-break-retrieval-or-does-distance), [https://unsloth.ai/docs/models/qwen3.8-next](https://unsloth.ai/docs/models/qwen3.8-next), [https://peaster.io/articles/local-LLMs/qwen-3-8-flash-next-first-impressions](https://peaster.io/articles/local-LLMs/qwen-3-8-flash-next-first-impressions) Next: Run the paired probe on UD-Q6, UD-Q5 and UD-Q4 with the same llama.cpp build and f16 KV cache, and publish the per-depth accuracy. ## Two-box R9700 with llama.cpp RPC: the official Docker images can't do it, and both boxes need the identical build 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-llama-rpc-two-box-build-parity Checked llama.cpp master (2026-10-25): the official ROCm and Vulkan Docker images are built without GGML_RPC, which defaults to OFF, so they contain no RPC backend or ggml-rpc-server. RPC also needs byte-identical builds on both machines: the protocol sends raw op enum values, mismatched builds have crashed after a successful handshake, and the protocol version (now 8.0.1, 102 ops) is bumped on every op change. Build one pinned image with RPC on and run it on both boxes. The upstream ROCm Dockerfile pins ROCm 7.2.1 and builds from rocm/dev-ubuntu-24.04:-complete. The newest -complete tag is 7.2.4; ROCm 7.14, 10.0 and 10.1 bases are published as -full, so changing only the version points at a tag that doesn't exist. It also compiles 11 GPU targets; gfx1201 alone is enough for R9700. ggml-rpc-server binds to 127.0.0.1 unless given -H and has no authentication, so bind it only to the direct 10GbE/USB4 link address. Its -c tensor cache lives in LLAMA_CACHE (default ~/.cache/llama.cpp/rpc); inside a container that must be a persistent volume or every restart re-sends the worker's weights. RDMA is only negotiated with RoCEv2 NICs; ordinary onboard 10GbE stays on TCP. Estimate, not measured: re-sending ~48 GiB (half of a ~97 GiB GPU-side Q5 Flash pack) at ~1.1 GiB/s practical 10GbE is ~45 s per load without the cache. Any local patch (for example a carried llama.cpp PR) must be identical on both sides too. - Engine: llama.cpp RPC - Engine version: master 2280f0f (2026-10-25) - GPUs: planned 2x2 Radeon AI PRO R9700 over 10GbE - RPC protocol version on master: 8.0.1 - ggml op count the protocol is pinned to: 102 - GPU targets in upstream ROCm image: 11 Sources: [https://github.com/ggml-org/llama.cpp/blob/master/.devops/rocm.Dockerfile](https://github.com/ggml-org/llama.cpp/blob/master/.devops/rocm.Dockerfile), [https://github.com/ggml-org/llama.cpp/blob/master/ggml/CMakeLists.txt](https://github.com/ggml-org/llama.cpp/blob/master/ggml/CMakeLists.txt), [https://github.com/ggml-org/llama.cpp/blob/master/ggml/include/ggml-rpc.h](https://github.com/ggml-org/llama.cpp/blob/master/ggml/include/ggml-rpc.h), [https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), [https://github.com/ggml-org/llama.cpp/issues/20259](https://github.com/ggml-org/llama.cpp/issues/20259), [https://github.com/ggml-org/llama.cpp/pull/20330](https://github.com/ggml-org/llama.cpp/pull/20330), [https://hub.docker.com/r/rocm/dev-ubuntu-24.04/tags](https://hub.docker.com/r/rocm/dev-ubuntu-24.04/tags) Next: Build a pinned gfx1201 image with RPC on, confirm it matches native speed on one box, then measure RPC loopback overhead and cold vs warm load time before the second desktop joins. ## vLLM on R9700: tuned MoE configs named 'AI_PRO_R9700' never load, and Flash-Next has no tuned config 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-vllm-r9700-moe-config-names vLLM looks up fused-MoE tile configs by exact filename. Since PR #38086 (merged 2026-04-02) it names the R9700 'AMD_Radeon_R9700' (PCI id 0x7551), so config files still circulating as 'AMD_Radeon_AI_PRO_R9700' are silently ignored. Upstream ships only three R9700 MoE configs and none for Qwen3.8-Flash-Next's 512-expert shapes, and the stock tuner script rejects the Flash-Next architecture. Rename old files (device_name=AMD_Radeon_AI_PRO_R9700 -> AMD_Radeon_R9700) and confirm the server log says 'Using configuration from' instead of 'Using default MoE config'. Flash-Next keys (int4_w4a16, the group size is not part of the key): tensor parallel 2 -> E=512,N=320; TP2 with expert parallel -> E=256,N=640; TP4 -> E=512,N=160; TP4 with EP -> E=128,N=640. On gfx1201, compressed-tensors W4A16 int packs reach the Triton MoE kernel that uses these files (the RDNA3 HIP kernel is gfx1100-only and Marlin is CUDA-only). benchmark_moe.py has no Qwen4Exp branch and stops with 'has no MoE experts'; a small wrapper that reads num_experts/moe_intermediate_size from text_config fixes it, and tuning needs only config.json. Payoff on other R9700 shapes from the cited PRs: 1.05-1.46x per MoE layer (int4, M=1-32), -4.2% ms/token single stream and +13.5% tok/s at 4 concurrent; FP8 TPOT -8% to -24%. Flash-Next numbers are not measured yet. - Model: Qwen3.8-Flash-Next (W4A16 int4 packs) - Engine: vLLM main (source read 2026-10-25) - GPUs: Radeon AI PRO R9700 (gfx1201) - Upstream R9700 fused-MoE config files: 3 files - Per-layer speed-up, tuned R9700 int4 MoE (other model): 1.05-1.46 x - TPOT change, tuned R9700 FP8 MoE (other model): -8 to -24 % Sources: [https://github.com/vllm-project/vllm/pull/38086](https://github.com/vllm-project/vllm/pull/38086), [https://github.com/vllm-project/vllm/pull/36659](https://github.com/vllm-project/vllm/pull/36659), [https://github.com/vllm-project/vllm/pull/60893](https://github.com/vllm-project/vllm/pull/60893), [https://github.com/vllm-project/vllm/issues/50436](https://github.com/vllm-project/vllm/issues/50436), [https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py), [https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py) Next: Tune the four Flash-Next keys on R9700 and measure default vs tuned per-layer time. ## ROCm's 'pinned' host memory is pageable, and transparent huge pages can make a big-MoE start take 5 minutes instead of 35 seconds 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-rocm-pinned-memory-thp On Linux, ROCm implements hipHostMalloc as an anonymous mmap with MADV_HUGEPAGE plus a KFD userptr mapping, so the pages are not locked (VmLck and VmPin stay 0). In a reported 2x RX 7900 XTX test, page-locking a 28.47 GiB expert cache took a median 300.5 s with THP at the default madvise setting versus 34.0 s with THP disabled for the engine process (4 interleaved starts each), with decode and prompt speed unchanged. For anyone keeping 30-100 GiB of MoE experts in DDR5 next to Radeon GPUs, starting the engine with THP off is a free fix for slow, variable starts. The slow starts showed thousands of EAGAIN retries of AMDKFD_IOC_ALLOC_MEMORY_OF_GPU (about 3,770 in one 198.6 s start). The reporter's unconfirmed explanation is that 2 MB huge-page faults trigger compaction, which invalidates the HMM walk so ROCr retries the whole range. Workaround: a tiny launcher that calls prctl(PR_SET_THP_DISABLE) and execs the server (no root); allocating in 2 GiB pieces was also fast in the good state (3.6 s). Separately, because RDNA GPUs have no XNACK, reclaim or compaction of these pageable pages evicts all of the process's GPU queues; an open Strata PR that mlocks every GPU-mapped host range reported no stalls in 30 minutes of heavy memory pressure (twice) versus stalls after 103 s and 63 s without it, and needs LimitMEMLOCK=infinity. The older HSA_USERPTR_FOR_PAGED_MEM=0 workaround avoids the stall but stops ROCr pinning the arena and roughly halved prompt reads in that test. AMD's IOMMU guidance is to keep the default remapping mode on Radeon GPUs that use peer-to-peer; iommu=pt is recommended for Instinct and for Radeon without P2P. THP itself is a small speed lever for hybrid inference; the risk it brings here is start time and stalls. - Engine: Strata (HIP build) - Engine version: 0.1.44.1 (start test), 0.1.48.1 + patch (stall test) - GPUs: 2x RX 7900 XTX (start-time test); 1x RX 7900 XTX (stall test) - Flags: THP enabled=madvise, defrag=madvise - Rocm: 7.2.4 (start test) - Expert cache page-lock, THP default (median of 4): 300.5 s - Expert cache page-lock, THP off for the process (median of 4): 34 s - Load to ready, THP default (median of 4): 366.5 s - Same allocation in 15 x 2 GiB pieces (fast state): 3.6 s Sources: [https://github.com/Niko1221/Strata/issues/2850](https://github.com/Niko1221/Strata/issues/2850), [https://github.com/Niko1221/Strata/pull/3913](https://github.com/Niko1221/Strata/pull/3913), [https://man7.org/linux/man-pages/man2/PR_SET_THP_DISABLE.2const.html](https://man7.org/linux/man-pages/man2/PR_SET_THP_DISABLE.2const.html), [https://rocm.docs.amd.com/en/docs-6.3.2/conceptual/iommu.html](https://rocm.docs.amd.com/en/docs-6.3.2/conceptual/iommu.html), [https://lists.openwall.net/linux-kernel/2026/06/25/875](https://lists.openwall.net/linux-kernel/2026/06/25/875) Next: A/B the same THP-off launcher on dual R9700 for the Q6 and Q5 Flash hybrid lanes (start time, decode and prefill), then a memlock pressure test. ## R9700 has no FP4 hardware, and vLLM 0.28+ picks a slower FP8 kernel on it by default; one env var gets ~10% back 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-r9700-fp8-fp4-kernel-paths RDNA4 (gfx1201) has matrix instructions for FP16/BF16 (191 TFLOPS), FP8 and INT8 (383) and INT4 (766 TOPS), but none for FP4, so MXFP4/NVFP4 models are unpacked to 16-bit inside the kernel: you get the memory savings, not 4-bit math. On vLLM 0.28 and later, per-token FP8 checkpoints on an R9700 select RowWiseTorchFP8ScaledMMLinearKernel; setting VLLM_DISABLED_KERNELS=RowWiseTorchFP8ScaledMMLinearKernel switches to ChannelWise and, in a reported single-R9700 test, raised decode from 34.71 to 38.48 tok/s (+10.9%) and prefill from 4,152 to 4,531 tok/s (+9.1%) with GSM8K unchanged within noise. Measurements are from vLLM issue #57838 (gemma-4-12B FP8 compressed-tensors, AMD rocm/vllm ROCm 10.1 image, vLLM 0.29.1.dev, fp8 KV, single stream, eight boots 38.43-38.54). The fix is not merged as of 2026-10-25; open PR #61091 adds a native Triton per-token FP8 backend (+14-17% per-request output rate vs RowWise at TP1, GSM8K -0.3 pt). Block-scaled FP8 checkpoints (128x128 weight blocks) are not affected by this kernel choice; they use AITER Triton blockscale for shapes in the gfx1201 tuned list (PR #43615) and generic Triton otherwise. MXFP4 MoE on RDNA4 runs through Triton's tl.dot_scaled decomposed onto 16-bit WMMA (PR #37826). llama.cpp's HIP backend runs GGUF k-quants on INT8 WMMA (PR #17156). No mainstream LLM engine uses the INT4 matrix rate yet; it needs W4A4 rotation and calibration, not a normal Q4 file. Separately, AITER unified attention on gfx12 gave silently wrong image answers (text fine) in the same issue thread; use --attention-backend TRITON_ATTN for vision models. - Quant: FP8 W8A8 per-token; MXFP4; GGUF k-quants (all >= 4-bit) - Engine: vLLM (ROCm) and llama.cpp HIP, cited - Engine version: vLLM 0.28-0.29.1.dev - GPUs: Radeon AI PRO R9700 (gfx1201) - Decode, RowWise default: 34.71 tok/s - Decode, RowWise disabled (ChannelWise): 38.48 tok/s - Prefill, RowWise default: 4152 tok/s - Prefill, RowWise disabled: 4531 tok/s - FP8 matrix peak (dense): 383 TFLOPS - FP4 matrix peak: 0 TFLOPS (no FP4 instruction) Sources: [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html), [https://rocm.docs.amd.com/projects/rocWMMA/en/develop/api-reference/data-type-support.html](https://rocm.docs.amd.com/projects/rocWMMA/en/develop/api-reference/data-type-support.html), [https://github.com/vllm-project/vllm/issues/57838](https://github.com/vllm-project/vllm/issues/57838), [https://github.com/vllm-project/vllm/pull/61091](https://github.com/vllm-project/vllm/pull/61091), [https://github.com/vllm-project/vllm/pull/43615](https://github.com/vllm-project/vllm/pull/43615), [https://github.com/vllm-project/vllm/pull/37826](https://github.com/vllm-project/vllm/pull/37826), [https://github.com/ggml-org/llama.cpp/pull/17156](https://github.com/ggml-org/llama.cpp/pull/17156) Next: Check the startup log of the dual-R9700 vLLM Flash lane for which FP8 and MXFP4 kernels it selects, and A/B any slow pick. ## llama.cpp HIP prefills Q6_K at about half speed on RDNA4, which makes UD-Q6 Flash packs ~1.6x heavier than Q5 (fix in an open PR) 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-rdna4-q6k-hip-prefill On RDNA4 (gfx1201, same chip as the R9700), llama.cpp's HIP matrix kernel for Q6_K runs at 36.6 TFLOPS versus 75-80 for Q4_K, Q5_K and Q8_0. Open PR #25940 lifts it to 69.7 (1.9x), with prefill gains of +68-82% on a pure Q6_K 8B model and +28-37% on R9700 Q6_K models; token generation is unchanged. For Qwen3.8-Flash-Next, 80% of UD-Q6_K_XL's expert-matmul time is Q6_K, so by our estimate its GPU expert prefill work is ~1.58x that of UD-Q4_K_XL on HIP master (UD-Q5_K_XL 0.94, AP-Q5_K_M 0.93), dropping to ~0.97 with the PR. Kernel numbers are test-backend-ops MUL_MAT at n=512 on an RX 9070 XT from the PR author; the author's 2026-09-09 re-run on master shows a pure Q6_K Llama 8B at pp2048 2,320 -> 3,891 t/s (ub 512) and 2,425 -> 4,422 (ub 2048), Q4_K_M and Q5_K_M +10% (their Q6_K tensors), Q8_0, Q4_0, IQ4_NL and IQ4_XS flat. An R9700 user on ROCm 7.2.4 measured Qwen3.6-27B Q6_K pp512 684 -> 935 t/s (+36.7%) and a 35B MoE Q6_K 2,153 -> 2,757 (+28.1%). The merged RDNA4 MMQ tuning (PR #26199, July) predates those master numbers, and llama.cpp master as of 2026-10-25 has no fix. The Flash pack estimate uses real per-type tensor sizes from the GGUF headers and covers routed-expert matmuls only (attention and PCIe copies excluded); offloaded layers that are PCIe-bound will hide part of the gap. Practical takeaway: on the HIP backend prefer Q5 packs for Flash until #25940 merges, and don't assume HIP's measured Q4_K_M prefill lead over Vulkan carries over to Q6. Vulkan has a smaller Q6_K penalty (open PR #22970 adds +15% on Q6_K). Strata uses its own kernels and is not covered. - Model: Qwen3.8-Flash-Next (estimate); Llama 8B, Qwen3.6-27B, 35B MoE (cited) - Quant: Q4_K / Q5_K / Q6_K / Q8_0 and UD/AP packs, all >= 4-bit - Engine: llama.cpp HIP (ROCm) - Engine version: master vs PR #25940 - GPUs: RX 9070 XT and Radeon AI PRO R9700 (gfx1201) - Context: pp512-pp4096 - Flags: -ngl 99, -ub 512 / 2048 - Q6_K MUL_MAT n=512, HIP master: 36.62 TFLOPS - Q6_K MUL_MAT n=512, with PR #25940: 69.66 TFLOPS - Q4_K MUL_MAT n=512, HIP master: 77.69 TFLOPS - R9700 Qwen3.6-27B Q6_K pp512, master -> PR: 684 -> 935 tok/s - Est. UD-Q6_K_XL expert prefill work vs UD-Q4_K_XL, HIP master: 1.58 x - Est. same with PR #25940: 0.97 x Sources: [https://github.com/ggml-org/llama.cpp/pull/25940](https://github.com/ggml-org/llama.cpp/pull/25940), [https://github.com/ggml-org/llama.cpp/pull/25587](https://github.com/ggml-org/llama.cpp/pull/25587), [https://github.com/ggml-org/llama.cpp/issues/27181](https://github.com/ggml-org/llama.cpp/issues/27181), [https://github.com/ggml-org/llama.cpp/pull/26199](https://github.com/ggml-org/llama.cpp/pull/26199), [https://github.com/ggml-org/llama.cpp/pull/22970](https://github.com/ggml-org/llama.cpp/pull/22970), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) Next: Measure HIP master vs master + PR #25940 on Flash-Next UD-Q6_K_XL and UD-Q5_K_XL hybrid prefill on 2x R9700 (decode expected flat). ## R9700 vs Arc Pro B70: same 32 GB, but the R9700 is ~1.7-1.9x faster on MoE and ~2x on dual-card Flash at 4-bit+ 2026-10-25 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-25-r9700-vs-arc-pro-b70 Both cards have 32 GB on a 256-bit bus (640 vs 608 GB/s), yet published results split by model type. Qwen3.6-35B-A3B Q4 on one card: R9700 165.2 tok/s (Vulkan) vs B70 97.7 (SYCL) or 87.3 (Vulkan, Mesa 26). Dense 27B Q4 is close (27.1 vs 21.4-25.3), and one dual-B70 tensor-parallel fork beats the R9700 Vulkan tensor PR on dense 27B (51.1 vs 45.0). For Qwen3.8-Flash-Next at 4-bit or higher on two cards with experts in RAM, the best B70 result we found is 23-33 tok/s against 52-65 (Q5, our measured) and 87-94 at 36K (vLLM MXFP4, our measured) on two R9700s. B70 notes: Mesa version is a 2x lever (Vulkan MoE decode 39.0 on Mesa 25.3.6 vs 86.1 on Mesa 26.2.5); SYCL is far faster for long prompts; upstream vLLM XPU serves FP16 only and has no BF16 path, with INT4/FP8/GPTQ via Intel's llm-scaler container. With MTP depth 2 on dense 27B a comparison video reports 46.2 (R9700) vs 28.4 (B70), consistent with the R9700's higher matrix throughput helping multi-token verification. A single-B70 Flash-Next result of 41.7 tok/s exists but uses a 3.05-bit pack, so it is excluded from this 4-bit-and-up comparison. Power: 230 W TBP (B70) vs 300 W (R9700). Newegg listings seen 2026-10-25: R9700 $1,749-$1,999.99, B70 $1,299.99-$2,878, so the B70's launch-price edge has mostly closed at that store. Cost per decode tok/s on two cards for Flash-Next 4-bit GGUF works out to roughly $55 (R9700) vs $78-117 (B70); this is an estimate built from different testers, engines and hosts, so treat all ratios as rough. - Model: Qwen3.6-35B-A3B, Qwen3.8-27B, Qwen3.8-Flash-Next - Quant: Q4 to Q5 GGUF, MXFP4 (all >= 4-bit) - Engine: llama.cpp Vulkan/SYCL/ROCm, Strata, vLLM (per source) - Engine version: per source - GPUs: 1-2x Radeon AI PRO R9700 vs 1-2x Arc Pro B70 - Context: short unless noted - Flags: per source - 35B-A3B Q4 decode, 1 card, R9700 Vulkan: 165.2 tok/s - 35B-A3B Q4 decode, 1 card, B70 SYCL / Vulkan Mesa 26: 97.7 / 87.3 tok/s - 27B Q4 decode, 1 card, R9700 / B70: 27.1 / 21.4-25.3 tok/s - 27B Q4_K_M decode, 2 cards tensor, R9700 / B70: 45.0 / 51.1 tok/s - Flash-Next 4-bit+ decode, 2 cards, best B70 found: 23.0-33.3 tok/s - Flash-Next decode at 36K, 2x R9700 vLLM MXFP4 (measured): 87-94 tok/s - Est. USD per decode tok/s, 2-card Flash GGUF, R9700 / B70: 55 / 78-117 USD Sources: [https://riceharvest.github.io/awesome-b70/performance.html](https://riceharvest.github.io/awesome-b70/performance.html), [https://www.pugetsystems.com/labs/articles/intel-arc-pro-b70-multi-gpu-ai-inference-performance/](https://www.pugetsystems.com/labs/articles/intel-arc-pro-b70-multi-gpu-ai-inference-performance/), [https://github.com/0xSero/qwen38-b70](https://github.com/0xSero/qwen38-b70), [https://www.intelinside.ai/runtimes/llamacpp](https://www.intelinside.ai/runtimes/llamacpp), [https://www.youtube.com/watch?v=pNYNRUOnPeU](https://www.youtube.com/watch?v=pNYNRUOnPeU), [https://github.com/ggml-org/llama.cpp/pull/25051](https://github.com/ggml-org/llama.cpp/pull/25051), [https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md), [https://www.newegg.com/p/pl?N=100008333+601191745](https://www.newegg.com/p/pl?N=100008333+601191745) Next: Watch for a 4-bit-or-higher dual-B70 Flash-Next result on vLLM or llm-scaler and refresh prices before turning this into a full comparison page. ## Flash-Next's 50 GiB n-gram table is read lazily from disk cache: a cold cache can halve prefill, and 256 GB of DDR5 avoids it 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-flash-next-cold-ngram-table Qwen3.8-Flash-Next GGUFs carry one huge per-layer n-gram embedding table (50.66 GiB in UD-Q5_K_XL, read from the file headers). By default llama.cpp does not load it; any tensor over 4 GiB is read row by row from the memory-mapped file during prefill. Published tests show a cold table costs 2.2x prefill on real text (2825 vs 1293 tok/s at UD-Q4_K_XL) and ~5% decode. The direct-read fix was closed unmerged, so on a box with enough RAM the options are keeping the table warm in page cache or loading it with --lazy-mode off. Separately, llama.cpp removed --no-mmap, --mmap, --mlock and -dio in September; launchers must use --load-mode. Header-only read of unsloth's UD-Q5_K_XL: per_layer_token_embd.weight is 50.66 GiB of 147.41 GiB total and the only tensor above the 4 GiB lazy threshold; UD-Q4_K_XL's table is 26.82 GiB (IQ4_NL). Current master: --lazy-mode auto reads tensors > 4 GiB on demand and still uses mmap even with --load-mode none; --load-mode auto means mmap on discrete GPUs (only iGPUs opt out), so dual R9700 setups get the lazy path by default. Cited costs, NVIDIA hosts: UD-Q4_K_XL, 24,576-token varied prompt, table in RAM 2825 tok/s prefill and 90.0 tok/s decode vs cold mmap 1293 and 85.9. A low-RAM box that can never keep the table cached saw cold 584 vs warm 2019 tok/s at 3.3K tokens, with the gap shrinking to almost nothing past 130K. Benchmark trap: prompts built from repeated filler text touch only a few table rows and always measure the warm case. The direct-read PRs (#28136, then #29030 '--lazy-mode on-direct') were both closed without merging. Flag change: #20834 folded the old flags into --load-mode (none = old --no-mmap, dio = old -dio) and #28334 removed the old flags on 2026-09-09. Not yet measured on R9700; a cold / warm / lazy-off A/B is planned. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL - Engine: llama.cpp (cited results on NVIDIA; source read on master b3fa69955e) - Engine version: master 2026-10-25 - GPUs: cited: RTX PRO 6000; target: 2x Radeon AI PRO R9700 - Context: 24,576-token prompt (cited) - Flags: --lazy-mode auto vs off; --load-mode - UD-Q5_K_XL n-gram table size: 50.66 GiB - Prefill, table in RAM (cited, UD-Q4_K_XL): 2825 tok/s - Prefill, cold mmap table (cited, UD-Q4_K_XL): 1293 tok/s - Decode, table in RAM vs cold (cited): 90.0 vs 85.9 tok/s Sources: [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q5_K_XL](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q5_K_XL), [https://github.com/ggml-org/llama.cpp/pull/28136](https://github.com/ggml-org/llama.cpp/pull/28136), [https://github.com/ggml-org/llama.cpp/pull/29030](https://github.com/ggml-org/llama.cpp/pull/29030), [https://github.com/ggml-org/llama.cpp/pull/26081](https://github.com/ggml-org/llama.cpp/pull/26081), [https://github.com/ggml-org/llama.cpp/pull/20834](https://github.com/ggml-org/llama.cpp/pull/20834), [https://github.com/ggml-org/llama.cpp/pull/28334](https://github.com/ggml-org/llama.cpp/pull/28334), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model-loader.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model-loader.cpp) Next: A/B on 2x R9700 at Q5: cold table vs pre-warmed vs --lazy-mode off vs --lazy-mode off with --load-mode none, using varied real prompts at 3K/13K/30K tokens. ## One badly cooled R9700 can silently cap a dual-card setup: watch the hotspot-to-edge gap 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-r9700-thermal-gating A public 2x R9700 investigation traced a ~6-10% per-card prefill deficit to a poor thermal interface, not weak silicon or power delivery. The bad card hit its ~110 C junction limit at ~230 W instead of ~300 W, with a 47 C hotspot-to-edge gap versus ~37-38 C on a healthy card. Because tensor-parallel runs move in lockstep, it dragged the pair's result down too, and the stock fan curve never reacted because it follows edge temperature. Deep prefill (pp131072, ROCm 7.14): defective card junction 110 C, edge 62 C, gap 47 C (peak 62), ~229-230 W at ~2790-2867 MHz, 297-300 t/s; healthy card gap ~38 C, ~297 W, ~3050 MHz. Swapping PSU cables between cards left the deficit with the card. After warranty service the same card measured a 37.4 C gap, 297 W, 3057 MHz and 323.2 t/s (+8.4% vs its own earlier run). The author treats the pair's tp=2 prefill of ~1841-1894 t/s as a lower bound next to a ~1965 t/s community 2x R9700 run, and gives ~37 C as healthy and a drift toward ~47 C as a warning. On Navi 48, fan control lives in gpu_od/fan_ctrl/fan_curve (five hotspot-temperature points, 25% minimum); staged points read back as if applied but only take effect after writing 'c' to commit. For scale, single-card sustained loads in reviews: 3,200-3,400 rpm and 57.9 dB(A) (igor'sLAB), 99 C hotspot without throttling in a 7.7-minute FurMark run, 98 C junction on a 32-minute agentic 35B run, and a 210 W cap cutting 97 C to 87 C for a 1.2% token-generation loss. Practical takeaway: before trusting any dual-R9700 benchmark, log each card's junction, edge, power and clock under a long load and compare the cards; a gap of 45 C or more, or one card stuck well below its power cap at a hot junction, points at the cooler rather than the software. Our own earlier note: hybrid DDR5-offload decode draws only ~180-190 W per card, so this mostly bites long prefill and fully in-VRAM tensor-parallel runs. - Model: per source (vLLM FP8 models; 35B MoE GGUF) - Quant: per source - Engine: vLLM ROCm tp=2, llama.cpp (published) - Engine version: ROCm 7.14, amdgpu DKMS 6.19.4 (thermal investigation) - GPUs: 2x Radeon AI PRO R9700 (investigation); 1x in reviews - Context: pp131072 deep prefill - Flags: stock fan curve, 300 W cap - Hotspot-to-edge gap, defective card: 47 C - Hotspot-to-edge gap, same card after service: 37.4 C - Sustained board power, defective card: 230 W - Sustained board power, after service: 297 W - Deep prefill gain after service: 8.4 % Sources: [https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-mem-vendor-bios-variance.md](https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-mem-vendor-bios-variance.md), [https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit](https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit), [https://docs.kernel.org/gpu/amdgpu/thermal.html](https://docs.kernel.org/gpu/amdgpu/thermal.html), [https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/](https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/), [https://pendakwah.tech/reviews/r9700ai/](https://pendakwah.tech/reviews/r9700ai/), [https://branetechnologies.com/blog/local-35b-model-gpu-thermals-waterblock](https://branetechnologies.com/blog/local-35b-model-gpu-thermals-waterblock), [https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/](https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/) Next: Baseline both of our R9700s under a 20-minute in-VRAM prefill load (per-card gap, power, clock, plus single-card runs) and re-check monthly. ## GPU_MAX_HW_QUEUES on the R9700: one variable behind idle-busy power, spawn-lottery decode and slow small kernels 2026-10-25 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-25-gpu-max-hw-queues Four public R9700 (gfx1201) reports point at HIP's hardware-queue count. With it uncapped, idle HIP servers have sat at 100% gfx and 92-95 W per card (=1 gave 21-22 W), vLLM decode landed at 33 or 26 tok/s depending on process start, and small kernels ran ~17 us slower behind pending cross-stream waits. Capping it cost nothing measurable in the one test that checked (0.1% decode, 0.3% prefill). Idle: ROCm #5706 single R9700 (MES 0x84, llama.cpp ROCm image) 92-95 W idle with a model loaded; GPU_MAX_HW_QUEUES=1 brought it to 3% / 21-22 W, while --no-warmup, --no-mmap and smaller context changed nothing. PyTorch resident processes stayed pinned even with =1. On ROCm 7.14 (MES 0x8b) a smaller residual remains: HIP 18-19 W vs Vulkan 12-13 W per card on the same 3x R9700 box (ROCm #6453, open). Speed: vLLM on one R9700 was bimodal per spawn (33.1-33.2 vs 25.7-26.2 tok/s); with =1, 8 of 8 cold spawns landed at 33.43-33.52 and the KFD queue count dropped 7 to 3. Two or more streams with pending hipStreamWaitEvent waits added ~17 us to every small kernel (gone at 2 or fewer queues; gfx1030 unaffected). Strata reported its MTP draft step bimodal per start (about 6% of decode on PCIe x16, up to ~25% on a chipset x4 link); =2 fixed every start on both links, =1 was slow on x4, =16 always slow (default 4). Estimate: a 24/7 resident two-card server stuck in the idle-busy state uses ~1,660 kWh/yr of card power (~$264/yr at the July 2026 Texas residential average) vs ~385 kWh/yr (~$61) with the cap. Single-GPU reports only so far; whether a dual-R9700 layer split shows a slow-start mode is untested, so we pin =2 before further A/B runs. - Model: various (Qwen3-14B FP8, Gemma 4 12B FP8, Qwen3.x 35B GGUF, Flash-Next via Strata) - Quant: per source - Engine: llama.cpp HIP, vLLM ROCm, Strata (published) - Engine version: ROCm 7.2.x to 10, per source - GPUs: 1-3x Radeon AI PRO R9700 - Context: per source - Flags: GPU_MAX_HW_QUEUES unset vs 1 vs 2 - Idle power per card, HIP idle-busy (model loaded): 95 W - Idle power per card, GPU_MAX_HW_QUEUES=1: 22 W - Idle power per card, Vulkan reference (ROCm 7.14 report): 12.5 W - vLLM decode, fast vs slow spawn (single R9700): 33.2 / 26.0 tok/s - Extra time per small kernel with pending cross-stream waits: 17 us - Cost of =1 on decode (measured, noise-level): 0.1 % - Est. yearly card energy, 2 cards idle-busy vs capped: 1664 / 385 kWh Sources: [https://github.com/ROCm/ROCm/issues/5706](https://github.com/ROCm/ROCm/issues/5706), [https://github.com/ROCm/ROCm/issues/6453](https://github.com/ROCm/ROCm/issues/6453), [https://github.com/ROCm/legacy-rocm-build/issues/6347](https://github.com/ROCm/legacy-rocm-build/issues/6347), [https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a](https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a) Next: Spawn-lottery A/B on 2x R9700 (unset vs 1 vs 2, 8 cold starts each) for llama.cpp UD-Q5 (then Q4, Q6) and Strata Q6, plus idle watts with the model resident. ## llama.cpp PR #29963 runs multi-GPU pipeline parallelism with MoE experts in RAM (+155% prefill at 32K on ROCm), but the usual --n-cpu-moe layout can never trigger it 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-pp-host-experts-gate Today llama.cpp switches pipeline parallelism off whenever tensor overrides put MoE experts in system RAM, so expert uploads go through the first GPU and the GPUs take turns. Open PR #29963 lets each GPU upload its own layers' experts. A ROCm test on 4x MI50 (DeepSeek-V4-Flash MXFP4) went from 122.5 to 312.5 tok/s prefill at a 32K prompt (+155%) and +28% at 8K, with decode up about 1.5%. Reading the PR code, it only engages when no single GPU's layer range owns more than 2/3 of the RAM-resident expert bytes. On two GPUs, --n-cpu-moe K puts layers 0..K-1 in RAM, and those all belong to the first card, so the gate fails for every Q4-Q6 Flash-Next layout. A spread override list (half the RAM-resident layers taken from each card's slice) passes it. Pinned host memory is also needed: in the PR's own tests the default mmap load gave no gain (2029 to 2088 tok/s) while --load-mode none went from 2705 to 5486 on Qwen3.5-35B-A3B with all experts in RAM. Merged PR #30449 splits CUDA-family pinned buffers into 4 GiB pieces so 50+ GiB of experts can be pinned. Other requirements: -sm layer, all layers offloaded, op offload and KQV offload on, and async plus events on every device; the Vulkan backend reports both, but no Vulkan result exists yet. CUDA results from the PR author on 3 GPUs with Qwen3.8-Flash-Next Q8_0: 52K-prompt prefill 1009 to 1808 tok/s and pp32768 1334 to 2474, with tg64 46.2 to 47.1. For Flash-Next on two cards (48 layers, 0-23 on card 1), a spread list that keeps the high-imatrix-energy layers 36-47 in VRAM also balances VRAM per card, so -ts 1,1 should fit. Not yet measured on R9700; expect decode within 2% and prefill gains that grow with prompt length. - Model: Qwen3.8-Flash-Next (layout analysis); DeepSeek-V4-Flash and Qwen3.5-35B-A3B (cited tests) - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (analysis); MXFP4, Q8_0 (cited) - Engine: llama.cpp PR #29963 (head 3ca7eed, open) - GPUs: 2x Radeon AI PRO R9700 (analysis); 4x MI50 ROCm 7.14 and 3x RTX PRO 4000 (cited) - Context: 131072-262144 - Flags: -sm layer, experts in RAM via -ot spread over both cards, --load-mode none - ROCm 4x MI50 prefill at 32K prompt, master: 122.5 tok/s - ROCm 4x MI50 prefill at 32K prompt, PR #29963: 312.5 tok/s - ROCm 4x MI50 prefill gain at 8K prompt: 28 % - ROCm 4x MI50 decode change: 1.5 % - Share of RAM experts on card 1 with --n-cpu-moe 24 (2 cards): 100 % - Gate limit per card: 66.7 % Sources: [https://github.com/ggml-org/llama.cpp/pull/29963](https://github.com/ggml-org/llama.cpp/pull/29963), [https://github.com/ggml-org/llama.cpp/pull/30449](https://github.com/ggml-org/llama.cpp/pull/30449), [https://github.com/ggml-org/llama.cpp/pull/24156](https://github.com/ggml-org/llama.cpp/pull/24156) Next: A/B on 2x R9700 with HIP: master --n-cpu-moe vs the PR with a spread override list and --load-mode none, Q4 then Q5 then Q6, pp8192/pp32768/tg64. ## R9700 ECC hides about 2 GiB of VRAM per card; turning it off is worth an estimated 4-12% on Flash-Next across two cards 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-r9700-ecc-vram Radeon AI PRO R9700s ship with VRAM ECC (GECC) on, so Linux shows about 30,576 MiB per card instead of 32,768. Disabling it with the amdgpu.ras_enable=0 kernel parameter (two reboots) frees about 2.1 GiB per card. On a calibrated bandwidth model for Qwen3.8-Flash-Next with experts in DDR5, that moves 2-3 MoE layers onto the GPUs and raises the decode ceiling 4-7% from capacity alone, or 8-12% if a published +9% single-card decode gain from ECC off also holds. Estimated decode ceilings, ECC on vs off (capacity only / capacity plus +9% GPU bandwidth): UD-Q4_K_XL 43.2 vs 46.1 / 48.5 tok/s, UD-Q5_K_XL 32.1 vs 34.2 / 35.4, UD-Q6_K_XL 28.7 vs 29.9 / 30.9. GPU-resident MoE layers go from 29/22/20 to 32/25/22 of 48. Our earlier dual-R9700 ceilings (45/34/30) assumed the full 32 GiB, so with ECC left on they are about 3-5% too high. The +9% figure is one source (Qwen3.6-35B-A3B Q4_K, 101 to 110 tok/s on one R9700 in a Vulkan engine) and is not yet reproduced. The trade-off: VRAM bit flips go undetected, so stress-test first and keep ECC on if the kernel log ever reported ECC errors. Check the state with cat /sys/module/amdgpu/parameters/ras_enable (-1 is the default, ECC on) and dmesg | grep GECC. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL - Engine: llama.cpp (model, not measured) - GPUs: 2x Radeon AI PRO R9700 - Context: 262144, f16 KV - Flags: layer split, experts of the first N layers in DDR5 (-ncmoe), no MTP - Visible VRAM per card, ECC on: 30576 MiB - Visible VRAM per card, ECC off: 32768 MiB - UD-Q5_K_XL decode ceiling, ECC on (estimate): 32.1 tok/s - UD-Q5_K_XL decode ceiling, ECC off (estimate): 34.2 tok/s - Published single-R9700 decode gain, ECC off: 9 % Sources: [https://github.com/ROCm/ROCm/issues/5832](https://github.com/ROCm/ROCm/issues/5832), [https://lunnova.dev/articles/amdgpu-disabling-ecc/](https://lunnova.dev/articles/amdgpu-disabling-ecc/), [https://forum.level1techs.com/t/sapphire-radeon-r9700-not-32gb/244649](https://forum.level1techs.com/t/sapphire-radeon-r9700-not-32gb/244649), [https://zolotukhin.ai/zinc/docs/rdna4-tuning/](https://zolotukhin.ai/zinc/docs/rdna4-tuning/), [https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md](https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md) Next: Measure ECC on vs off with llama-bench on two R9700s at Q4 and Q5, 32K depth, with a soak test before keeping it off. ## Q5 experts on GPU, Q4 in DDR5: placement-aware Flash-Next quant only wins if late layers matter more 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-placement-aware-quant A custom Flash-Next GGUF with UD-Q5 experts on the VRAM-resident layers and UD-Q4 on the DDR5 layers is estimated at ~39 tok/s ceiling on 2x R9700 (UD-Q4 46, UD-Q5 34), but with equal per-layer sensitivity its quality lands right on the Q4-Q5 line: extra GPU bits cost VRAM, which pushes layers to DDR5. Unsloth's public imatrix shows routed-expert inputs in layers 36-47 carry 4-19x the median energy, and llama.cpp's --n-cpu-moe keeps exactly those on the GPU, so it may land near Q5 quality at +14% speed. Needs a KLD measurement. Two concrete facts from the llama.cpp source: --tensor-type / --tensor-type-file take regexes and the first match wins, and Flash-Next's expert down tensors have 640 columns, so 256-block k-quants silently fall back (Q4_K to Q5_0, Q5_K to Q5_1, Q6_K to Q8_0). That is why Unsloth's UD files use Q5_1/Q8_0 downs, and why a hand-made 'Q4_K everywhere' recipe still gets 5.5-bit downs; IQ4_NL or Q4_0 are the only 4.5-bit options. Unsloth's own Q8_0-down layer picks (2, 4, 30, 46, 47) do not follow imatrix energy, so energy is a proxy, not a sensitivity measure. Keeping stock UD-Q4's Q8_0-down layers on GPU with -ot is a slight loss in the model (45.4 vs 46.1 tok/s). All speeds are roofline estimates (GPU at 75% of 640 GB/s, 55 GB/s DDR5, 262K f16 KV, layer split), not benchmarks. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / custom mix (all >=4-bit) - Engine: llama.cpp (model) - GPUs: 2x Radeon AI PRO R9700 - Context: 262K - Flags: layer split, --n-cpu-moe, f16 KV - GPU-Q5 / DDR5-Q4 mix, decode ceiling (est.): 39.1 tok/s - UD-Q4_K_XL ceiling (est.): 46.1 tok/s - UD-Q5_K_XL ceiling (est.): 34.2 tok/s - Mix KLD, equal-layer est.: 0.038 - Mix KLD, energy-weighted est.: 0.033 - Layer 47 expert-down input energy vs median: 18.7 x Sources: [https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://unsloth.ai/docs/models/qwen3.8-next](https://unsloth.ai/docs/models/qwen3.8-next) Next: Build the mix from BF16 with Unsloth's imatrix and measure held-out KLD and decode against UD-Q4 and UD-Q5 on the local desktop; adopt only if KLD is within 15% of UD-Q5 at 10%+ more speed. ## Qwen3.8-Flash-Next at long context on R9700: keep f16 KV, get the QSA indexer fixes 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-flash-depth-decay Flash-Next's sparse attention should keep decode nearly flat with depth, but llama.cpp issues show three separate slowdowns: a HIP-only CPU top-k fallback past 1K context (about 20 to 6 tok/s, fixed by PR #27466), an indexer that re-pooled the whole cache every token (pooled-key cache PR #28699, +9% at 63K-114K), and top-k applied as a dense mask. Quantized KV does not help: q8_0 decoded 24-30% slower than f16 at 128K-250K on this model, and on gfx1201 batch-1 quantized KV at head 256 takes a slower kernel. Bandwidth estimate for 2x R9700 with experts in DDR5 (UD-Q4, 45 tok/s at short context): about 35 tok/s at 131K and 29 at 262K with dense attention and f16 KV, versus about 44 and 43 if the sparse path and pooled cache both apply. The f16 KV cache at 262K is only about 6 GiB across both cards (q8_0: 3.2 GiB), so the VRAM it would free is worth a few percent at most. Sparse flash attention (PR #28770) was built for CUDA; whether HIP batch-1 decode reaches it on gfx1201 is not yet verified. A patched CUDA fork went from 10.2 to 33.7 tok/s at 250K on 6x RTX 3090, which shows how much is at stake. The k-pool indexer layout (#29751) doubles the indexer cache and mirrors it on every GPU under tensor split (issue #30005). - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL - Engine: llama.cpp (HIP) - Engine version: per source - GPUs: 2x Radeon AI PRO R9700 (estimate); cited runs on gfx1151, gfx1201, A100, RTX 3090 - Context: 0-262K - Flags: layer split, -fa on, --parallel 1 - HIP decode before top-k fix, d0 to d1024 (gfx1151): 19.6 -> 6.1 tok/s - Pooled-key cache gain at 63K / 114K (CUDA): +9.3 / +9.4 % - q8_0 vs f16 KV decode at 128K / 250K (A100): -24 / -30 % - q4_1 vs f16 KV decode at 32K, gfx1201 27B: 20.55 vs 28.48 tok/s - Est. UD-Q4 decode at 131K, dense f16 vs sparse+pooled (2x R9700): 35 vs 44 tok/s (estimate) - f16 KV cache at 262K, both cards: 6 GiB Sources: [https://github.com/ggml-org/llama.cpp/issues/27856](https://github.com/ggml-org/llama.cpp/issues/27856), [https://github.com/ggml-org/llama.cpp/issues/28012](https://github.com/ggml-org/llama.cpp/issues/28012), [https://github.com/ggml-org/llama.cpp/pull/28699](https://github.com/ggml-org/llama.cpp/pull/28699), [https://github.com/ggml-org/llama.cpp/pull/28770](https://github.com/ggml-org/llama.cpp/pull/28770), [https://github.com/ggml-org/llama.cpp/issues/28734](https://github.com/ggml-org/llama.cpp/issues/28734), [https://github.com/ggml-org/llama.cpp/issues/27796](https://github.com/ggml-org/llama.cpp/issues/27796), [https://github.com/ggml-org/llama.cpp/issues/30005](https://github.com/ggml-org/llama.cpp/issues/30005) Next: Measure decode at 0, 32K, 128K and 200K on 2x R9700 with f16 and q8_0 KV on a current HIP build. If 128K decode is below 75% of short-context decode, look at bringing the sparse decode path to HIP. ## Which Flash MoE suits 2x R9700 + DDR5 offload? Qwen3.8-Flash-Next by 2.6-4x (bandwidth estimate) 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-flash-model-shootout From byte counts read out of the Unsloth GGUF headers: Qwen3.8-Flash-Next reads 6.1-6.7 GiB per token, DeepSeek-V4-Flash 10.1, GLM-5.3-Flash 13.6-16.0 and MiniMax-M3.1-Flash 21.3. On two R9700s with experts spilled to DDR5, that puts the decode ceiling (estimate) at 45/34/30 tok/s for Qwen Q4/Q5/Q6, 17 for DeepSeek-V4 Q4, 11/9/7 for GLM Q4/Q5/Q6 and about 6 for MiniMax. Same model as our earlier roofline: GPU at 75% of 640 GB/s, measured 55 GB/s DDR5 read, 2.5 GiB reserve per card, 262K f16 KV, layer split, no MTP. These are upper-bound estimates, not benchmarks; llama.cpp has measured 55-66% of the ceiling on Flash-Next. GLM-5.3-Flash is attention-heavy (6.15 GiB of attention weights read every token at UD-Q4; 12 MLA + 34 linear layers). DeepSeek-V4-Flash and MiniMax-M3.1-Flash store experts as native 4-bit (MXFP4 / NVFP4) in every Unsloth file, so their Q8 files only upgrade the dense part and there is no Q5/Q6 choice. MiniMax keeps full KV in 60 layers (about 30 GiB at 262K, estimate). DeepSeek-V4.1-Flash-1017 is 466 GiB at UD-Q4, too big for 256 GB RAM plus 64 GB VRAM, and its architecture is not in llama.cpp master yet. - Model: Qwen3.8-Flash-Next, GLM-5.3-Flash, DeepSeek-V4-Flash, MiniMax-M3.1-Flash - Quant: UD-Q4_K_XL to UD-Q6_K_XL (UD-Q8_K_XL where experts are native 4-bit) - Engine: bandwidth model (llama.cpp layout) - GPUs: 2x Radeon AI PRO R9700 - Context: 262144 - Flags: layer split, f16 KV, experts spilled to DDR5, no MTP - Qwen3.8-Flash-Next UD-Q5_K_XL ceiling: 34.1 tok/s (est.) - DeepSeek-V4-Flash UD-Q4_K_XL ceiling: 17 tok/s (est.) - GLM-5.3-Flash UD-Q5_K_XL ceiling: 8.8 tok/s (est.) - MiniMax-M3.1-Flash UD-Q4_K_XL ceiling: 6.4 tok/s (est.) - Bytes read per token, Qwen vs GLM (UD-Q5): 6.5 vs 14.6 GiB Sources: [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF), [https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF](https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF), [https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF](https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF), [https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF](https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp) Next: Measure DeepSeek-V4-Flash UD-Q4 against GLM-5.3-Flash UD-Q5 on the local desktop to check the ~1.6x estimate; Qwen3.8-Flash-Next stays the main dual-R9700 lane. ## Expert pruning could fit a Q4 Qwen3.8-Flash-Next fully on two R9700s, but it costs world knowledge 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-reap-q4plus Community REAP builds of Qwen3.8-Flash-Next drop the least-used experts (512 to 320 or 288 per layer) and keep the survivors bit-for-bit. Pruning alone does not speed up decode: with all experts on CPU, a 256-expert build decoded at 22.0 vs 21.5 tok/s for the full model (prefill 353 vs 223 tok/s). The gain comes only from fitting more experts in VRAM. Almost all public REAP files are Q2/Q3; the only Q4+ ones we found are a REAP-288 Q4_K_M (78.05 GiB) and Q5_K_M (87.37 GiB). Because each layer's experts are contiguous in a GGUF, a published 320-expert keep-list can also be applied to Unsloth's UD-Q4_K_XL by byte copy, giving an estimated 76.8 GiB file whose compute path fits two 32 GB cards (about 95% VRAM at 262K context). Our roofline estimate: about 74 tok/s decode vs about 45 for the unpruned UD-Q4_K_XL with 17 expert layers in DDR5, the same gain we estimated for adding a third card. The cost: the REAP-320 author measured general-knowledge fabrication at 25% vs 0% for the unpruned model on a 12-question probe, while HumanEval held (96.3% vs 95.7%). Builders report rare-token reliability and clean termination degrading below about 288-320 kept experts, and that retained routing mass does not rank two prunes; only behavioral tests do. One also reported a silent slowdown on a 32 GB CUDA card: a fully resident config at 98.7% VRAM ran 10.82 tok/s at 98K context vs 116 tok/s at 8K, with only a fit warning in the log, so leave headroom. The REAP-288 Q4_K_M looks faster in our estimate (about 100 tok/s) mainly because its attention tensors are quantized lower than Unsloth's (1.75 vs 3.04 GiB), which is a quality trade too. File sizes from the Hugging Face API; tensor sizes from the GGUF headers; tok/s figures are our estimates, not measurements on R9700. - Est. decode, UD-Q4_K_XL unpruned vs 320-expert slice (2x R9700): 45 -> 74 tok/s (estimate) - Est. decode, UD-Q5_K_XL unpruned vs 288-expert slice + q8_0 KV: 34 -> 67 tok/s (estimate) - CPU-only decode, 256-expert prune vs full model: 22.0 vs 21.5 tok/s - General-knowledge fabrication, REAP-320 vs unpruned (12 questions): 25 vs 0 % - HumanEval-164, REAP-320 vs unpruned UD-IQ1_M: 96.3 vs 95.7 % Sources: [https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF](https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF), [https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-256-duo-GGUF](https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-256-duo-GGUF), [https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-GGUF](https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-GGUF), [https://github.com/CerebrasResearch/reap](https://github.com/CerebrasResearch/reap) Next: Slice UD-Q4_K_XL with the published 320-expert list, measure decode at 8K and 98K on two R9700s, and run a tool-call plus fabrication gate against the unpruned file before using it for anything beyond coding agents. ## Qwen 3.5/3.6/3.8 on R9700: make sure your llama.cpp ROCm build has PR #28102, or long-context prefill can run ~2x slower 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-rdna4-fa-head256 Every Qwen 3.5/3.6/3.8 model (including the Flash-Next MoE) uses attention head size 256. After llama.cpp removed its rocWMMA FlashAttention kernel (PR #26046), head 256 on RDNA4 had no tensor-core attention path and fell back to a scalar tile kernel. Reported on a single R9700 with Qwen3.6-35B-A3B Q4_K_M: pp4096 at 127K context dropped from 777 to 395 tok/s; on dual R9700s with a 122B-A10B, pp512 at 65K dropped from 571 to 344, and rocprofv3 put attention at 25.4 s vs 65.2 s. PR #28102 (merged 2026-09-11, commit 16378d93f) enabled and tuned the WMMA FlashAttention kernel for head 256 on RDNA4: Qwen3.8-27B IQ4_XS pp512 at 150K went from 164 to 457 tok/s on an R9700, and a dual-R9700 27B Q4_K_M tensor-split run measured 747.5 tok/s at 127K vs 341 before (2.2x), beating the old rocWMMA revert (600). Token generation is unchanged by both changes. If you build llama.cpp for ROCm, check `git merge-base --is-ancestor 16378d93f HEAD`. Open caveat (issue #28867): the new dispatch sends small multi-token batches (speculative-decoding verify batches, merged parallel slots) to the WMMA kernel once n_q x gqa_ratio_eff > 16. One R9700 owner measured MTP decode on Qwen3.8-27B (24 Q / 4 KV heads) fall from 51.7 to 41.4 tok/s, recovered by raising the threshold to 64 with no prefill loss; another owner on a tuned fork saw the opposite. Reading the quoted gate, Flash-Next (24 Q / 2 KV heads, effective GQA factor 4) keeps plain decode and single-stream MTP with up to 3 draft tokens on the tile kernel, so it should not hit this; two or more parallel slots with MTP would. This is a reading of quoted source, not a measurement. The old rocWMMA path also crashed around 106K tokens of prefill with HIP graphs on gfx1201 (#24961; GGML_CUDA_DISABLE_GRAPHS=1 avoided it), another reason to move to a post-#28102 build instead of reverting. - 27B pp512 @ 150K, before vs after #28102 (1x R9700): 164 -> 457 tok/s - 27B Q4_K_M pp4096 @ 127K, 2x R9700 tensor split, old master vs post-#28102: 341 -> 747.5 tok/s - 35B-A3B Q4_K_M pp4096 @ 127K, rocWMMA vs post-removal tile kernel: 777 -> 395 tok/s - 27B MTP decode regression reported in #28867 (threshold 16 vs 64): 41.4 vs 51.7 tok/s Sources: [https://github.com/ggml-org/llama.cpp/pull/28102](https://github.com/ggml-org/llama.cpp/pull/28102), [https://github.com/ggml-org/llama.cpp/issues/26220](https://github.com/ggml-org/llama.cpp/issues/26220), [https://github.com/ggml-org/llama.cpp/issues/28867](https://github.com/ggml-org/llama.cpp/issues/28867), [https://github.com/ggml-org/llama.cpp/issues/24961](https://github.com/ggml-org/llama.cpp/issues/24961) Next: Measure Flash-Next UD-Q5_K_XL prefill at 0/32K/128K on two R9700s: pre- vs post-#28102 ROCm builds vs Vulkan, with a kernel trace to confirm which attention kernel runs (its sparse-attention indexer path is unverified). ## Pooling 4 R9700s across two desktops: use 10GbE, not USB4, and it pays off most at Q5/Q6 (estimates + cited link data) 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-two-box-4x-r9700-link On X870 boards the USB4 ports use an ASMedia ASM4242. Linux host-to-host networking on that chip drops every packet unless the kernel carries a fix that turns off transmit end-to-end flow control; with the fix it measured 0.28 ms RTT but only 4.2-5.2 Gb/s. A direct 10GbE link gives similar latency (0.17-0.30 ms) at ~9.4 Gb/s. For a llama.cpp RPC layer split with one boundary between boxes, decode is latency-bound, so the estimated network tax is about 5-11% on either link; 10GbE mainly helps prefill (est. ~4% vs ~9% tax) and needs no kernel patch. Estimated Flash-Next decode, 262K context: two boxes x 2 cards keep every >=Q4 pack fully in VRAM (UD-Q5_K_XL est. ~62-66 tok/s vs est. ~51 on three cards in one box; for reference, Strata UD-Q5_K_XL measures 49-52 tok/s on two cards today, while a llama.cpp roofline estimate for two cards is ~34). At UD-Q4_K_XL a third card in one box is slightly better (~74 vs ~66-70) because Q4 already fits in 3 cards. Estimates use a bandwidth roofline (75% of 640 GB/s per card, 55 GB/s measured DDR5 read), 2-4 network round trips per token, and a ~3% per-token RPC software tax reported for a 1 local + 1 RPC GPU layer split. Do not put only the experts on the remote box: that layout makes two round trips per layer per token. In vLLM, keep tensor parallel inside each box and use pipeline parallel across boxes; tensor parallel across a network link all-reduces every layer. llama.cpp -sm tensor currently asserts on Qwen3 MoE with 2x R9700 on ROCm, so use -sm layer. Not yet measured on hardware. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimate) - Engine: llama.cpp RPC, -sm layer (estimate) - GPUs: 2 desktops x 2 Radeon AI PRO R9700 - Context: 262K, f16 KV, single request - ASM4242 USB4 host-to-host throughput (cited): 4.2-5.2 Gb/s - 10GbE direct link throughput (cited): 9.41 Gb/s - Est. UD-Q5_K_XL decode, 4 cards over 10GbE: 62-66 tok/s - Est. UD-Q5_K_XL decode, 3 cards one box: 51 tok/s - Est. UD-Q4_K_XL decode, 3 cards one box: 74 tok/s Sources: [https://github.com/torvalds/linux/commit/1881f2efbf7f78dc0a79a387b29fde6ff56d3731](https://github.com/torvalds/linux/commit/1881f2efbf7f78dc0a79a387b29fde6ff56d3731), [https://qiita.com/faliye/items/2a2d8def5298f6c1c10e](https://qiita.com/faliye/items/2a2d8def5298f6c1c10e), [https://github.com/kjaiswal/llama-cpp-distributed-benchmarks](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks), [https://github.com/ggml-org/llama.cpp/pull/24122](https://github.com/ggml-org/llama.cpp/pull/24122), [https://github.com/ggml-org/llama.cpp/issues/22307](https://github.com/ggml-org/llama.cpp/issues/22307), [https://docs.vllm.ai/en/stable/serving/parallelism_scaling/](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/), [https://jingchaozhang.github.io/LLM-Inference-Across-NVLink-and-InfiniBand/](https://jingchaozhang.github.io/LLM-Inference-Across-NVLink-and-InfiniBand/), [https://www.asrock.com/MB/AMD/X870%20Taichi%20Creator/](https://www.asrock.com/MB/AMD/X870%20Taichi%20Creator/) Next: Measure it: iperf3 and RTT on the 10GbE link, then llama-bench with --rpc at Q4, Q5 and Q6 once the second desktop is built. ## 3 or 4 R9700s: when Flash-Next fits fully in VRAM, and what 4-card owners report 2026-10-24 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-24-card-count-resident A cited 4x R9700 vLLM tensor-parallel-4 run of Qwen3.8-Flash-Next (MXFP4 experts, FP8 dense, ROCm 10) reports 219 tok/s at 32K context, one request. Our fit model (estimate) says UD-Q4_K_XL becomes fully VRAM-resident on 3 cards at 262K context, and UD-Q5/Q6_K_XL need 4. Estimate from measured GGUF header sizes, keeping the n-gram embedding table in system RAM, 262K f16 KV, 2.5 GiB reserve per card, 75% of 640 GB/s, 55 GB/s DDR5. GPU bytes needed: UD-Q4_K_XL 82.9 GiB, UD-Q5_K_XL 102.8, UD-Q6_K_XL 113.0, versus about 59 free on 2 cards, 88.5 on 3 and 118 on 4. Layer-split decode ceilings with no MTP: Q4 45 tok/s on 2 cards vs ~74 resident on 3; Q5 34 vs ~69 resident on 4; Q6 30 vs ~67 on 4 (only ~5 GiB slack). Once resident, the Q4/Q5/Q6 ceilings converge, so a 4-card box would favor Q5 or Q6. Layer split only removes the DDR5 bottleneck; the much larger 219 tok/s figure comes from 4-way tensor parallel. Stock vLLM requires the tensor-parallel size to divide the attention-head count and recommends pipeline parallel for uneven GPU counts, so 3-way TP likely needs a non-stock engine. A cited 3x R9700 tensor-parallel run (~182-191 tok/s) used a fine-tune with an unreported quant, so it is not counted here. Different rigs; not a measurement on our hardware. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (model); MXFP4 experts + FP8 dense (cited) - Engine: llama.cpp layer split (model); vLLM TP4 (cited) - GPUs: 2-4x Radeon AI PRO R9700 - Context: 262K (model); 32K (cited) - Cited 4x R9700 vLLM TP4 decode, 1 request: 219 tok/s - Cited 4x R9700 vLLM TP4 prefill, 11.8K input: 3571 tok/s - UD-Q4_K_XL GPU bytes at 262K (estimate): 82.9 GiB - UD-Q4_K_XL ceiling, 3 cards resident (estimate): 74 tok/s - UD-Q4_K_XL ceiling, 2 cards + DDR5 (estimate): 45 tok/s - UD-Q5_K_XL ceiling, 4 cards resident (estimate): 69 tok/s Sources: [https://amd-pro-bench.com/gpus/r9700/4-gpus](https://amd-pro-bench.com/gpus/r9700/4-gpus), [https://amd-pro-bench.com/gpus/r9700/3-gpus](https://amd-pro-bench.com/gpus/r9700/3-gpus), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://docs.vllm.ai/en/stable/serving/parallelism_scaling/](https://docs.vllm.ai/en/stable/serving/parallelism_scaling/), [https://github.com/vllm-project/vllm/issues/11797](https://github.com/vllm-project/vllm/issues/11797) Next: Measure a 3-card resident UD-Q4_K_XL run and a 4-card vLLM TP4 run if extra cards are pooled into one box. ## Other engines for Flash MoE on R9700 (gfx1201): AMD's HRX backend isn't supported yet, ik_llama.cpp is untested, KTransformers ROCm is stale 2026-10-24 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-24-alt-engines-gfx1201 AMD's native llama.cpp backend (ggml-hrx, draft) does not run on gfx1201 yet: a user hit 'no HRX dispatch registry for target gfx1201' on 2026-10-15 and AMD said support is planned with no timeline. ik_llama.cpp's HIP build was fully broken until a repair merged 2026-08-25, validated only on RDNA3 (gfx1101); it now supports Qwen3.8-Flash-Next and MTP, but nobody has published a gfx1201 run. KTransformers lists ROCm as historical documentation only. HRX early data is from RDNA3/RDNA3.5: AMD claims about 30-50% prefill uplift and parity to +15% non-MTP decode vs the faster of Vulkan and HIP. Community Strix Halo (gfx1151) runs on Qwen3-30B-A3B Q4_K_M show tg128 85.1 vs 68.0 tok/s for HIP (+25%), but another run had Vulkan ahead (89.5 vs 85.3); HRX could not chunk prefill past 1024 tokens, and one development branch produced garbage text while llama-bench still reported healthy speed, so check output, not just tok/s. Lemonade ships an HRX engine recipe (merged 2026-10-02) without MTP. The ik_llama.cpp HIP repair also fixed AMD GPUs being detected as NVIDIA (NaNs in MoE matmul) and a software byte-permute that made 4-bit table quants about 15x slower on HIP (IQ4_XS 17.16 to 266.31 tok/s on a small model, RDNA3). Its maintainer says ROCm has not been a project goal and he cannot test it, so expect breakage. Takeaway for dual-R9700 owners today: stay on mainline llama.cpp, Strata or vLLM; ik_llama.cpp is the one alternative worth a gated try for the CPU-expert path. - Engine: ggml-hrx draft, ik_llama.cpp, KTransformers (per sources) - GPUs: Radeon AI PRO R9700 (gfx1201); sources cover gfx1100/1101/1151 - HRX vs HIP tg128, Qwen3-30B-A3B Q4_K_M, gfx1151 (community): 25 % faster - HRX prefill uplift claimed by AMD (RDNA3/3.5): 30-50 % - ik_llama.cpp IQ4_XS tg before/after HIP fix, small model, RDNA3: 17.16 / 266.31 tok/s Sources: [https://github.com/ggml-org/llama.cpp/discussions/27219](https://github.com/ggml-org/llama.cpp/discussions/27219), [https://github.com/ggml-org/llama.cpp/pull/27218](https://github.com/ggml-org/llama.cpp/pull/27218), [https://github.com/lemonade-sdk/lemonade/pull/3723](https://github.com/lemonade-sdk/lemonade/pull/3723), [https://github.com/ikawrakow/ik_llama.cpp/pull/2339](https://github.com/ikawrakow/ik_llama.cpp/pull/2339), [https://github.com/ikawrakow/ik_llama.cpp/discussions/562](https://github.com/ikawrakow/ik_llama.cpp/discussions/562), [https://ktransformers.net/en/docs/hardware-platforms/cpu-gpu-requirements](https://ktransformers.net/en/docs/hardware-platforms/cpu-gpu-requirements) Next: Gated test of ik_llama.cpp (HIP) vs mainline on 2x R9700 with a Q5 Flash model: build, correctness check, then speed with identical expert placement. Watch for HRX gfx1201 support. ## Q4 vs Q5 for agents: small tool-call evals can't tell the quants apart 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-toolcall-eval-noise A published Qwen3.8-Flash-Next engine comparison found llama.cpp UD-Q4_K_XL at 73.0% on 300 BFCL v4 tool-calling samples, but its three repeats ranged 70.7 to 76.3 points, and the multi-turn subset swung by plus or minus 14. Unsloth's quality gap between UD-Q4_K_XL and UD-Q5_K_XL is mean KLD 0.047 vs 0.030. A standard McNemar sample-size calculation shows that confirming a 3-point tool-call gap between two quants needs about 870 paired cases (at 10% disagreement), and a 5-point gap about 310. Before picking the dual-R9700 Flash quant on agent accuracy, compare quants with paired runs on the same cases, require the same sign in every repeat, and treat 'no difference found' as 'gap under about 8 points', not 'equal'. The same comparison saw a failure that BFCL missed: one engine printed a tool call into its reply text instead of dispatching it, in a real two-turn agent job. Tool-call checks should count calls leaked into text separately. A pruned community GGUF of the same model also reports a run-to-run band of about 3 cases out of 40 at temperature 0 across backends. Sample sizes are from Connor's approximation (alpha 0.05 two-sided, 80% power) and are estimates; repeats of the same case are not independent, so real needs are higher. We built a 40-case paired tool-call harness for OpenAI-compatible servers (llama.cpp, Strata, vLLM) and will run Q4/Q5/Q6 on two R9700s. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL (published BFCL run); UD-Q4/Q5 KLD from Unsloth - Engine: llama.cpp (published) - GPUs: published run on one RTX 6000 Blackwell, not R9700 - Context: 131K served - Flags: temp 0.6, 3 repeats, Q8_0 KV, MTP 3 draft tokens - BFCL v4, llama.cpp UD-Q4_K_XL, mean of 3: 73 % - BFCL v4 repeat range: 5.67 points - UD-Q4_K_XL mean KLD: 0.046893 - UD-Q5_K_XL mean KLD: 0.030415 - Paired cases to confirm a 3-point gap (estimate, 10% disagreement): 870 cases - Paired cases to confirm a 5-point gap (estimate): 312 cases Sources: [https://www.demietrich.com/blog/q3.8fn_engine_comparison/](https://www.demietrich.com/blog/q3.8fn_engine_comparison/), [https://unsloth.ai/docs/models/qwen3.8-next](https://unsloth.ai/docs/models/qwen3.8-next), [https://huggingface.co/Cyronius/Qwen3.8-Flash-Next-131B-A6B-GGUF](https://huggingface.co/Cyronius/Qwen3.8-Flash-Next-131B-A6B-GGUF) Next: Run the paired 40-case tool-call harness on Q4, Q5 and Q6 Flash on two R9700s with identical server flags, 5 repeats each, and publish pass rates, leaked-call counts and McNemar results. ## Dual R9700 vs the hosted Qwen3.8 Flash API: the vLLM 4-bit lane is cheaper per token on power, the Strata Q5 lane is not 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-local-vs-api-cost-flash Using our measured speeds, cited prices and an estimated 465-615 W at the wall, one stream of the vLLM TP2 MXFP4 lane (about 90 tok/s) costs about $0.23-0.35 in electricity per million output tokens, against $0.47 on OpenRouter, and runs about 1.7x the hosted median speed (47-53 tok/s). Strata UD-Q5_K_XL (about 49 tok/s) costs about $0.42-0.64, roughly break-even or worse. Two cards only pay for themselves after 14-16.5 billion output tokens, about 5-6 years running 24x7 at one stream. API: OpenRouter qwen/qwen3.8-flash on 24 Oct 2026 lists $0.15 per million input, $0.47 output and $0.016 cached input; the weighted average actually paid for input is $0.0278 because of caching. Electricity: EIA July 2026 residential averages, Texas 15.88 and US 18.31 cents per kWh. Power is an estimate, not measured on our machine: 315 W for both cards (Puget Systems measured 310-315 W for two R9700s at one user under vLLM TP=2 on a 27B model) plus 150 W for a consumer desktop or Puget's ~300 W workstation overhead. Prefill is where local wins: about $0.005-0.014 per million input tokens versus $0.15 uncached, though cached API input ($0.016) is close. If batching scaled the way Puget measured on the 27B model (also an estimate, not measured on Flash), 4-8 streams would bring local to about $0.05-0.09 per million output tokens. Card prices: $1,699.99-$1,999.99 per R9700 on Newegg in October 2026. Idle draw matters for an always-on server: a ROCm 7.14 issue reports 18-19 W per idle R9700 versus 12-13 W on Vulkan. - Model: Qwen3.8 Flash - Quant: MXFP4-FP8-GPTQ (vLLM) / UD-Q5_K_XL (Strata) - Engine: vLLM TP2, Strata - GPUs: 2x Radeon AI PRO R9700, local desktop - Context: 36K test - vLLM MXFP4 lane, electricity per 1M output tokens (estimate): 0.23-0.35 USD - Strata UD-Q5 lane, electricity per 1M output tokens (estimate): 0.42-0.64 USD - Hosted API, per 1M output tokens: 0.47 USD - Hosted API median throughput: 47-53 tok/s - Local prefill, electricity per 1M input tokens (estimate): 0.005-0.014 USD - Output tokens to pay back two cards (vLLM lane, 1 stream): 14-16.5 billion Sources: [https://openrouter.ai/qwen/qwen3.8-flash](https://openrouter.ai/qwen/qwen3.8-flash), [https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a](https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a), [https://energyfactbook.com/electric-power-monthly/](https://energyfactbook.com/electric-power-monthly/), [https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/](https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/), [http://www.nowinstock.net/computers/videocards/amd/aipror9700/](http://www.nowinstock.net/computers/videocards/amd/aipror9700/), [https://github.com/ROCm/ROCm/issues/6453](https://github.com/ROCm/ROCm/issues/6453) Next: Measure GPU and CPU power on our machine during vLLM and Strata runs, plus vLLM aggregate speed at 2, 4 and 8 streams, to replace the estimates. ## llama.cpp n-gram speculation on Flash-Next with experts in RAM: off by default, and why 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-ngram-spec-hybrid-offload Reading llama.cpp master (71255fe, 2026-10-24): ngram-mod only drafts 48-64-token chains, so each verify batch is 49-65 tokens. That is above GGML_OP_OFFLOAD_MIN_BATCH (32), so RAM-resident experts get copied over PCIe for that step instead of being read by the CPU. Also, n-gram types never arm recurrent rollback, so on a hybrid model like Qwen3.8-Flash-Next every drafted round takes a full, host-serialized state checkpoint and a partial accept restores it and replays the accepted tokens. Our bandwidth model (estimate) says default ngram-mod is a net loss for this setup except on near-pure file rewrites. Sourced data points: llama.cpp PR #28123 (merged 2026-09-01) gave Flash-Next native recurrent rollback; with an MTP head at n-max 3 on an RTX PRO 6000 it went from 108 tok/s (no draft) to 183 on code and 144 on prose, vs 123 / 83 before. Open PR #28118 measured the host checkpoint at about 600 ms of an 825 ms round on Strix Halo (Vulkan, Flash-Next Q4_K_M, MTP): 32.4 tok/s without drafts, 6.2 with host checkpoints, 41.5 with on-device checkpoints. Open issue #28019: ngram-map-k4v with 3 parallel slots on a GB10 gave about 50 tok/s aggregate vs 66-70 with speculation off, and reports a recurrent-rollback test mismatch on master that affects MTP when slots are reused across conversations. Elsewhere, ngram-mod took a 177B MoE from 22.09 to 77.56 tok/s on a file edit (90.3% accepted) with no change on prose. Our estimate for UD-Q5_K_XL on 2x R9700 (bandwidth model, uniform routing, single slot): rewrite turns 0.87x at defaults, about 1.9x with GGML_OP_OFFLOAD_MIN_BATCH=256 if checkpoints are cheap, about 1.0x if they cost 600 ms; mixed agent turns 0.47x at defaults. Not measured on R9700 yet. - Model: Qwen3.8-Flash-Next - Quant: UD-Q5_K_XL (model), Q4-Q6 lane - Engine: llama.cpp - Engine version: master 71255fe (source read) - GPUs: 2x Radeon AI PRO R9700 - Context: 262K - Flags: -sm layer, routed experts partly in DDR5, -np 1 - MTP n-max 3 after #28123, code (RTX PRO 6000, published): 183 tok/s - No draft, same rig (published): 108 tok/s - Host checkpoint share of a speculative round (Strix Halo, published): 600 ms of ~825 - ngram-mod verify batch size: 49-65 tokens - Estimated UD-Q5 rewrite speedup, defaults (2x R9700): 0.87 x - Estimated UD-Q5 rewrite speedup, MIN_BATCH=256, cheap checkpoints: 1.9 x Sources: [https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), [https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp), [https://github.com/ggml-org/llama.cpp/pull/28123](https://github.com/ggml-org/llama.cpp/pull/28123), [https://github.com/ggml-org/llama.cpp/pull/28118](https://github.com/ggml-org/llama.cpp/pull/28118), [https://github.com/ggml-org/llama.cpp/issues/28019](https://github.com/ggml-org/llama.cpp/issues/28019), [https://ai-muninn.com/en/blog/qwen4exp-177b-three-2080ti](https://ai-muninn.com/en/blog/qwen4exp-177b-three-2080ti), [https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md](https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md) Next: Measure on 2x R9700: baseline vs ngram-mod (defaults, MIN_BATCH=256, n-max 30) on rewrite, agent and prose turns, plus the checkpoint cost. MTP with native rollback is the likelier win once a Q4-or-better MTP head is confirmed. ## New fastest Flash setup on dual R9700: vLLM TP2 with a 4-bit MXFP4 pack, 87-94 tok/s decode at 36K 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-vllm-tp2-mxfp4-flash-fastest vLLM 0.29.0.dev0 (tcclaviger/vllm:latest) with tensor parallel 2 and the tcclaviger MXFP4-FP8-GPTQ Qwen3.8 Flash pack decodes 83-114 tok/s short (story 81-94, code 100-114) and 87-94 tok/s at 36K, with about 3,800 tok/s prefill at 36K. That is about 1.8x Strata Q5 decode (about 49 at 36K) and 1.6x its prefill (about 2,300), and well above the best published 4-bit-or-higher dual-3090 llama.cpp result (37-41 tok/s). Pack revision 6fe3e74: experts MXFP4 at 4.25 bpw, n-gram table int6, MTP head 4.25 bpw, so every tensor is 4-bit or higher. 262K context, fp8 KV, card-default max-num-batched-tokens 4096. VRAM 27.55 / 27.40 GB, system RAM 90 GB including a 38.7 GB pinned n-gram table. Startup takes about 10 minutes. MTP acceptance 51.8%. Tool calls parse cleanly and reasoning comes back in a separate 'reasoning' field. 36K prefill runs were 3809 / 3804 / 3789 tok/s on a clean relaunch. An earlier run right after a 110 GB download got only about 1,030 tok/s prefill and logged a 'couldn't lock n-gram table' warning; page-cache pressure is the likely cause but this is unproven, so relaunch on a quiet system before benchmarking. Failed: max-num-batched-tokens 8192 at 262K, 16384 at 131K and 8192 at 131K all fail KV-cache sizing, and no flag to cap it was found. Quality (KL divergence vs a reference) of this pack has not been measured yet. For reference, a CUDA vLLM fork with an expert cache reports 85-90 tok/s on two 3090s, but with a W4A16 fine-tune of the model, so it is not a like-for-like comparison. - Model: Qwen3.8 Flash - Quant: MXFP4-FP8-GPTQ (tcclaviger), rev 6fe3e74 - Engine: vLLM - Engine version: 0.29.0.dev0 (tcclaviger/vllm:latest) - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) - Context: 262K - Flags: tensor parallel 2, fp8 KV, MNBT 4096, MTP - Decode, short (story / code range): 83-114 tok/s - Decode at 36K: 87-94 tok/s - Prefill at 36K: 3800 tok/s - Strata Q5 decode at 36K (same machine): 49 tok/s - Strata Q5 prefill at 36K (same machine): 2300 tok/s - MTP acceptance: 51.8 % Next: Measure KL divergence / top-token match of the MXFP4 pack against a Q8_0 reference before calling it the default. ## Strata Q5 tuning on dual R9700: the defaults are already about right 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-strata-q5-tuning-ab STAGE_TRIM=1 with layer-split=23 raised 36K decode from 48.5 to 50.3 tok/s (+3.7%) but cut 36K prefill from 2374 to 2171 tok/s (-8.6%), so the default stays. Strata's auto split point (K=23) beat 19, 21, 25 and 27. Speculation depth 3, 5 and 6 were no better than 4, SPLIT_MTP_BATCH had no effect, and adapt-async is refused in layer-split mode. Qwen3.8 Flash UD-Q5_K_XL. The n-gram table (per_layer_token_embd, Q8_0, 50.66 GiB) stays in the DDR5 page cache, with only small buffers on the GPUs, so it costs no VRAM. - Model: Qwen3.8 Flash - Quant: UD-Q5_K_XL - Engine: Strata - Engine version: 0.2.3 - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) - Context: 36K test - Flags: layer split, MTP - 36K decode, default: 48.5 tok/s - 36K decode, STAGE_TRIM=1 + split 23: 50.3 tok/s - 36K prefill, default: 2374 tok/s - 36K prefill, STAGE_TRIM=1 + split 23: 2171 tok/s Next: No further Strata tuning planned; vLLM MXFP4 is now faster. ## llama.cpp --moe-cache-mib on dual R9700: about +20-25% decode, but still about 2.5x behind Strata 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-llamacpp-moe-cache-mib-measured With the HIP dsv4 build on Qwen3.8 Flash UD-Q5_K_XL (262K, f16 KV), the VRAM expert cache lifted warm decode from 18.2 to 23.5 tok/s (story) and 36K decode from 17.0 to 19.6, at a 93.7% hit rate. Prefill fell from 231 to 162 tok/s at 36K because prompt batches only hit the cache 8-15% of the time. Warm story/code decode, then 36K decode/prefill, then hit rate: fit only 18.2/18.2, 17.0/231; +6544 MiB 22.0/19.6, 18.0/201, 79.75%; +12000 MiB 23.4/21.0, 19.4/183, 88.74%; -cmoe +40000 MiB 23.5/21.5, 19.6/162, 93.72%. Q6 at the -cmoe +40000 setting: 22.3/20.4, 18.8/147. --moe-cache-mib is a total split across both GPUs, and it can't be combined with tensor split ('MoE cache requires a GPU backend'). - Model: Qwen3.8 Flash - Quant: UD-Q5_K_XL (Q6 for comparison) - Engine: llama.cpp (HIP, dsv4 build) - Engine version: dsv4 build - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) - Context: 262K - Flags: f16 KV, --moe-cache-mib 6544 / 12000 / 40000 with -cmoe - Warm story decode, fit only: 18.2 tok/s - Warm story decode, -cmoe +40000: 23.5 tok/s - 36K prefill, fit only: 231 tok/s - 36K prefill, -cmoe +40000: 162 tok/s - Cache hit rate, -cmoe +40000: 93.72 % Next: Use a modest cache size if prefill matters; the prompt-batch hit rate is the limiter. ## GLM 5.3 Flash Q5_K_M on dual R9700: about 6 tok/s decode, HIP prefill 2.3x Vulkan, 524K context loads 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-glm53-flash-q5-dual-r9700 bartowski GLM 5.3 Flash Q5_K_M (227 GiB) on llama.cpp-dsv4 HIP decodes 6.4 tok/s short, 6.0 at 36K (prefill 112.7) and 5.1 at 113K (prefill 74.4). HIP prefill is about 2.3x Vulkan (48.8). The largest context that loads is 524K. Strata and the RDNA4 llama.cpp fork don't support GLM yet. q8 KV, ubatch/batch 2048, -t 16, 262K context. VRAM 24.8 / 27.8 GB. Failed: Vulkan with ubatch 4096 hit an amdgpu ring timeout (device lost) at about 29K tokens; stay on HIP or a smaller ubatch. - Model: GLM 5.3 Flash - Quant: Q5_K_M (bartowski), 227 GiB - Engine: llama.cpp (HIP, dsv4 build) - Engine version: dsv4 build - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) - Context: 262K - Flags: q8 KV, -ub 2048 -b 2048, -t 16 - Decode, short: 6.4 tok/s - Decode at 36K: 6 tok/s - Prefill at 36K: 112.7 tok/s - Decode at 113K: 5.1 tok/s - Prefill at 113K: 74.4 tok/s - Vulkan prefill (for comparison): 48.8 tok/s Next: Revisit if Strata or vLLM adds GLM 5.3 support. ## DeepSeek V4 Flash UD-Q4_K_XL on dual R9700: about 10 tok/s decode, and a full 1M context loads 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-deepseek-v4-flash-q4-dual-r9700 unsloth DeepSeek V4 Flash UD-Q4_K_XL (144.4 GiB) on llama.cpp-dsv4 HIP decodes 10.0 tok/s short (prefill 105-147), 9.1 at 36K (prefill 168.6) and 7.9 at 112K (prefill 107.9), about 1.5x GLM 5.3 Flash. A full 1M context loads, with short decode at 9.4 tok/s. Same build and settings as the GLM run, 262K context for the main numbers. VRAM 29.1 / 29.0 GB. - Model: DeepSeek V4 Flash - Quant: UD-Q4_K_XL (unsloth), 144.4 GiB - Engine: llama.cpp (HIP, dsv4 build) - Engine version: dsv4 build - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) - Context: 262K (1M also loads) - Flags: q8 KV, -ub 2048 -b 2048, -t 16 - Decode, short: 10 tok/s - Decode at 36K: 9.1 tok/s - Prefill at 36K: 168.6 tok/s - Decode at 112K: 7.9 tok/s - Prefill at 112K: 107.9 tok/s - Decode, short, 1M context: 9.4 tok/s Next: None planned. ## systemd-oomd killed a benchmark: a 227 GiB mmap'd model plus a big rsync crossed the memory-pressure limit 2026-10-24 · Dual R9700 speed · Failed · https://amdai.net/lab-log/#2026-10-24-systemd-oomd-mmap-kill Running a 227 GiB mmap'd model while a large rsync was copying files pushed memory pressure past 50%, and systemd-oomd killed the whole scope the job ran in. Fix: run heavy model jobs as their own systemd --user units so they are isolated from other work. systemd-oomd acts on memory pressure (PSI), not just free RAM, and kills the whole cgroup it picks. Page-cache churn from a large copy next to a huge mmap'd model is enough to trip it. Launch each server with systemd-run --user --unit=NAME (or a .service file) and avoid big copies during benchmarks. - Engine: llama.cpp (HIP, dsv4 build) - GPUs: 2x Radeon AI PRO R9700, local desktop (dual-channel DDR5) Next: Fixed by running heavy jobs as separate systemd --user units. ## Sharing llama-bench JSON? It includes the full model path you typed 2026-10-24 · Idea lab · Progress · https://amdai.net/lab-log/#2026-10-24-llama-bench-json-path-privacy llama-bench's json/jsonl output copies the -m argument verbatim into model_filename, so a shared raw file usually shows your home directory and username. There is no hostname field, and cpu_info/gpu_info are only device names. Trim model_filename to the file name before posting. Read from llama.cpp tools/llama-bench/llama-bench.cpp at master df9beac (2026-10-24): each row has 43 fields plus per-run samples_ns/samples_ts; model_type comes from llama_model_desc (e.g. 'qwen3next 80B.A3B Q5_K - Medium'), so the quant can be checked without the GGUF; test_time is UTC. For like-for-like comparisons, rows must match on model_type, n_prompt, n_gen, n_depth, type_k/type_v, flash_attn and n_ubatch. We wrote a converter that keeps only setup fields, reduces the path to the file name, rejects quants below Q4 and flags noisy rows (over 5% spread), fewer than 3 runs and missing long-context rows; it will back a result-submission form when the site's next build lands. - Engine: llama.cpp llama-bench - Engine version: master df9beac Sources: [https://github.com/ggml-org/llama.cpp/blob/df9beac/tools/llama-bench/llama-bench.cpp](https://github.com/ggml-org/llama.cpp/blob/df9beac/tools/llama-bench/llama-bench.cpp) Next: Suggested sharing command: llama-bench -m MODEL.gguf -fa 1 -p 512 -n 128 -d 0,8192,32768 -r 3 -o jsonl ## The CPU half of Flash decode in llama.cpp: no x86 repack for Q5/Q6 experts, keep the host buffer, and use -lm mlock 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-cpu-expert-path-repack-mlock Reading llama.cpp master (df9beac, Oct 24): x86 CPU weight repack exists only for Q4_0, Q4_K, IQ4_NL and MXFP4, so Unsloth UD-Q5_K_XL and UD-Q6_K_XL experts (Q5_K, Q6_K, Q8_0) get none, while AP-Q4_K_XL is 100% eligible and UD-Q4_K_XL 58%. With a ROCm or Vulkan GPU loaded, -ncmoe experts go to the GPU host buffer and are never repacked unless --no-host is set, and repacked weights can't be offloaded to the GPU for prefill. For pinned, swap-proof experts, -lm mlock now means no mmap plus mlock again. Why: with Flash-class experts in DDR5, the CPU reads most of every token's bytes, so its path matters. Repack: ggml_repack_get_optimal_repack_type() has x86 (AVX2) paths for Q4_0, Q4_K, IQ4_NL and MXFP4 (Q2_K on AVX-512); Q5_K, Q6_K and Q8_0 repack is ARM-only, and the AVX-512 Q6_K repack PR (#19706) is still open. make_cpu_buft_list() lists the GPU host buffer before the repack buffer, and under mmap the loader swaps the host buffer for a plain CPU buffer; ggerganov confirmed repack never triggers with CUDA_Host and -ncmoe (PR #19738). The repack buffer isn't host memory and the scheduler only offloads ops whose weights are host memory, so --no-host trades prefill offload for repack; where repack did trigger (Apple, gpt-oss 20B MXFP4) he measured pp512 0.64x and tg32 1.07x. Verdict: keep --no-host off on dual R9700. Load mode: after the July --load-mode refactor, --load-mode none was reported swapped out under load (25-27 to 7.55 tok/s) and mmap+mlock with -ncmoe was OOM-killed from double residency (issue #26110); PR #26135 (merged Jul 27) made -lm mlock mean no mmap plus mlock, which keeps the pinned host buffer that fast PCIe prefill copies need. Raise the memlock limit (LimitMEMLOCK=infinity) or the lock fails. Untested hypothesis, estimate only: Zen 4/5 desktop CCDs read at 32 bytes per FCLK cycle, 64 GB/s each at 2000 MHz (Chips and Cheese), and our measured 54-56 GB/s read / ~27 GB/s write sits just under one CCD's limits. If the CPU has two CCDs and threads were packing onto one, spreading them could lift a 55 GB/s read to about 70, worth about +14% at UD-Q5 (34 to 39 tok/s ceiling, layer split, no MTP), +16% at UD-Q6 and +10% at UD-Q4. Zero if the CPU has one CCD. - Model: Qwen3.8-Flash-Next GGUFs - Engine: llama.cpp master df9beac (source read) - GPUs: 2x Radeon AI PRO R9700 - Flags: -ncmoe, --no-host, --repack, -lm mlock, -t / -C / --cpu-strict - x86 repack-eligible expert bytes, AP-Q4_K_XL: 100 % - x86 repack-eligible expert bytes, UD-Q4_K_XL: 57.6 % - x86 repack-eligible expert bytes, AP-Q5_K_M: 29 % - x86 repack-eligible expert bytes, UD-Q5_K_XL / UD-Q6_K_XL: 0 % - Zen 4/5 desktop per-CCD read limit at FCLK 2000: 64 GB/s - Est. UD-Q5 2x R9700 ceiling if DDR5 read goes 55 to 70 GB/s (estimate): 34 to 39 tok/s Sources: [https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-cpu/repack.cpp](https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-cpu/repack.cpp), [https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/src/llama-model.cpp](https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/src/llama-model.cpp), [https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-backend.cpp](https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-backend.cpp), [https://github.com/ggml-org/llama.cpp/pull/19738](https://github.com/ggml-org/llama.cpp/pull/19738), [https://github.com/ggml-org/llama.cpp/pull/19706](https://github.com/ggml-org/llama.cpp/pull/19706), [https://github.com/ggml-org/llama.cpp/issues/26110](https://github.com/ggml-org/llama.cpp/issues/26110), [https://github.com/ggml-org/llama.cpp/pull/26135](https://github.com/ggml-org/llama.cpp/pull/26135), [https://github.com/ggml-org/llama.cpp/pull/20596](https://github.com/ggml-org/llama.cpp/pull/20596), [https://chipsandcheese.com/p/amds-ryzen-9950x-zen-5-on-desktop](https://chipsandcheese.com/p/amds-ryzen-9950x-zen-5-on-desktop) Next: Check the CPU's CCD count, measure read bandwidth with threads on one CCD vs both, then a llama-bench thread and CPU-mask ladder and -lm none vs mlock at UD-Q5. ## Vulkan beats ROCm for MoE decode on R9700s; in hybrid Flash runs the gap shrinks to ~16-25% 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-vulkan-vs-rocm-flash Every recent R9700 comparison we found shows llama.cpp's Vulkan backend 1.5-1.8x faster than ROCm/HIP for MoE decode. Fresh 2x R9700 numbers posted today on the Vulkan tensor-parallel PR: Qwen3.6-35B-A3B Q4_K_M decodes at 165.2 tok/s on one card with Vulkan vs 91.5 with ROCm, and 132.6 vs 87.6 with layer split. Dense 27B is a tie in layer split. For Flash-class models with experts in DDR5, the CPU part doesn't depend on the backend, so we estimate ROCm would cost about 16% at UD-Q6, 18% at UD-Q5 and 21-25% at Q4. Our pick for the dual-R9700 Flash lane: Vulkan with layer split. Prefill is the exception: on one R9700, HIP leads at long prompts (pp2048 4635 vs 3675, pp4096 4325 vs 3386, Qwen3-30B-A3B UD-Q4_K_XL), but offloaded-expert prefill is PCIe-bound, so that lead should shrink in hybrid runs (to be measured). Vulkan tensor split (llama.cpp PR #25051) is still open: on 2x R9700 it reaches 45.0 tok/s on dense Qwen3.8-27B Q4_K_M (ROCm tensor 40.2, layer split 27), but for a MoE that fits one card it is slower than a single GPU (144.8 vs 165.2). Whether tensor split works with CPU-resident experts for Flash is untested. A real multi-agent workload on one R9700 also found HIP 1.3-1.4x slower at every concurrency level. The hybrid percentages are estimates from our bandwidth model (GPU share of each decode step 37-64%), not measurements. - Model: Qwen3.6-35B-A3B / Qwen3.8-27B (published); Qwen3.8-Flash-Next (estimate) - Quant: Q4_K_M (published); Q4-Q6 (estimate) - Engine: llama.cpp - Engine version: master + PR #25051 (published) - GPUs: 1-2x Radeon AI PRO R9700 - Context: tg128 / pp512-4096 - Flags: -fa 1 - 35B-A3B Q4_K_M decode, 1 GPU, Vulkan: 165.2 tok/s - 35B-A3B Q4_K_M decode, 1 GPU, ROCm: 91.5 tok/s - 35B-A3B Q4_K_M decode, 2 GPU layer, Vulkan: 132.6 tok/s - 35B-A3B Q4_K_M decode, 2 GPU layer, ROCm: 87.6 tok/s - 27B Q4_K_M decode, 2 GPU tensor, Vulkan: 45 tok/s - 27B Q4_K_M decode, 2 GPU tensor, ROCm: 40.2 tok/s - Est. ROCm decode loss vs Vulkan, Flash UD-Q5 hybrid: 18 % Sources: [https://github.com/ggml-org/llama.cpp/pull/25051](https://github.com/ggml-org/llama.cpp/pull/25051), [https://github.com/ggml-org/llama.cpp/discussions/22463](https://github.com/ggml-org/llama.cpp/discussions/22463), [https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html](https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html), [https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/](https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/) Next: Same-commit Vulkan vs HIP A/B on Flash-Next Q5 with experts in DDR5 on the local desktop; test Vulkan tensor split with CPU-resident experts once the PR merges. ## Parallel slots with experts in DDR5: a 32-token decode cliff in llama.cpp, and what batching buys on 2x R9700 2026-10-24 · Dual R9700 speed · Progress · https://amdai.net/lab-log/#2026-10-24-parallel-slots-offload-cliff In llama.cpp, a MoE expert op whose weights sit in system RAM is moved to a GPU once a ubatch holds 32 or more tokens (GGML_OP_OFFLOAD_MIN_BATCH, default 32, in both the HIP/CUDA and Vulkan backends). In decode, tokens per step = active slots x (1 + draft tokens), so 32 parallel slots, or just 8 slots with 3 MTP drafts each, switch the DDR5-resident layers from CPU reads to PCIe uploads of the used experts. Our bandwidth estimate says that roughly halves aggregate decode at that point; raising the threshold to about 256 avoids it while large prefill batches still offload. Source reading (llama.cpp master 3642238, 2026-10-24): for MUL_MAT_ID the offload batch size is op->ne[2], the tokens in the ubatch; sched_copy_experts() then uploads only the experts the router used (MoE-cache hits come from VRAM), after a synchronous readback of expert ids per layer. Estimates below are bandwidth-only, assume uniform routing (the least overlap between slots, so real sharing should be better), use our measured 55 GB/s DDR5 read and ~23 GB/s achieved on one PCIe 5.0 x8 link, and ignore CPU compute limits, which may cap the CPU path past about 8-16 tokens per step. Qwen3.8-Flash-Next UD-Q5_K_XL, layer split, 23 expert layers in RAM: 34 tok/s at 1 slot, 47 total at 4 slots, 52 at 8, 67 at 31, then about 31 at 32 tokens per step with the default threshold. AP-Q4_K_XL: 60, 97 and 111 at 1, 4 and 8 slots. So parallel slots add only about 1.4-1.5x total throughput at 4-8 slots in this worst-case routing model, and per-agent speed drops fast. Not yet measured on hardware. - Model: Qwen3.8-Flash-Next - Quant: UD-Q5_K_XL, AP-Q4_K_XL (estimates) - Engine: llama.cpp (source reading + bandwidth model) - Engine version: master 3642238 - GPUs: 2x Radeon AI PRO R9700 - Context: 262K f16 KV pool, 8K fill per slot - Flags: -sm layer, -ncmoe per quant, -np N - Default offload threshold (tokens per ubatch): 32 tokens - Est. UD-Q5 aggregate decode, 8 slots: 52 tok/s - Est. UD-Q5 aggregate decode, 31 slots: 67 tok/s - Est. UD-Q5 aggregate decode, 32 tokens/step at default threshold: 31 tok/s - Est. 8 slots x 4 MTP tokens verify step, default vs threshold 256: 1.02 vs 0.46 s Sources: [https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-cuda/ggml-cuda.cu](https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-cuda/ggml-cuda.cu), [https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-vulkan/ggml-vulkan.cpp](https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-vulkan/ggml-vulkan.cpp), [https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/src/llama-context.cpp](https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/src/llama-context.cpp), [https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/tools/batched-bench/README.md](https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/tools/batched-bench/README.md) Next: Measure with llama-batched-bench at 1-48 parallel sequences, default threshold vs GGML_OP_OFFLOAD_MIN_BATCH=256, Q5 first. ## Dual R9700 platform: M.2 slots that steal GPU 2's lanes, and what DDR5 speed and channels are worth 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-dual-r9700-platform-lanes-ddr5 From manufacturer specs: on the ASUS ProArt X870E-Creator, populating M.2_2 drops the second GPU slot to x4; on the Crosshair X870E Apex, M.2_3 gives x8/x4 and M.2_2 plus M.2_3 disables slot 2; on the Crosshair X870E Dark Hero, slot 2 runs PCIe 3.0 x4 until x8/x8 is set in the BIOS. Our calibrated estimate says AM5 RAM tuning (5200 to 6000 MT/s) is worth about +8% decode at UD-Q5, slipping to AMD's stock DDR5-3600 for four DIMMs costs about 20%, and a 4-channel Threadripper TRX50 platform would be worth about +52%. Why it matters: with Flash-class experts in DDR5, prefill is PCIe-bound and Q5/Q6 decode is DDR5-bound, so a silently halved link or a BIOS that fell back to 3600 MT/s costs real speed. The R9700 is a two-slot blower (about 267 x 111 x 39-40 mm), so slot spacing is a non-issue on any board with two CPU-wired x16 slots. Ryzen 9000 has 28 PCIe 5.0 lanes with 4 to the chipset, so x8/x8 is the AM5 maximum; boards like the Strix X870E-E have only one CPU x16 slot. The Apex is 2-DIMM (128 GB max), too small for 147-158 GiB Q5/Q6 Flash GGUFs. TRX50 with Threadripper 9000 runs four channels of DDR5-6400 RDIMM (204.8 GB/s theoretical) and x16/x16 Gen5. Estimated 2x R9700 decode ceilings (layer split, no MTP, 262K f16 KV, 8K fill), all from a bandwidth model calibrated to a measured 55 GB/s read on 4x64 GB DDR5-5200 and scaled at the same efficiency, not measured on the other platforms: UD-Q5_K_XL 27 / 34 / 37 / 52 / 63 tok/s and AP-Q4_K_XL 52 / 60 / 64 / 77 / 85 tok/s for AM5 at 3600 / 5200 / 6000, TRX50 and WRX90. Check with lspci -vv under load that both cards show Width x8 at 32GT/s. - Model: Qwen3.8-Flash-Next GGUFs - Engine: bandwidth model (estimate) + manufacturer specs - GPUs: 2x Radeon AI PRO R9700 - Context: 262K - Est. UD-Q5 decode, AM5 4x64 GB @3600: 27.2 tok/s - Est. UD-Q5 decode, AM5 4x64 GB @5200 (calibrated): 34.1 tok/s - Est. UD-Q5 decode, AM5 4x64 GB @6000: 37 tok/s - Est. UD-Q5 decode, TRX50 4ch @6400: 51.7 tok/s Sources: [https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x870e-creator-wifi/techspec/](https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x870e-creator-wifi/techspec/), [https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM5/ROG_CROSSHAIR_X870E_APEX/E25780_ROG_CROSSHAIR_X870E_APEX_EM_WEB.pdf](https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM5/ROG_CROSSHAIR_X870E_APEX/E25780_ROG_CROSSHAIR_X870E_APEX_EM_WEB.pdf), [https://rog.asus.com/ea/motherboards/rog-crosshair/rog-crosshair-x870e-dark-hero/spec/](https://rog.asus.com/ea/motherboards/rog-crosshair/rog-crosshair-x870e-dark-hero/spec/), [https://rog.asus.com/us/motherboards/rog-strix/rog-strix-x870e-e-gaming-wifi7-neo/wtb/](https://rog.asus.com/us/motherboards/rog-strix/rog-strix-x870e-e-gaming-wifi7-neo/wtb/), [https://www.sapphiretech.com/en/commercial/radeon-ai-pro-r9700](https://www.sapphiretech.com/en/commercial/radeon-ai-pro-r9700), [https://en.wikipedia.org/wiki/Template:AMD_Ryzen_9000_Series](https://en.wikipedia.org/wiki/Template:AMD_Ryzen_9000_Series), [https://www.tweaktown.com/news/105825/msi-releases-amd-agesa-1-2-0-3e-bios-officially-supports-4-64gb-sticks-up-to-256gb-ram/index.html](https://www.tweaktown.com/news/105825/msi-releases-amd-agesa-1-2-0-3e-bios-officially-supports-4-64gb-sticks-up-to-256gb-ram/index.html), [https://www.kingston.com/en/memory/server-memory/memory-population-rules-wrx90-trx50](https://www.kingston.com/en/memory/server-memory/memory-population-rules-wrx90-trx50) Next: Verify both cards train at x8 Gen5 under load and DDR5 runs at its set speed on our dual-R9700 machine. ## Qwen3.8-Flash-Next at Q4 or higher: where the extra GiB in Q5 and Q6 actually go 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-quality-per-gib-q4plus We read the tensor headers of seven Flash-Next GGUFs at 4-bit or more without downloading the weights. Most of the 43.7 GiB that UD-Q5_K_XL adds over UD-Q4_K_XL never touches the GPU: 23.9 GiB is the n-gram table moving from IQ4_NL to Q8_0, which stays in system RAM and is read only a few rows per token. Another 10.5 GiB is the expert down projections moving to Q8_0, and only 9.4 GiB is the gate and up experts moving to Q5_K. On agentionai's published held-out test, their AP-Q5_K_M scores the same as UD-Q4_K_XL (mean KL 0.0771 vs 0.0778) with the same number of GPU-side bytes, and every tensor in it is 4-bit or more. Bytes that matter for decode (everything except the n-gram table): UD-Q4_K_XL 76.8 GiB, AP-Q5_K_M 76.7, UD-Q4_K_XL with IQ4_NL downs (bitlamas DN4) 68.6, AP-Q4_K_XL 67.3, UD-Q5_K_XL 96.7, UD-Q6_K_XL 106.8. Two R9700s hold 64 GiB, so every option spills some experts to DDR5. On Unsloth's own KL table, Q4 to Q5 cuts mean KL by 35% (0.0469 to 0.0304), and Q5 to Q6 cuts it by only 11% more (0.0271). AP-Q5_K_M keeps its trunk at Q6_K instead of Q8_0, so its always-read dense part is about 0.9 GiB smaller. Our bandwidth model puts its decode ceiling on 2x R9700 at about 50 tok/s, against 45 for UD-Q4_K_XL, 34 for UD-Q5_K_XL and 30 for UD-Q6_K_XL. These are estimates only: layer split, no MTP, no expert cache, 55 GB/s measured DDR5. agentionai warns that this model's n-gram table has memorised wikitext, which pushes top-1 agreement 6.9 to 8.6 points higher than on held-out text. That means wikitext KL figures understate how much a quant hurts. Agentionai's lower tiers (AP-Q4_K_M and below) use IQ3 or IQ2 experts and fall below a 4-bit floor. Quality numbers from different publishers use different references and can't be compared with each other. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4/Q5/Q6_K_XL, AP-Q4_K_XL, AP-Q5_K_M, AP-Q5_K_XL, Q4_K_XL-DN4 - Engine: llama.cpp (GGUF header audit + bandwidth model) - GPUs: 2x Radeon AI PRO R9700 (modeled) - Context: 262K f16 KV (modeled) - Flags: layer split, n-gram table in system RAM - UD-Q5_K_XL extra bytes over UD-Q4_K_XL that are n-gram table: 23.9 GiB - GPU-side bytes, AP-Q5_K_M vs UD-Q4_K_XL: 76.7 vs 76.8 GiB - Held-out mean KL, AP-Q5_K_M vs UD-Q4_K_XL (agentionai): 0.0771 vs 0.0778 - Estimated 2x R9700 decode ceiling, AP-Q5_K_M / UD-Q4 / UD-Q5 / UD-Q6: 50 / 45 / 34 / 30 tok/s (estimate) Sources: [https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF](https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF), [https://unsloth.ai/docs/models/qwen3.8-next](https://unsloth.ai/docs/models/qwen3.8-next), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://huggingface.co/bitlamas/Qwen3.8-Flash-Next-Q4_K_XL-DN4](https://huggingface.co/bitlamas/Qwen3.8-Flash-Next-Q4_K_XL-DN4) Next: Measure held-out KL and decode speed for AP-Q5_K_M, AP-Q4_K_XL, UD-Q4_K_XL and UD-Q5_K_XL on 2x R9700 with identical flags. ## Which native 4-bit Qwen3.8-Flash-Next packs for 2x R9700 are really 4-bit or more 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-native-4bit-gate0 We read the tensor headers of tcclaviger's MXFP4-FP8-GPTQ pack for vLLM without downloading the weights. Every tensor class is 4-bit or more: routed experts 4.25 bits per weight (MXFP4), n-gram table 6.53 (int6), attention FP8, MTP head BF16, 110.6 GiB total. Radiance's stock Flash-Next pack keeps 4-bit codebook experts and an int8 trunk, but its recipe stores the MTP draft head at 2 bits (u2), so it fails a strict no-sub-4-bit rule. Per Radiance's docs the 2-bit draft head is only a coarse drafting pass: candidates are re-scored against bf16 weights and the target verifies every token, so output should not change. Turning speculation off doesn't help, because the 2-bit tensor is still in the file. The publisher's 2x R9700 TP2 recipe for the vLLM pack claims 100 tok/s single-request decode and 323 tok/s at 16 concurrent, using 82 GiB of system RAM, with experts that don't fit VRAM held in RAM and the n-gram table streamed from NVMe. Radiance's card implies about 160 tok/s (15 ms per step at about 2.4 accepted tokens). These speeds come from the publishers and we haven't checked them. The quality figures use different references and can't be compared directly: the vLLM pack scores mean KL 0.055 and 95.2% top-1 against BF16, and the Radiance pack 0.079 overall (0.024 on text turns) and 92.9% top-1. - Model: Qwen3.8-Flash-Next - Quant: MXFP4 experts / int6 n-gram / FP8 attention (vLLM); w4 codebook + int8 (Radiance) - Engine: vLLM (tcclaviger image), Radiance - Engine version: HF revisions 6fe3e74 and 8267a67 - GPUs: 2x Radeon AI PRO R9700 - Context: 262K / 200K - Flags: header audit only, no inference run - vLLM pack routed experts: 4.25 bits/weight - vLLM pack n-gram table: 6.53 bits/weight - vLLM pack total size: 110.6 GiB - Radiance pack MTP draft head: 2 bits Sources: [https://huggingface.co/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ](https://huggingface.co/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ), [https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe), [https://hub.docker.com/r/stilldeadcode/vllm-radiance/](https://hub.docker.com/r/stilldeadcode/vllm-radiance/), [https://codeberg.org/StillDeadcode/radiance](https://codeberg.org/StillDeadcode/radiance) Next: Run the vLLM pack's TP2 recipe on two R9700s and measure decode, prefill and memory against our GGUF Q5/Q6 results. ## llama.cpp HIP builds for R9700 (gfx1201) on LLVM 23 / new ROCm need b11847 or later 2026-10-24 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-24-llamacpp-hip-rdna4-llvm23-fix Until llama.cpp PR #30885 merged on 2026-10-23 (in release b11847), HIP builds for RDNA4 (gfx1200/gfx1201) and RDNA3 APUs (gfx1150/gfx1151) failed to compile with LLVM 23 and the newest ROCm, because the bf16 WMMA operands were typed __bf16 instead of short. The fix is bit-identical and needs no version gate. Found by a new release-radar script that scans llama.cpp, vLLM, ROCm, Lemonade and related repos daily for RDNA4/HIP/ROCm changes. If you build llama.cpp with HIP for an R9700 against a fresh toolchain, update to b11847 or newer first. Prebuilt Vulkan builds are not affected by this compile error. - Engine: llama.cpp (HIP backend) - Engine version: b11847 - GPUs: Radeon AI PRO R9700 (gfx1201) Sources: [https://github.com/ggml-org/llama.cpp/pull/30885](https://github.com/ggml-org/llama.cpp/pull/30885), [https://github.com/ggml-org/llama.cpp/issues/30839](https://github.com/ggml-org/llama.cpp/issues/30839), [https://github.com/ggml-org/llama.cpp/releases/tag/b11847](https://github.com/ggml-org/llama.cpp/releases/tag/b11847) Next: Use b11847 or later for the dual-R9700 HIP test lane. ## Why llama.cpp prefill is 10x slower than Strata with Flash experts in DDR5: it's PCIe-bound at the default ubatch 2026-10-24 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-24-offload-prefill-pcie In llama.cpp, any prompt chunk of 32+ tokens copies every expert kept in system RAM over PCIe to a GPU, so prefill speed is roughly ubatch size divided by the time to copy those experts. Our measured baseline (Vulkan, Q5_K_M, --n-cpu-moe 28, 205 tok/s at 36K) works out to about 23 GB/s at the default -ub 512, which is 73% of a PCIe 5.0 x8 link: it was PCIe-bound. Estimated ceiling for UD-Q5_K_XL on two R9700s rises from about 230 tok/s at -ub 512 to about 880-960 at -ub 2048 and 1,700-1,840 at -ub 4096, before GPU compute limits. From llama.cpp master (commit 8e5a58df87, 2026-10-24): the scheduler offloads big-batch ops on CPU-resident weights to the first GPU that accepts them, so with -sm layer on two cards all expert copies use one card's PCIe link while the other link idles (source reading, not yet measured). Under mmap the loader replaces pinned host buffers with plain CPU memory; PR #28223 (keep explicit host overrides) was closed unmerged and PR #26659 is still open, so on stock master only --load-mode none pins the experts. That costs RAM for the whole file (UD-Q4/Q5/Q6_K_XL are 103.7 / 147.4 / 157.5 GiB, including the 27-51 GiB n-gram table), which fits on a 256 GB DDR5 machine. Published evidence: on an RX 9070 XT (gfx1201, Vulkan) with Qwen3.6-35B-A3B Q4_K_M offloaded, going from -b 1024 -ub 512 to -b 8192 -ub 2048 took 48.5K prefill from 705 to 1,566 tok/s, while -ub 4096 was slower (808); on CUDA, pinned experts took Flash-Next UD-Q6_K_XL prefill from 166 to 330-379 tok/s at 26K with decode flat; on a 3090, gpt-oss-120b went from 380 to 2,091 tok/s between -ub 512 and 8192 with decode down 7%. Short prompts pay the full copy too (about 2 s for ~46 GiB of Q5 experts), and raising GGML_OP_OFFLOAD_MIN_BATCH to 2048 cut one 300-token prompt's time to first token from 29 s to 7.7 s on a box with 135 GB of experts. Our ceilings are estimates from measured GGUF byte counts, using UD-Q5_K_XL expert sizes as a stand-in for Q5_K_M. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimates); baseline Q5_K_M (measured) - Engine: llama.cpp master source + llama.cpp Vulkan baseline - GPUs: 2x Radeon AI PRO R9700 (PCIe 5.0 x8 each) - Context: 262K planned, 36K baseline - Flags: -sm layer, --n-cpu-moe 24-28, -ub 512/2048/4096 - Baseline prefill (measured, -ub 512 assumed): 205 tok/s - Implied PCIe copy speed at baseline: 23 GB/s - Q5 ceiling at -ub 2048 (estimate): 880-960 tok/s - Q5 ceiling at -ub 4096 (estimate): 1700-1840 tok/s - gfx1201 Vulkan, ub 512 to 2048 (published): 705 to 1566 tok/s Sources: [https://github.com/ggml-org/llama.cpp/issues/25859](https://github.com/ggml-org/llama.cpp/issues/25859), [https://inovello.dev/writeups/pinned-host-experts-under-mmap/](https://inovello.dev/writeups/pinned-host-experts-under-mmap/), [https://github.com/ggml-org/llama.cpp/pull/28223](https://github.com/ggml-org/llama.cpp/pull/28223), [https://github.com/ggml-org/llama.cpp/pull/26659](https://github.com/ggml-org/llama.cpp/pull/26659), [https://github.com/ggml-org/llama.cpp/pull/19976](https://github.com/ggml-org/llama.cpp/pull/19976), [https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend.cpp](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend.cpp), [https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide](https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide), [https://github.com/gamozolabs/llama.cpp](https://github.com/gamozolabs/llama.cpp), [https://bittide.aicompass.dev/article/c9df2d89-8522-46ec-aa3f-fc1aed5b17d3](https://bittide.aicompass.dev/article/c9df2d89-8522-46ec-aa3f-fc1aed5b17d3), [https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html](https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html) Next: Measure on our two R9700s, Q5 first: -ub 512 vs 2048 vs 4096 with -b 8192, then --load-mode none, RADV's transfer queue, HIP vs Vulkan, a higher offload threshold for short prompts, and whether only one card's link carries the copies. ## Agents on Qwen3.8-Flash-Next: without context checkpoints every turn re-reads the whole prompt 2026-10-23 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-23-hybrid-flash-prompt-cache Flash-Next is a hybrid DeltaNet model, so llama.cpp's --cache-reuse does nothing for it; context checkpoints are its only prompt cache. Published measurements: --ctx-checkpoints 0 gives a 0% cache hit and 20.5 s per follow-up turn, versus 99.5% and 4.2 s with a single checkpoint. One checkpoint hits as often as the default 32 while using far less RAM. Each Flash-Next checkpoint costs 112.6 MiB, plus 2,072 bytes per prefix token when MTP is on (about 555 MiB at 224K). Current llama.cpp flags are -ctxcp/--ctx-checkpoints (default 32), -cms/--checkpoint-min-step (default 8192) and -cram/--cache-ram (default 8192 MiB); --checkpoint-every-n-tokens no longer exists. Hybrid checkpoints can still be invalidated by upstream issue #24055; the fix, PR #25592, was unmerged at last check, and users who cherry-picked it report 35-41 s re-prefills dropping to 0.1-1.3 s. On the client side, anything that changes the start of the prompt (a per-minute timestamp, a mid-session model switch) forces a full re-read; Hermes Agent made its timestamp date-only for this reason. Strata has a separate host-RAM conversation cache: two alternating 57K-token chats went from 9.2 s to 2.1 s prefill on re-entry, at about 1.5 GB of RAM per conversation. These are cited numbers from other rigs; our own dual-R9700 measurement is still to come. - Model: Qwen3.8-Flash-Next - Engine: llama.cpp server / Strata (published reports) - Flags: --ctx-checkpoints, --cache-ram, --conversation-cache-mib - Follow-up turn, 0 checkpoints: 20.5 s - Follow-up turn, 1 checkpoint: 4.2 s - Cache hit, 1 checkpoint: 99.5 % - Checkpoint size without MTP: 112.6 MiB Sources: [https://kksoftwareag.github.io/strix-halo-qwen-3.8-flash-next/memory.html](https://kksoftwareag.github.io/strix-halo-qwen-3.8-flash-next/memory.html), [https://github.com/ggml-org/llama.cpp/issues/18497](https://github.com/ggml-org/llama.cpp/issues/18497), [https://github.com/ggml-org/llama.cpp/pull/25592](https://github.com/ggml-org/llama.cpp/pull/25592), [https://particula.tech/blog/prompt-reprocessing-swa-hybrid-models-kv-cache](https://particula.tech/blog/prompt-reprocessing-swa-hybrid-models-kv-cache), [https://github.com/NousResearch/hermes-agent/pull/27675](https://github.com/NousResearch/hermes-agent/pull/27675), [https://thomaswildetech.com/software-development/ai-agents/strata/](https://thomaswildetech.com/software-development/ai-agents/strata/) Next: Measure per-turn cache hit and turn latency for Flash on our dual R9700s, Strata versus llama.cpp with and without checkpoint flags. ## The fair dual-3090 target for Qwen3.8-Flash-Next at 4-bit and up: 37-41 tok/s 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-3090-gap-q4plus Most viral dual-RTX-3090 Flash-Next speeds (111-160 tok/s) are 3-bit or lower, or don't state a quant. At 4-bit and above, the best published 2x 3090 llama.cpp result is 37-41 tok/s (49 on code output) with UD-Q4_K_XL at full 261K f16 KV, using an expert VRAM cache plus MTP. That box started at 17 tok/s without the cache, close to our own 16.9-20.3 llama.cpp baseline, so the gap is software, not hardware. Plain banded expert offload at UD-IQ4_XS on 2x 3090 gives 35-38 tok/s. Excluded from the comparison: Strata IQ3_XXS 111 tok/s, tqLLM 77 tok/s at 128K (IQ3_S, one 3090, CUDA only, no multi-GPU yet), EXL3 2.5 bpw 38.6 tok/s, and 150-160 tok/s posts with no quant named. On the 3090 box the cache hit rate depended on slot count, not quant (Q4 and Q6 both 84-85% at 135 slots); Q4 won only by fitting more slots, and -ub 512 freed about 50 slots of VRAM versus 2048. Two R9700s have 64 GB of VRAM against 48 GB; our estimate is about 302 / 236 / 213 cache slots per layer at Q4 / Q5 / Q6 with MTP, against 150 on the 3090 pair (estimate, R9700 compute buffers unmeasured). Reading the merged llama.cpp expert-cache code (PR #29887): every selected expert gets its own real slot, batches up to 32 tokens use the cache, and the draft model's context has it off, so the duplicate-id bug reported on the older draft cache PR (#27861) should not apply to MTP + cache on master. Also from the 3090 write-up: q8_0 KV was slower (-4.8% short, -18% at 131K), MTP acceptance collapsed to ~20% at 131K depth where an n-gram draft won, and DIMM thermal throttling halved decode after a few minutes until a fan was added. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL - Engine: llama.cpp with expert cache + MTP - GPUs: 2x RTX 3090 (comparison, published) - Context: 261888, f16 KV - Flags: experts in host RAM, -ub 512, MTP draft n-max 3 - Best >=4-bit 2x 3090 llama.cpp decode (thinking prompt): 37-41 tok/s - Same, code emission: 49 tok/s - Same box before expert cache (UD-Q6_K_XL): 17 tok/s - 2x 3090 banded offload, UD-IQ4_XS: 35-38 tok/s - Our 2x R9700 llama.cpp baseline (Q5_K_M, no cache): 16.9-20.3 tok/s - Est. R9700 pair cache slots per layer, Q5 + MTP: 236 slots (estimate) - 3090 pair cache slots per layer, Q4 + MTP: 150 slots Sources: [https://inovello.dev/writeups/qwen3-flash-next-2x3090-q4-mtp/](https://inovello.dev/writeups/qwen3-flash-next-2x3090-q4-mtp/), [https://www.reddit.com/r/LocalLLM/comments/1w069fh/qwen38flashnext_on_2x_rtx_3090s_3343_toks/](https://www.reddit.com/r/LocalLLM/comments/1w069fh/qwen38flashnext_on_2x_rtx_3090s_3343_toks/), [https://x.com/geldeki/status/2095560770163458426](https://x.com/geldeki/status/2095560770163458426), [https://x.com/coffeecup2020/status/2113265219283660816](https://x.com/coffeecup2020/status/2113265219283660816), [https://github.com/r0b0tlab/qwen38-flashnext-exl3](https://github.com/r0b0tlab/qwen38-flashnext-exl3), [https://github.com/ggml-org/llama.cpp/pull/29887](https://github.com/ggml-org/llama.cpp/pull/29887), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-moe-cache.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-moe-cache.cpp) Next: Run the same recipe on two R9700s with llama.cpp master (expert cache sized per card, -ub 512, MTP on and off), Q5 first. The target is beating 41 tok/s at full 262K context with Q5. ## Dual R9700 multi-GPU traps: what hangs and the fixes that work 2026-10-23 · Dual R9700 speed · Progress · https://amdai.net/lab-log/#2026-10-23-dual-r9700-known-traps Cited roundup for two or more R9700s. vLLM tensor parallel (tp=2) deadlocks with both cards at 100% on ROCm 7.2.x RCCL; NCCL_PROTO=Simple is the fix (an AMD engineer notes the proposed RCCL patch is equivalent), while --enforce-eager, NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce do not help. vLLM v0.21.0 adds a shared-memory broadcast hang on big models; v0.20.2 works. llama.cpp's ROCm build can hold an idle card at 100% and about 95 W; GPU_MAX_HW_QUEUES=1 brought it to 22 W. Also seen: a Ryzen iGPU visible to ROCm makes llama.cpp split onto it and segfault (hide it or pin the device); an open ROCm 7.14 report shows a llama.cpp generation hang that ends in an MES queue failure and a GPU reset failure until reboot, while Vulkan passes on the same machine; a missing TensileLibrary_lazy_gfx1201.dat means an incomplete rocBLAS install. For context, Level1Techs measured Qwen3.6-35B-A3B FP8 decode at 17.4 tok/s with tp=2 and 15.7 with tp=4 on R9700s (vLLM 0.20.2, NCCL_PROTO=Simple), so more cards did not speed up a small-active MoE at batch 1. Whether ROCm 7.14's newer RCCL avoids the deadlock without the env var is not confirmed, so we keep NCCL_PROTO=Simple pinned. We wrote a read-only pre-flight script that checks a machine for all of these; our own A/B of GPU_MAX_HW_QUEUES on decode speed is still to come. - Engine: vLLM ROCm images, llama.cpp HIP and Vulkan - Engine version: per source - GPUs: 2-4x Radeon AI PRO R9700 (gfx1201), published reports - Idle draw, llama.cpp HIP, GPU_MAX_HW_QUEUES unset: 95 W - Idle draw, GPU_MAX_HW_QUEUES=1: 22 W - 35B-A3B FP8 decode, vLLM tp=2 (Level1Techs): 17.4 tok/s - 35B-A3B FP8 decode, vLLM tp=4 (Level1Techs): 15.7 tok/s Sources: [https://github.com/vllm-project/vllm/issues/40980](https://github.com/vllm-project/vllm/issues/40980), [https://github.com/ROCm/rocm-systems/issues/5480](https://github.com/ROCm/rocm-systems/issues/5480), [https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504](https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504), [https://github.com/ROCm/ROCm/issues/5706](https://github.com/ROCm/ROCm/issues/5706), [https://github.com/lemonade-sdk/llamacpp-rocm/issues/96](https://github.com/lemonade-sdk/llamacpp-rocm/issues/96), [https://github.com/ROCm/legacy-rocm-build/issues/6630](https://github.com/ROCm/legacy-rocm-build/issues/6630), [https://www.reddit.com/r/LocalLLaMA/comments/1vjtokl/dual_radeon_ai_pro_r9700_server_much_slower_than/](https://www.reddit.com/r/LocalLLaMA/comments/1vjtokl/dual_radeon_ai_pro_r9700_server_much_slower_than/) Next: Measure GPU_MAX_HW_QUEUES=1 against unset on our dual-R9700 Flash config for decode speed, and run a small tp=2 vLLM smoke test with NCCL_PROTO=Simple before the vLLM lane. ## Flash decode ceiling on dual R9700 + DDR5: about 34 tok/s at Q5 without MTP, so cutting DDR5 reads per token is the main lever 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-flash-decode-roofline A bandwidth-only ceiling (estimate) built from measured GGUF byte counts, the R9700's 640 GB/s spec and our measured 55 GB/s DDR5 read speed puts Qwen3.8 Flash on two R9700s (layer split, experts that don't fit in VRAM kept in DDR5, no MTP) at about 45 / 34 / 30 tok/s for UD-Q4 / Q5 / Q6_K_XL, or 63 / 44 / 37 with ideal tensor split. Our llama.cpp baseline (16.9-20.3 tok/s) runs at 55-66% of its ceiling. Strata's measured 52-65 tok/s at Q5 beats the no-MTP ceiling, so it can be reading at most 0.15-0.34 GiB of experts from DDR5 per generated token instead of 0.86 GiB, which points to MTP plus experts served from VRAM. Per token, time = bytes read from VRAM / GPU bandwidth + expert bytes read from DDR5 / DDR5 bandwidth. Dense weights read every token (attention and DeltaNet, lm_head, router, shared expert): about 4.46 GiB at UD-Q5_K_XL; routed experts: 10 of 512 per layer. With experts in DDR5, DDR5 is 57-77% of each token's time at Q5/Q6, so ideal tensor split helps Q4 by about 39% but Q5/Q6 only by 23-27%. Q4 to Q5 costs about 24% of ceiling, Q5 to Q6 about 13%. If every Q5 expert fit in VRAM (four cards), the ceiling would be about 69 tok/s on R9700 vs about 101 on RTX 3090: the 3090's bandwidth edge only shows once DDR5 is out of the loop. Assumptions: GPU decode at 75% of peak bandwidth, 2.5 GiB compute reserve per card, 262K f16 KV read to an 8K fill, tensor split modeled with no all-reduce cost. It ignores CPU compute limits, kernel launch overhead and PCIe, so real numbers land below these ceilings. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL - Engine: model (bandwidth roofline), checked against measured llama.cpp Vulkan and Strata 0.2.3 runs - GPUs: 1-2x Radeon AI PRO R9700 - Context: 262K f16 KV, 8K fill - Flags: DDR5 4x64 GB at 5200, measured 53.8-56.2 GB/s read - Q4 ceiling, 2 GPUs layer split, no MTP (estimate): 45.1 tok/s - Q5 ceiling, 2 GPUs layer split, no MTP (estimate): 34.2 tok/s - Q6 ceiling, 2 GPUs layer split, no MTP (estimate): 29.8 tok/s - Q5 ceiling, 2 GPUs ideal tensor split (estimate): 43.5 tok/s - llama.cpp baseline vs its ceiling: 55-66 % - Max DDR5 expert reads per token implied by Strata Q5: 0.15-0.34 GiB Sources: [https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html](https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622](https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622) Next: Check the model's slope with a llama.cpp --n-cpu-moe and thread sweep, and run Strata Q5 with MTP on vs off to split MTP gains from expert-cache gains. ## Qwen3.6-35B-A3B Q5 at full 262K context on one R9700 leaves under 2.75 GB for compute buffers 2026-10-23 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-23-35b-one-card-vram-budget Exact tensor sizes read from the GGUF headers: UD-Q5_K_XL has 22.26 GB of routed experts and 1.99 GB of other GPU weights, and its f16 KV cache at 262,144 tokens is 5.00 GB (10 of 40 layers use full attention). That is 29.25 GB on a 32 GB card, and this config runs fully on one R9700 with nothing offloaded, so llama.cpp's compute buffers fit in at most 2.75 GB. Fit calculators that assume 3 GB or more per card will wrongly say this model needs offload. The 0.50 GB token embedding table stays in system RAM (llama.cpp's default for input-layer tensors), so it doesn't count against VRAM. KV size: 10 full-attention layers x 2 KV heads x (256 + 256) dims x 2 bytes x 262,144 tokens = 5.00 GB. GB here means GiB. Also useful for anyone building tools: Hugging Face allows browser CORS Range requests on GGUF files (the resolve URL permits the Range header and its CDN returns 206 with an open CORS header), so a web page can read these headers without downloading weights or running a proxy. - Model: Qwen3.6-35B-A3B - Quant: Unsloth UD-Q5_K_XL - Engine: llama.cpp (Vulkan) - GPUs: 1x Radeon AI PRO R9700 (32 GB) - Context: 262144 - Flags: f16 KV, all layers on GPU - Routed experts: 22.26 GB - Other GPU weights: 1.99 GB - f16 KV at 262K: 5 GB - Max left for compute buffers: 2.75 GB Sources: [https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) Next: Measure actual VRAM in use on the card to pin the compute-buffer number exactly, and use it as the default in amdai.net's fit calculator. ## Measured on our dual R9700s: Strata UD-Q5_K_XL runs Qwen3.8 Flash at about 52-65 tok/s, 3x llama.cpp and faster than Q6 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-measured-flash-q5-vs-q6-strata On the local desktop (two R9700s plus DDR5 expert offload, 256K context), Strata 0.2.3 with Unsloth UD-Q5_K_XL, MTP and int8 KV decoded about 52 tok/s on a short story and about 65 tok/s on code (range 48-68), with 2043-2514 tok/s prefill at 36K. That is about 3x the decode and about 10x the prefill of our llama.cpp Vulkan Q5_K_M baseline (16.9-20.3 tok/s decode, 205 prefill). UD-Q6_K_XL on Strata was slower: median about 40 tok/s warm decode. Q5 stays the sweet spot; Q4 is not yet measured on our machine. Baseline: llama.cpp v0.6.0 Vulkan, Q5_K_M, --n-cpu-moe 28, about 23 GB VRAM per card; short-context decode 16.9-20.3 tok/s, at 36K decode 20.0 / prefill 205. Strata 0.2.3.1 (engine 0.1.48 HIP), UD-Q6_K_XL, MTP, int8 KV, 256K: warm decode 31-57 (median ~40); at 36K decode 22.8-51.1 / prefill 1204-1833; at 131K decode 35.0-47.4 / prefill 1242-1864; VRAM 29.8/29.6 GB, RAM 126.9 GB. Strata 0.2.3, UD-Q5_K_XL, MTP, int8 KV, 256K: story ~52, code ~65 (48-68); at 36K decode 43.9-52.4 / prefill 2043-2514; at 131K decode 47.6-54.3 / prefill 2519-2882; VRAM 29.8/29.6 GB, RAM ~112 GB, MTP acceptance 70-75%, swap flat. --tune quick kept the defaults. Q5 also used about 15 GB less system RAM than Q6 (about 112 vs 126.9 GB). Command: strata launch --model ud-q5_k_xl --family unsloth --context 256k --kv int8 --mtp on --gpu 0,1 --parallel 1 --set STRATA_ROUTE_TAIL_SKIP=0 - Model: Qwen3.8 Flash - Quant: UD-Q5_K_XL, UD-Q6_K_XL (baseline Q5_K_M) - Engine: Strata 0.2.3 / 0.2.3.1 (engine 0.1.48 HIP); baseline llama.cpp v0.6.0 Vulkan - GPUs: 2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write) - Context: 256K (Strata) - Strata UD-Q5_K_XL decode, code: 65 tok/s - Strata UD-Q5_K_XL decode, short story: 52 tok/s - Strata UD-Q5_K_XL decode at 131K: 47.6-54.3 tok/s - Strata UD-Q5_K_XL prefill at 131K: 2519-2882 tok/s - Strata UD-Q6_K_XL warm decode (median): 40 tok/s - llama.cpp Vulkan Q5_K_M baseline decode: 16.9-20.3 tok/s - llama.cpp Vulkan baseline prefill at 36K: 205 tok/s - MTP acceptance (Q5): 70-75 % Next: Measure Q4 (UD-Q4_K_XL) on the same Strata setup to confirm the 2-7% Q5 penalty we expect from published numbers. ## RDNA4 llama.cpp fork on dual R9700s: 1.3-1.6x baseline decode with MTP, but half of Strata and 10-15x slower prefill 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-measured-rdna4-fork-vs-strata The RDNA4 llama.cpp fork (commit 185252d, ROCm 7.2.0, gfx1201, -sm layer -ts 1,1, q8 KV, flash attention) reached 22-26 tok/s on a story and 32 tok/s on code with MTP (shared Q8_0 head, n-max 3, -t16). That beats our llama.cpp Vulkan baseline by 1.3-1.6x but is about half of Strata UD-Q5_K_XL, and its prefill (144-175 tok/s) is 10-15x slower. Thread count matters: without MTP, -t32 gave only 8.9-9.3 tok/s because the CPU was oversubscribed; -t16 gave 20.3-20.5. With MTP at -t16: story 22-26, code 32, VRAM 29.9/28.2 GB. At 140K context: 36K decode 22.8 / prefill 175; 131K decode 16.2 / prefill 144. Strata at the same 131K depth: decode 47.6-54.3 / prefill 2519-2882. - Model: Qwen3.8 Flash - Engine: RDNA4 llama.cpp fork, commit 185252d, ROCm 7.2.0 (gfx1201) - GPUs: 2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write) - Context: 140K - Flags: -sm layer -ts 1,1, q8 KV, FA, MTP shared Q8_0 head n-max 3, -t16 - Fork + MTP decode, code: 32 tok/s - Fork + MTP decode, story: 22-26 tok/s - Fork no MTP, -t16: 20.3-20.5 tok/s - Fork no MTP, -t32 (oversubscribed): 8.9-9.3 tok/s - Fork decode / prefill at 131K: 16.2 / 144 tok/s Next: Use -t16 (physical cores minus headroom), not all threads, for hybrid CPU-expert runs. ## A system-wide HSA_OVERRIDE_GFX_VERSION=12.0.0 segfaults llama.cpp in op_repeat on R9700; set 12.0.1 per process 2026-10-23 · Dual R9700 speed · Failed · https://amdai.net/lab-log/#2026-10-23-hsa-override-segfault With HSA_OVERRIDE_GFX_VERSION=12.0.0 set system-wide, the RDNA4 llama.cpp fork (ROCm 7.2.0) segfaulted in op_repeat on our dual R9700s. Setting HSA_OVERRIDE_GFX_VERSION=12.0.1 (matching the R9700's gfx1201) for just the llama.cpp process fixed it. The R9700 is gfx1201, so the 12.0.0 value does not match the card. Remove any global override (for example in /etc/environment or a shell profile) and pass HSA_OVERRIDE_GFX_VERSION=12.0.1 only to the process that needs it, so other ROCm apps are not affected. - Engine: RDNA4 llama.cpp fork, commit 185252d, ROCm 7.2.0 - GPUs: 2x Radeon AI PRO R9700 (gfx1201) Next: Fixed; no further action. ## R9700 card tuning for decode: power caps barely matter, memory overclock backfires 2026-10-23 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-23-r9700-card-tuning-decode From published R9700 measurements: decode is memory-bound and draws about 180-190 W under a 300 W cap, so raising the cap doesn't add tok/s. A -120 mV undervolt gave +7.7-8.9% prefill but only +1.3% decode. On amdgpu DKMS 6.19.4 any memory overclock above the 1258 MHz top step is accepted, then pins memory lower and cost 6.6% decode in a clean A/B. Undervolt pays only when the card is actually power-limited (sustained prefill near 300 W). -150 mV was stable at 300 W but crashed with a page fault at 330 W on a 32K prompt after a 2K prompt passed, so short benchmarks don't prove stability. Verify memory clocks with pp_dpm_mclk, not OD_MCLK, which reads back the requested value even when it is ignored. The common 210 W cap with about -75 mV is reported as ~30% lower peak power for a few percent speed loss (community self-reports). The same pattern shows on datacenter GPUs: on H200, decode never hit a power cap and clock locking saved up to 32% decode energy at minimal throughput loss. PCIe ASPM performance mode was reported +10.8% decode on a dense 27B and ~0% on MoE. Not yet measured on our dual-R9700 hybrid-offload setup. - Model: Qwen3.6-27B Q4_K_M (undervolt and memory-clock tests) - Engine: llama.cpp (published) - GPUs: Radeon AI PRO R9700 (published single-card tests) - Decode power draw under 300 W cap: 180-190 W - -120 mV undervolt, prefill pp32768: 8.9 % - -120 mV undervolt, decode tg32: 1.3 % - Memory OC above 1258 MHz, decode tg32: -6.6 % - H200 decode energy saved by clock locking (up to): 32 % Sources: [https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md](https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md), [https://www.reddit.com/r/ROCm/comments/1vuu58x/](https://www.reddit.com/r/ROCm/comments/1vuu58x/), [https://www.reddit.com/r/ROCm/comments/1vr5j5e/](https://www.reddit.com/r/ROCm/comments/1vr5j5e/), [https://github.com/ggml-org/llama.cpp/discussions/21043](https://github.com/ggml-org/llama.cpp/discussions/21043), [https://arxiv.org/abs/2605.11999](https://arxiv.org/abs/2605.11999) Next: A/B test power caps, a locked memory clock, COMPUTE profile and PCIe ASPM on dual R9700s with Flash experts in DDR5, where idle gaps between expert steps may make clock ramp-up matter more. ## Community reports: the fastest 4-bit Flash numbers on R9700s come from native engines, not GGUF 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-x-community-4bit-native A scan of public R9700 posts on X (378 in the last 7 days) found the biggest Qwen3.8-Flash-Next numbers at 4-bit or above come from non-GGUF engines: about 228 tok/s single stream on two R9700s with Radiance's 4-bit experts, and about 200 tok/s on four R9700s with vLLM + r9700-stack MXFP4, versus 67 on llama.cpp on the same four cards. All figures are self-reported by their posters and unverified here. Headline dual-R9700 Flash results of 216-272 tok/s (vLLM + Paiton W3A8) and 130 tok/s (SGLang, EXL3 3.05 bpw) are 3-bit, so they are excluded from our comparisons, which only cover Q4 and up. Other useful reports: splitting a model that fits on one card across two R9700s cost about 7% decode in llama.cpp (75.4 to 70.3 tok/s, Qwen3.6-35B-A3B Q4_K_M), and a chipset PCIe 4.0 x4 slot cost about 12% (66.2 tok/s). For reference, our own measurement on one R9700 plus DDR5 is 32.6 tok/s for Qwen3.8 Flash UD-Q6_K_XL on Strata versus 16 on llama.cpp. We plan to add a 4-bit native lane (vLLM MXFP4 at TP2 and Radiance W4) to the dual-R9700 sweep after confirming every expert tensor is at least 4-bit. - Model: Qwen3.8-Flash-Next - Quant: W4 / MXFP4-FP8 (4-bit class) - Engine: Radiance 1.4.0; vLLM + r9700-stack (as reported) - GPUs: 2x and 4x Radeon AI PRO R9700 (as reported) - Context: up to 256K (as reported) - Radiance W4, 2x R9700, single stream (reported): 228 tok/s - Radiance W4, 2x R9700, 16 concurrent (reported): 945 tok/s - vLLM + r9700-stack MXFP4 TP4, 4x R9700, single stream (reported): 200 tok/s - llama.cpp, same 4x R9700 (reported): 67 tok/s - Public R9700 posts on X, last 7 days: 378 posts Sources: [https://x.com/ECander84852/status/2110182801639408108](https://x.com/ECander84852/status/2110182801639408108), [https://x.com/dad__vibes/status/2111137765396210053](https://x.com/dad__vibes/status/2111137765396210053), [https://x.com/dad__vibes/status/2111573866996977673](https://x.com/dad__vibes/status/2111573866996977673), [https://x.com/dad__vibes/status/2111908353802998061](https://x.com/dad__vibes/status/2111908353802998061), [https://x.com/sincoice/status/2113225429679210835](https://x.com/sincoice/status/2113225429679210835) Next: Confirm Radiance and r9700-stack expert formats are at least 4-bit, then measure both at TP2 against the best GGUF Q5 result. ## llama.cpp now caches offloaded MoE experts in VRAM, and Flash can use tensor split 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-llamacpp-moe-expert-cache llama.cpp master added --moe-cache-mib (merged Oct 7, 2026; per-GPU multi-GPU support Oct 8), an LRU cache in VRAM for experts kept in system RAM. On Qwen3.8-Flash-Next Q4_0 with 2x RTX 4090, the author measured decode going from 34.78 to 64.39 tok/s (1.85x, 95.8% cache hits), with prompt speed at 0.73x in that all-cache setup and 0.91x with a smaller cache. A single-R9700 test on Windows showed the cache only paying off near 94% hits. The current source also allows --split-mode tensor for the Flash architecture (qwen4exp) and Qwen3.5/3.6 MoE files, so 'tensor mode does not work on MoE' is too broad: it is blocked for specific architectures such as DeepSeek2, GLM-DSA and MiniMax. Estimate, not measured: two R9700s at 262K context with f16 KV leave about 46.8 GiB for experts (assuming 3 GiB per card for compute buffers). An offline LRU trace study posted on the PR suggests that cached share would give roughly 97.6% / 95.2% / 94.1% hits for UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL, all in the range where the cache paid off on an R9700. On ROCm, tensor mode's all-reduce is host-staged; a community direct-P2P patch measured +7.5% to +11.2% decode on 2x R9700 with dense Q8_0 models. Linux R9700 pairs report about 28 GB/s GPU-to-GPU and 21.5 GB/s CPU-to-GPU copy bandwidth. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (planned) - Engine: llama.cpp master (HIP) - GPUs: 2x Radeon AI PRO R9700 (planned target) - Cited: 2x RTX 4090 decode, master --fit: 34.78 tok/s - Cited: 2x RTX 4090 decode, -cmoe + 15000 MiB cache: 64.39 tok/s - Cited: cache hit rate at that setting: 95.8 % - Estimated expert VRAM budget, 2x R9700 at 262K f16: 46.8 GiB Sources: [https://github.com/ggml-org/llama.cpp/pull/29887](https://github.com/ggml-org/llama.cpp/pull/29887), [https://github.com/ggml-org/llama.cpp/pull/30112](https://github.com/ggml-org/llama.cpp/pull/30112), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp), [https://github.com/ggml-org/llama.cpp/pull/25028](https://github.com/ggml-org/llama.cpp/pull/25028), [https://github.com/JohnTDI-cpu/llama-hip-p2p-allreduce](https://github.com/JohnTDI-cpu/llama-hip-p2p-allreduce), [https://forum.level1techs.com/t/does-the-lrlink-pcie-bridge-fix-rccls-broken-p2p-on-dual-r9700-update-yes-it-does/249623](https://forum.level1techs.com/t/does-the-lrlink-pcie-bridge-fix-rccls-broken-p2p-on-dual-r9700-update-yes-it-does/249623) Next: Measure UD-Q5_K_XL at 262K on two R9700s: the current -ncmoe baseline vs --fit with a partial cache vs all experts cached, with real hit rates, then Q4 and Q6 and a tensor-split check. ## Qwen3.8-Flash-Next: the n-gram table is 27-51 GiB and belongs in DDR5 2026-10-23 · Idea lab · Result · https://amdai.net/lab-log/#2026-10-23-ngram-table-vram-budget Reading the GGUF headers shows the model's n-gram embedding table (per_layer_token_embd) is a single lookup tensor of 26.85 GiB at UD-Q4_K_XL (IQ4_NL) and 50.70 GiB at both UD-Q5_K_XL and UD-Q6_K_XL (Q8_0). Routed experts are 71.73, 91.61 and 101.75 GiB. Since only a few table rows are read per token, the VRAM question on two R9700s (64 GiB) is really only about experts. Q5 and Q6 differ by just ~10 GiB of experts. llama.cpp master tags per_layer_token_embd as an input-layer get-rows tensor, so it stays on the CPU like token_embd. Where Strata puts it is not yet checked. Estimated -ncmoe starting points at 262K context with f16 KV: 17 for Q4, 24 for Q5, 26 for Q6 (assumes 3 GiB compute reserve per GPU and KV from the model card's 12 sparse-attention layers, about 6.75 GiB at 262K). These start points are estimates until measured. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL - Engine: GGUF header analysis (no inference) - GPUs: 2x Radeon AI PRO R9700 (planned target) - N-gram table, UD-Q4_K_XL: 26.85 GiB - N-gram table, UD-Q5_K_XL / Q6_K_XL: 50.7 GiB - Routed experts, UD-Q4_K_XL: 71.73 GiB - Routed experts, UD-Q5_K_XL: 91.61 GiB - Routed experts, UD-Q6_K_XL: 101.75 GiB Sources: [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp) Next: Check n-gram table placement in Strata and run an -ncmoe sweep around the estimated start points on the local desktop. ## Earlier dual ROCm + DDR5 offload run judged not viable 2026-10-23 · Dual R9700 speed · Failed · https://amdai.net/lab-log/#2026-10-23-ddr5-offload-failure On the local desktop, Qwen Flash UD-Q6_K_XL on llama.cpp with DDR5 expert offload ran at about 16 tok/s, versus about 32.6 tok/s on Strata v0.1.42 at 262K context. Back then, dual ROCm plus DDR5 offload in llama.cpp was judged not viable. Likely causes from research: Q6 instead of Q5, no VRAM expert cache or MTP in stock llama.cpp for HIP, and an older Strata version (newer releases report 1.34-1.40x decode from keeping every expert in RAM). - Model: Qwen3.8-Flash-Next (Qwen Flash) - Quant: UD-Q6_K_XL - Engine: llama.cpp (ROCm) and Strata - Engine version: Strata v0.1.42 - GPUs: 2x Radeon AI PRO R9700 - Context: 262K - Flags: DDR5 expert offload - llama.cpp decode: 16 tok/s - Strata v0.1.42 decode: 32.6 tok/s Sources: [https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md) Next: Retest with UD-Q5_K_XL, Strata 0.2.3 and the RDNA4 llama.cpp fork with MTP. ## Cited comparison: dual 3090 vs dual R9700 at Q4 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-3090-vs-r9700 Same model and quant (Qwen3.8-Flash-Next UD-Q4_K_XL): dual RTX 3090 on the llama.cpp moe-cache fork posts 39.28 tok/s decode; dual R9700 on Strata posts 64 tok/s (different host rigs). The gap on our rig is software and version, not hardware. A 3090 has more memory bandwidth (936 vs 640 GB/s), yet R9700s win when the engine keeps experts in RAM well. Stock llama.cpp on one R9700 with --n-cpu-moe manages only 18-20 tok/s at Q4 on the same model (published). llama.cpp lacks a VRAM expert cache on HIP; the moe-cache work is CUDA-only and unmerged. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL - Engine: llamacpp-club3090 moe-cache (3090) vs Strata (R9700) - Engine version: per sources - GPUs: 2x RTX 3090 vs 2x Radeon AI PRO R9700 - Context: 204,800 (3090) / 40K (R9700) - Flags: per sources - 2x RTX 3090 decode: 39.28 tok/s - 2x R9700 decode (Strata): 64 tok/s - 1x R9700 decode (stock llama.cpp): 18-20 tok/s Sources: [https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md](https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md), [https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md), [https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622](https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622) Next: Reproduce on the local desktop with Strata 0.2.3 and the RDNA4 Flash-Next llama.cpp fork. ## Sweet-spot quant for Flash models on dual R9700: UD-Q5_K_XL 2026-10-23 · Dual R9700 speed · Result · https://amdai.net/lab-log/#2026-10-23-sweet-spot-q5 From published R9700 measurements: Q5_K_XL is almost as good as Q6_K_XL (mean KL vs Q8_0 0.038 vs 0.033, same top token 94.55% vs 94.61%) and only 2-7% slower than Q4_K_XL on two R9700s (60 vs 64 tok/s on short story, 94 vs 96 on code). Q4 is the speed fallback; Q6 is dominated. Qwen3.8-Flash-Next file sizes: UD-Q4_K_XL 103.7 GiB, UD-Q5_K_XL 147.4 GiB, UD-Q6_K_XL 157.5 GiB. All fit in 64 GiB VRAM plus DDR5 with experts spilled to RAM. GLM-5.3-Flash Q6_K_XL (271.8 GiB) is too large for comfort, so Q5 is its ceiling. DeepSeek-V4-Flash has no Q5/Q6 GGUF; UD-Q4_K_XL is the pick. Published numbers come from an EPYC 8-channel DDR4 host, not this rig. - Model: Qwen3.8-Flash-Next - Quant: UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL - Engine: Strata (published) - Engine version: per source - GPUs: 2x Radeon AI PRO R9700 - Context: 40K - Flags: layer split, 8-bit KV, MTP, every expert in RAM - Q5_K_XL decode, 2x R9700 (story): 60 tok/s - Q4_K_XL decode, 2x R9700 (story): 64 tok/s - Q5_K_XL mean KL vs Q8_0: 0.038 - Q6_K_XL mean KL vs Q8_0: 0.033 Sources: [https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md](https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md), [https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF), [https://unsloth.ai/docs/models/qwen3.8-next](https://unsloth.ai/docs/models/qwen3.8-next) Next: Run the ordered Q5 benchmark sequence on the local desktop, with Q4 and Q6 as secondary runs.