Lab log

Case limits checked on maker pages: Fractal Torrent takes a 360 radiator only in front or bottom and air coolers up to 188 mm; Fractal Meshify 2 takes a 360 on top or front and air coolers up to 185 mm; Phanteks Enthoo Pro 2 Server Edition takes radiators on top, front or side and air coolers up to 195 mm; Thermalright Peerless Assassin 120 SE (155 mm) and Noctua NH-D15 G2 (168 mm) fit all three; ARCTIC's Liquid Freezer III Pro 360 needs 63 mm of radiator clearance, and ARCTIC warns that some boards' oversized M.2_1 heatsinks clash with it. with MoE expert offload, token generation is limited by memory and PCIe traffic rather than CPU compute, so AMD's liquid recommendation is about peak boost, not decode speed; we have not measured cooler choice on our own cards.

Stack

Spec review: amd.com product pages for 9600X/9900X/9950X; Fractal, Phanteks, Thermalright, Noctua and ARCTIC product pages; published llama.cpp MoE offload guides

• https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-5-9600x.html
• https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9900x.html
• https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9950x.html
• https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/
• https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/
• https://phanteks.com/product/enthoo-pro-2-server-edition-tg/
• https://www.thermalright.com/product/peerless-assassin-120-se/
• https://noctua.at/en/nh-d15-g2/specification
• https://www.arctic.de/en/Liquid-Freezer-III-Pro-360/ACFRE00180A
• https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0

Next Add a CPU-cooler line to the amdai.net builder parts list.

In a published 4x R9700 build, case fans on the stock CPU-following curve let the card area reach its 68 C warning and pause work; full-speed case fans cooled the card area from 67 C to 57 C but changed GPU junction temperature by only 1-4 C, because each card's own blower cools its GPU. tuning the case fans to follow card power and temperature kept the same GPU temperatures at 26% lower fan speed. owners stacking R9700 blowers with no gap report it works, but one owner found the ASUS Turbo's fan is not recessed and may choke without a slot gap, so use matching cards. slot spacing for the ASUS ProArt X870E/B850 boards and the ASRock X870E Taichi was not verified here; check the manual layout or choose an 8+ slot case. checked case specs: Fractal Torrent (7 slots, 2x 180 mm front + 3x 140 mm bottom fans included, PSU up to 230 mm), Fractal Meshify 2 (7 slots, E-ATX, PSU up to 250 mm), Phanteks Enthoo Pro 2 Server Edition (11 slots, PSU up to 240 mm). temperatures are published by others, not measured on our cards.

Stack

Spec review: R9700 board-partner spec sheets, ASRock/Gigabyte board pages and owner reports, Fractal and Phanteks case pages, one published 4x R9700 build guide

• https://forum.level1techs.com/t/asrock-x870-taichi-creator-dual-gpu/243897
• https://pg.asrock.com/general/news.us.asp?id=5643
• https://www.reddit.com/r/LocalLLaMA/comments/1nqi0di/are_there_any_consumer_mobos_that_can_fit_2x/
• https://nicholasreese.com/guides/quad-r9700/quad-r9700-build-guide.md
• https://www.reddit.com/r/LocalLLM/comments/1vyaesz/blower_r9700_cards_stacked_with_very_little_gap/
• https://forum.level1techs.com/t/radeon-pro-ai-r9700-fan-noise-which-model-to-buy/247552?page=2
• https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/
• https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/
• https://phanteks.com/product/enthoo-pro-2-server-edition-tg/

Next Add a case step to the amdai.net builder parts list, and verify slot spacing for the ASUS ProArt boards from their manuals.

A llama.cpp user saw about 2.4 GB/s buffered loads from NVMe rated over 7 GB/s, with one core at 100%. llama.cpp's direct I/O loader reads at disk speed: its PR reports GPT-OSS-120B loading in about 10.5 s versus about 110 s on the first mmap load. One third-party test found a 40.5 GB Q4_K_M file loading in 15.2 s on a Gen4 990 PRO versus 13.8 s on a Gen5 T705 (about 9%). So capacity and a Gen4 TLC drive with DRAM is the better buy for most builds. Sizes are summed GGUF shard bytes from Hugging Face; load figures are published, not measured on our cards.

Stack

Spec and file-size review: Samsung 990 PRO 2/4 TB, WD_BLACK SN850X 4/8 TB, Samsung 9100 PRO 4/8 TB; unsloth GGUF repos

• https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
• https://github.com/ggml-org/llama.cpp/pull/18012
• https://github.com/ggml-org/llama.cpp/discussions/12223
• https://craftrigs.com/guides/nvme-speed-llm-load-times-benchmarks/
• https://download.semiconductor.samsung.com/resources/data-sheet/samsung_nvme_ssd_990_pro_datasheet_rev.2.0.pdf
• https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_9100_PRO_Datasheet_Rev.2.0.pdf
• https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/product/internal-drives/wd-black-ssd/data-sheet-wd-black-sn850x-nvme-ssd.pdf

Next Time a cold Flash-Next load on R9700s with the mmap loader versus direct I/O to get our own number.

Numbers come from different hosts and harnesses, so treat ratios as rough: 1.5x to 2.5x in llama.cpp's favour short, about 2.3x at depth, with SGLang using both cards. Where SGLang does stand out is a dense 27B FP8 model with a speculator: 34.3 / 44.5 / 41.6 / 30.1 tok/s at 24 / 7.3K / 58K / 197K input, the only speculative lane shown holding near 200K on R9700 (llama.cpp Vulkan tensor split does 45.0 tg128 on the same model at Q4_K_M). Lessons that carry over to any TP2 engine on R9700: the kernel needs CONFIG_HSA_AMD_P2P and CONFIG_PCI_P2PDMA plus iommu=pt (HSA_FORCE_FINE_GRAIN_PCIE=1 changed nothing); a hard-killed run can leave RCCL shared-memory segments that crash the next launch; and aborting a deep TP2 prefill mid-collective can hang a GPU until reset. AMD's ROCm docs say to disable AITER and use the Triton attention backend on Radeon.

Model

Qwen3.6-35B-A3B; Qwen3.8-27B (dense)

Quant

AWQ int4 / FP8 (SGLang); Q4_K_M and UD-Q5_K_XL (llama.cpp)

Engine

SGLang (patched RDNA4 fork) vs llama.cpp

Engine version

SGLang v0.5.15-v0.5.22; llama.cpp PR #25051 build

GPUs

1-2x Radeon AI PRO R9700

Context

short to 197K input

Flags

SGLang TP=2, Triton attention, HIP graphs; llama.cpp Vulkan 1 GPU / layer / tensor

• https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference
• https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference/blob/main/benchmarks/qwen3.6-35b-moe-awq/results.json
• https://github.com/ggml-org/llama.cpp/pull/25051#issuecomment-6382846940
• https://github.com/sgl-project/sglang/issues/30599
• https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/inference/sglang.html
• https://www.lmsys.org/blog/2025-10-22-KTransformers/

Next Keep llama.cpp and Strata for the Flash lanes. Check the TP2 kernel and IOMMU prerequisites before the next vLLM tensor-parallel test.

Sourced, not a new measurement. Mapping from the amd-smi 10.1 CLI docs: rocm-smi --showproductname --showmeminfo vram becomes amd-smi static --asic --vram; --showuse / --showmemuse become amd-smi metric --usage --mem-usage. --rocm-smi is a top-level flag and cannot be combined with subcommands. Readers on llama.cpp Vulkan have neither tool installed; the amdgpu kernel driver exposes mem_info_vram_total, mem_info_vram_used and gpu_busy_percent under /sys/class/drm/cardN/device/, which is enough to prove a model is on the card. ROCm SMI can still be built from source, and older legacy installs (7.2 and earlier) may still have it.

Stack

ROCm Core SDK 10.1.0 (released 2026-10-05) documentation

Site Scan

706 sitemap pages, 2026-10-27

• https://rocmdocs.amd.com/en/latest/about/transition-guide-TheRock.html
• https://rocm.docs.amd.com/projects/rocm_smi_lib/en/docs-10.1.0/index.html
• https://rocm.docs.amd.com/projects/amdsmi/en/docs-10.1.0/how-to/amdsmi-cli-tool.html
• https://docs.kernel.org/gpu/amdgpu/driver-misc.html
• https://docs.kernel.org/gpu/amdgpu/thermal.html

Next Confirm the replacement commands on our dual-R9700 box, then update the verify, troubleshoot, serve and run tutorials to amd-smi with a sysfs fallback for Vulkan readers.

Derived from source, not yet a measured allocation. GLM-5.3-Flash: config.json layer_types lists 34 linear_attention + 11 deepseek_sparse_attention layers, and the GGUF head_count_kv mask agrees. llama.cpp keeps a K-only MLA cache (kv_lora_rank 512, no RoPE part) plus an indexer cache of 3 x 128 elements per token on those 11 layers, and a fixed f32 recurrent state (~146 MiB per sequence) on the 34 linear layers. Qwen3.8-Flash-Next: 12 full-attention layers at 2 KV heads x 256 plus a 2 x 128-element indexer row per token (raw + pooled key). Gemma 4 31B/12B: global layers use 4x512 / 1x512 heads but sliding layers use 16x256 / 8x256, and llama.cpp sizes the sliding cache at window + micro-batch (1,536 cells at the default 512). Dense models (Llama 3.x, Qwen3) and Qwen3.5/3.6 hybrids matched; hybrids also hold a 50-150 MiB fixed state per slot. All figures: f16 cache, one sequence, GiB.

Engine

llama.cpp master 19ff817f (2026-10-27), source review

Cache

f16 K/V, 1 sequence, -ub 512

• https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json
• https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json
• https://huggingface.co/google/gemma-4-31b/blob/main/config.json
• https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-model.cpp
• https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/models/glm5-next.cpp
• https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-kv-cache-iswa.cpp
• https://amdai.net/calculator/

Next Confirm against llama-server startup logs on the 2x R9700 lanes, then fix the calculator presets (sparse-layer count, indexer cache, Gemma 4 window heads, fixed state per slot).

Checked on the retailer pages and on vga.live's card pages around 12:00 CT, before tax. Micro Center: Gigabyte AI TOP $1,949.99 (was $1,999.99, in-store pickup only); ASRock Creator, ASUS Turbo, Sapphire and XFX $1,999.99. Newegg: ASUS Turbo $1,999.99 sold by Newegg, but Newegg's own page showed it out of stock; Newegg marketplace sellers list the Gigabyte AI TOP from $2,079.97. B&H lists the ASUS Turbo at $1,999.99 and the Gigabyte AI TOP at $1,899.95, but vga.live shows the B&H Gigabyte listing as unavailable. Amazon: third-party sellers from $1,943.99 (Sapphire), Gigabyte $2,199.99, ASUS $2,495.00, PowerColor $2,664.51. The lowest price seen since Sep 29 is $1,599.99, so today's cheapest card is $350 above that floor. At $1,949.99 the R9700 costs $60.9 per GB of VRAM. Prices move daily; check before buying.

GPUs

Radeon AI PRO R9700 32 GB (6 board-partner models)

Region

US, new condition

• https://www.microcenter.com/product/699521/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation-graphics-card
• https://www.newegg.com/asus-turbo-ai-pro-r9700-32g-32gb-graphics-card/p/N82E16814126826
• https://www.newegg.com/p/pl?d=radeon+ai+pro+r9700
• https://www.bhphotovideo.com/c/search?q=radeon%20ai%20pro%20r9700
• https://www.amazon.com/dp/B0G75W64R3
• https://vga.live/gpus/radeon-ai-pro-r9700
• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html

Next Keep the daily price snapshot; flag when the cheapest new card drops back to $1,599.99.

Checked against llama.cpp master 27b7a8a009de (2026-10-27) and six dated snapshots of common/arg.cpp back to 2025-04-27. The load flags were folded into --load-mode on 2026-07-23 (#20834) and warned as deprecated until the 2026-09-09 removal. llama.cpp reads environment variables only for options that still exist, so a removed option's env var produces no warning. Other flags that now fail: --draft / --draft-max / --draft-min (removed in the speculative-decoding argument refactor, #22397 and #22964, April-May 2026; use --spec-draft-n-max / --spec-draft-n-min or --spec-ngram-mod-n-max), -cd / --ctx-size-draft, -cpent / --checkpoint-every-n-tokens and --spec-replace (gone between the 2026-04-27 and 2026-07-27 snapshots). -dt / --defrag-thold is still accepted but does nothing. -fa needs a value (on, off or auto); a bare -fa followed by another flag fails. We wrote a small linter for commands, scripts, systemd units and env files and ran it on every llama.cpp command shown on amdai.net (130 distinct command lines across 704 pages): none use a removed or unknown flag.

Engine

llama.cpp

Engine version

master 27b7a8a009de (2026-10-27); removal in b10875

Method

source audit of common/arg.cpp and tools/llama-bench/llama-bench.cpp

• https://github.com/ggml-org/llama.cpp/pull/28334
• https://github.com/ggml-org/llama.cpp/commit/14a9d09f75
• https://github.com/ggml-org/llama.cpp/pull/20834
• https://github.com/ggml-org/llama.cpp/pull/26135
• https://github.com/ggml-org/llama.cpp/pull/22397
• https://github.com/ggml-org/llama.cpp/pull/22964
• https://github.com/ggml-org/llama.cpp/releases/tag/b10875
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Next Before rebuilding llama.cpp past b10875, check every launcher, systemd unit and env file for the removed flags and env vars; switch to --load-mode / LLAMA_ARG_LOAD_MODE. The site's mmap page and glossary get updated wording (removed, not deprecated).

Int4 MoE on gfx12 runs vLLM's Triton WNA16 path, which reached about 18% of memory bandwidth at batch 1 on Strix Halo. The faster W4A8 MoE kernel (open PR #62480) hard-checks for gfx11, but its core int8 dot instruction is the one llama.cpp already uses on RDNA4, so a gfx12 port looks plausible (untested). An open tuned-config PR for E=256,N=512 on the R9700 cut fused-MoE kernel time 19% at batch 1 and raised end-to-end output 3.9% at 1 request and 17.7% at 4. Int4 checkpoints usually keep lm_head in bf16; open PR #62708 quantizes it at load on gfx11 and gfx12 (1.12-1.18x on Strix Halo). Our estimate (640 GB/s spec bandwidth at 75% efficiency, not measured): about 1.5 ms per token saved on a 35B-A3B on one R9700, about 0.9 ms per card on a Flash-class model at TP2. GGUF support moved out of vLLM core in June into vllm-gguf-plugin; vLLM's docs table still lists it as AMD-supported, the plugin's PyPI wheel links the CUDA runtime (libcudart.so.13), so ROCm needs a source build, and its kernels were copied from llama.cpp b2899. For GGUF on Radeon, llama.cpp remains the path. Also open: on R9700, vLLM main runs 13.5% below v0.29 on gpt-oss-20b because of the MoE combine step (fix PR #64055 restores +17.6%).

Quant

W4A16 / GPTQ / AWQ / MXFP4 / GGUF (4-bit class)

Engine

vLLM (source reading)

Engine version

main e7fa26d5c8 (2026-10-27)

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://github.com/vllm-project/vllm/pull/57199
• https://github.com/vllm-project/vllm/pull/62480
• https://github.com/vllm-project/vllm/pull/60413
• https://github.com/vllm-project/vllm/pull/62708
• https://github.com/vllm-project/vllm/pull/64045
• https://github.com/vllm-project/vllm/issues/63597
• https://github.com/vllm-project/vllm/pull/52637
• https://github.com/vllm-project/vllm/issues/62697
• https://github.com/vllm-project/vllm/pull/64055
• https://github.com/vllm-project/vllm/pull/39612
• https://github.com/vllm-project/vllm-gguf-plugin/issues/142
• https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/common.cuh

Next Measure on two R9700s: attention backend correctness, the gfx115x tuning enabled on gfx12, a tuned int4 MoE file, int4 lm_head, then try the W4A8 MoE kernel on gfx12.

Read at llama.cpp master 0c8c59ef3d (2026-10-27): each /metrics request sets metrics_reset_bucket, and generation stats reach the bucket only in the slot reset callback at request end. The 0-reading bug was filed as #27436 and auto-closed as stale; fix PRs #27760 and #30543 are still open. Use tokens_predicted_total / tokens_predicted_seconds_total and prompt_tokens_total / prompt_seconds_total deltas instead; prompt_tokens_cached_total (cache reuse) exists but is missing from the README table. The gauge and per-request timings count n_gen-1 decode steps while the counter counts n_gen, so counter-derived speed reads about 5% high on 20-token tool-call replies; speculative/MTP tokens count in both. Separately, average tok/s falls as conversations get longer, so a speed alert should compare like depths: we bin requests by context depth (0-8K, 8-32K, 32-64K, 64-128K, 128K+) using the 'stop processing: n_tokens' and 'eval time' log lines and flag a bin only when its median drops more than 5% and more than 3 robust standard errors. Strata's JSON /metrics keeps the last 500 requests with prompt size, decode tok/s and the share of experts read over PCIe, which feeds the same check.

Engine

llama.cpp llama-server / Strata (source read)

Engine version

llama.cpp 0c8c59ef3d; Strata main 2026-10-25

GPUs

2x Radeon AI PRO R9700 (target)

• https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-common.h
• https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-context.cpp
• https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-task.cpp
• https://github.com/ggml-org/llama.cpp/issues/27436
• https://github.com/ggml-org/llama.cpp/pull/27760
• https://github.com/ggml-org/llama.cpp/pull/30543
• https://github.com/Niko1221/Strata/blob/main/serve/server.py
• https://docs.kernel.org/gpu/amdgpu/thermal.html

Next Build 7-day per-depth speed baselines for the always-on dual-R9700 lanes and check them after every driver or engine update.

For Flash-class MoE models that don't fit in VRAM, gigabytes per card decide how many experts stay on the GPU, so $/GB, W/GB and card count matter more than per-card bandwidth. At yesterday's $1,699.99 the R9700 was 10-17% cheaper per GB than a used 3090; at its $1,599.99 low since Sep 29, 15-21% cheaper. Break-even against a $1,528 used 3090 is about $2,037. The 3090 still has about 1.5x the bandwidth per card (936 vs 640 GB/s, vendor spec), which matters when the whole model fits in VRAM. Builds (GPUs only): 64 GB = 2x R9700 $3,900, 600 W vs 3x 3090 (72 GB) $4,584, 1,050 W; 128 GB = 4x R9700 $7,800, 1,200 W vs 6x 3090 (144 GB) $9,168, 2,100 W. Cheapest per GB among used cards: RX 7900 XTX ($1,004, $41.8/GB) and Quadro RTX 8000 48 GB ($2,080, $43.3/GB, 5.4 W/GB). Used RTX 4090 ($3,037) and new RTX 5090 ($5,999.99) cost $126-188/GB. Used prices carry no warranty; one tracker (getpcparts) shows a much lower $1,093 3090 average and was treated as an outlier. Prices are before tax, US.

• https://gpusold.com/
• https://compsniper.com/worth/rtx-3090
• https://www.pcprice.watch/search/RTX3090/US/GPU
• https://vga.live/gpus/radeon-ai-pro-r9700
• https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation
• https://rtx.live/gpus/geforce-rtx-5090
• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html

Next Add a VRAM-per-dollar table to the buying pages, refreshed with the price data; for a two-card purchase, wait for the R9700 to return to about $1,600.

TheRock #9279: ROCm llama.cpp hang leading to MES REMOVE_QUEUE failure, GPU reset and PSP error; on 2026-10-22 a reporter showed it still fails on a TheRock 10.2 nightly with amdgpu-dkms and firmware 31.60, including a pure dual-R9700 layer-split run, while Vulkan passes. TheRock #9297: tiny kernels slow down when two or more other streams hold pending waits; reported penalty by MES firmware is +7 to +39 us at 0x89, +5 to +17 us at 0x91 and about +9 us at 0x93, and GPU_MAX_HW_QUEUES=2 or 1 removes it. TheRock #9245: an AMD contributor traced the extra idle draw to the kernel disabling GFXOFF on GC 12 once a compute queue exists (Linux commit 90505894c4ed); the reporter measured about 5-6 W more per card than Vulkan. vLLM #57838: from v0.28.0 a RowWise FP8 kernel is selected on RDNA4 and the report measures 5-24% slower decode than v0.27.1's choice. llama.cpp #28093: with an AMD iGPU enabled next to an R9700, Vulkan llama-server reserved most host RAM at idle; it was closed as stale, not fixed. Housekeeping: the ROCm/ROCm repo now redirects to ROCm/legacy-rocm-build and GitHub issue search on the old name returns an error, so new ROCm reports go to ROCm/TheRock.

• https://github.com/ROCm/TheRock/issues/9279
• https://github.com/ROCm/TheRock/issues/9297
• https://github.com/ROCm/TheRock/issues/9245
• https://github.com/ROCm/TheRock/issues/9228
• https://github.com/vllm-project/vllm/issues/40980
• https://github.com/vllm-project/vllm/issues/57838
• https://github.com/ggml-org/llama.cpp/issues/28093
• https://github.com/ggml-org/llama.cpp/issues/29892
• https://github.com/ROCm/rocm-libraries/issues/12346
• https://github.com/ROCm/rocm-libraries/issues/7992

Next Check our own cards' MES firmware, launcher environment and iGPU state against this list, rerun the search weekly, and add an open-upstream-bugs panel to /rocm-problems.

Qwen Community License 1.0 (Qwen3.8-Flash-Next): free for personal and internal use, but any company running a Model-as-a-Service or coding/office AI-assistant business needs a separate Qwen license for commercial use, with no revenue floor; the Qwen3.8-Max license (2.4T-A95B) only requires that above US$50M revenue in 12 months. GLM-5.3: MIT-style, only Model-as-a-Service firms above US$10B revenue need a Z.AI security review. MiniMax-M3.1-Flash: the base grant is non-commercial; commercial use needs a 'Built with MiniMax M3.1-Flash' notice and a one-time email, and written approval above US$20M/year. LiquidAI's LFM Open License 1.0 (5 models): commercial use only for entities under US$10M annual revenue. Tencent ContextPilot 8B/14B/E4B ship Apache 2.0 text with an added clause limiting use to scientific research. Youtu-Parsing-Omni is 'not intended for use within the European Union'. Also, 36 of 61 repos tagged apache-2.0, mit or llama3 had no LICENSE file at the repo root; none of the 25 that did added extra terms. Plain-language reading, not legal advice.

• https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
• https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE
• https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE
• https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/LICENSE
• https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/main/LICENSE
• https://huggingface.co/tencent/ContextPilot-8B/blob/main/LICENSE
• https://huggingface.co/tencent/Youtu-Parsing-Omni/blob/main/LICENSE

Next Add a filterable 'Use' column (open, conditions, revenue cap, non-commercial) with a LICENSE link to /new-models.

Method: each CLI pointed at a recording mock of /v1/chat/completions, /v1/messages and /v1/responses in a 3-file git repo, prompt 'Say done.'; tokens counted with the Qwen3 tokenizer over system text, tool schemas and messages (chat-template tokens excluded). Claude Code's 19,384 tokens are mostly tool schemas (16,465 for 24 tools). Two fresh runs of every CLI produced identical prompts; Claude Code 2.1.197's attribution block was stable, unlike older builds that changed it per request. The captured request bodies were then replayed in order on llama-server b11912 (CPU, Qwen3.5-0.8B Q8_0, a hybrid model, -np 1). Qwen Code turn 2: 21 tokens re-read (0.29 s) with the default --cache-ram, 15,602 tokens (31.0 s on this CPU) with --cache-ram 0, and 21 tokens with --cache-ram 0 when its setting memory.enableManagedAutoMemory is false. OpenCode's title call runs only on the first turn, before the main request, so turn 2 re-read 21 tokens either way. Prefill seconds on Radeon will differ; the token counts are what carry over. Also note llama-server's default port changed from 8080 to 9931 (merged 2026-10-09), so guides that assume 8080 need --port.

Model

Qwen3.5-0.8B

Quant

Q8_0

Engine

llama-server

Engine version

b11912 (CPU build)

Context

65536

Flags

-np 1, --reasoning off; arms: default --cache-ram, --cache-ram 0

Clients

Claude Code 2.1.197, Qwen Code 0.15.10, Codex CLI 0.164.0, OpenCode 1.19.4, Aider 0.86.2

• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• https://github.com/ggml-org/llama.cpp/pull/30159
• https://github.com/ggml-org/llama.cpp/issues/20003
• https://code.claude.com/docs/en/llm-gateway-protocol
• https://opencode.ai/docs/providers/

Next Repeat the turn-by-turn count on the dual-R9700 Flash and single-R9700 35B lanes, and publish ready-to-use configs for each CLI.

Cited results: 2x R9700 (PCIe 4.0 x8/x8, ROCm 7.14), gemma4 31B Q6_K, -sm tensor, pp2048/tg512 at depth 4096: 959.9/30.0 t/s before vs 1127.5/32.5 t/s after #27825; that tester also reported frequent GPU hangs with RCCL enabled. 2x R9700, Qwen3.8-27B UD-Q4_K_XL server: RCCL on vs off gave +3.6% to +10.1% prefill at 60K-120K and identical decode at 1-3 parallel slots. 2x 7900 XTX, Qwen3.8-27B Q8_0: RCCL ~9% faster prefill, built-in AllReduce ~2.5-2.8% faster decode, also at 100K depth. Layer split (-sm layer) has no AllReduce and is unaffected. Dead or changed flags on master (2026-10-26): GGML_HIP_ROCWMMA_FATTN was removed 2026-07-24 and is silently ignored; -ffast-math and -funsafe-math were dropped from HIP builds by mid-August 2026 (unsafe math broke MTP output identity), so builds from July differ numerically; GGML_CUDA_FA_ALL_QUANTS became GGML_CUDA_FA_QUANTS, whose default covers only f16, bf16, q8_0 and q4_0 matched pairs. A mixed KV cache such as -ctk q8_0 -ctv f16 then converts K and V to f16 on every FlashAttention call, with a single log warning. Head-dim-256 models still lack the MMA FlashAttention path on RDNA4 (PR #26419 open; +21.8% prefill at 127K depth on one R9700 in its tests).

Engine

llama.cpp (HIP/ROCm)

Engine version

master 7aca0d2 (2026-10-26) source read; results from cited testers

GPUs

2x Radeon AI PRO R9700 (and 2x RX 7900 XTX for the RCCL decode comparison)

Flags

-sm tensor, -fa on, GGML_HIP_RCCL OFF vs ON

• https://github.com/ggml-org/llama.cpp/pull/27825
• https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5535620413
• https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5699240655
• https://github.com/ggml-org/llama.cpp/pull/26046
• https://github.com/ggml-org/llama.cpp/pull/26696
• https://github.com/ggml-org/llama.cpp/pull/28079
• https://github.com/ggml-org/llama.cpp/pull/26419

Next Audit the build cache of each local lane, and A/B RCCL off vs on only if a tensor-split lane is used.

Reading temps: the spd5118 hwmon driver (Linux 6.11 and later) exposes temp1_input, temp1_crit and alarm files per DIMM; on AMD boards the SMBus driver registers the SPD hubs automatically, and an AM5 example shows sensors named spd5118-i2c-1-51 and -53 with a default 'high' of 55 C (not a problem by itself) and 'crit' of 85 C. Field reports: one owner measured DDR5-6400 RDIMMs at 95 C and above, cooled them under 60 C with shrouds and fans, and saw about 10% faster inference; a dual-3090 hybrid build reported decode falling from 17 to 8-12 tok/s after minutes from DIMM heat. Stability: consumer AM5 sticks are non-ECC, and DDR5's on-die ECC fixes bits inside the chip without telling the OS, so EDAC/rasdaemon show nothing; a marginal four-stick speed appears only as rare crashes, kernel oopses, machine-check events or wrong output. A published 4x64 GB AM5 setup was 12 h y-cruncher-stable at DDR5-4800 (36-40-40-78, about 56 GB/s), and the same settings failed after about 7 h on an older AGESA. On Linux, y-cruncher's VT3, FFTv4 and N63 components plus stressapptest are the practical tests; TestMem5 and Karhu are Windows tools.

• https://docs.kernel.org/hwmon/spd5118.html
• https://cateee.net/lkddb/web-lkddb/SENSORS_SPD5118.html
• https://lkml.indiana.edu/2405.3/07725.html
• https://www.phoronix.com/forums/forum/hardware/motherboards-chipsets/1512681-linux-6-14-to-add-sensor-monitoring-for-a-~-180-asrock-am5-motherboard/page2
• https://hivebook.wiki/wiki/dram-row-buffers-and-bank-conflicts-activate-rate-tfaw-and-refresh
• https://semiengineering.com/dram-thermal-issues-reach-crisis-point/
• https://www.reddit.com/r/LocalLLaMA/comments/1p89j2t/today_i_learned_that_ddr5_can_throttle_itself_at/
• https://forum.level1techs.com/t/ddr5-4-dimms-on-am5-whats-working-whats-not/197153?page=14
• https://pakt.cc/en/ddr5/testing/

Next Log DIMM temperatures next to tok/s during a 30-minute hybrid decode on our dual-R9700 desktop with four 64 GB sticks, and test whether better airflow changes the speed.

Card numbers: Igor's Lab (compute/AI full load ~390-400 W), PCGH (only ~1.5 W from the PCIe slot, so nearly all power rides the single 12V-2x6 connector), Puget Systems (two cards, sysfs power: llama.cpp Q4_K_M layer split 339 W at 1 request, 353 W at 8; vLLM TP2 FP8 310-315 W at 1 request, up to 510 W at 4-8). CPU: Ryzen 9 9950X measured at its 200 W PPT, 210-235 W with PBO. Power cap results from a published llama.cpp sweep on one R9700: 260 W cost 0.0% decode and 4.0% prefill; 210 W cost 1.9% decode and 11.2% prefill. Rest of each system (board, four DDR5 sticks, NVMe, fans, NIC) is our estimate of 60-100 W, and wall power assumes an 80 PLUS Platinum supply at 92%. Per box: two cards at 400 W plus a PBO CPU is ~1,135 W DC, which is 76% of a 1,500 W supply, 95% of 1,200 W and over 100% of 1,000 W. A UPS should be sized on input watts times 1.25 and be pure sine wave for active-PFC supplies; one desktop in hybrid Flash decode needs at least ~925 W / ~1,030 VA by that rule.

• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html
• https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/
• https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/5/
• https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/
• https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/
• https://www.tomshardware.com/pc-components/cpus/amd-ryzen-9-9950x-cpu-review/4
• https://www.infobytes.de/pages/atx3.1powerrev2.1.pdf
• https://electricaltoolbox.com/continuous-load-calculator/
• https://www.se.com/us/en/faqs/FA158939/

Next Measure real CPU, GPU and rest-of-system draw per workload on our dual-R9700 desktop to replace the estimates, and pick the second desktop's supply (1,200-1,300 W with two native 12V-2x6 cables, or 1,000 W only with cards capped at 260 W).

Measured on a shared 8-vCPU cloud VM with a CPU build, not a Radeon card, so the size of the noise here is a worst case; the pattern (repetitions within one invocation share warm caches and clock state, so their spread understates run-to-run noise) is general. Source facts are from llama.cpp master 7aca0d2: llama-bench defaults to -r 5 and --delay 0; its warmup runs one full prompt for pp tests but test_gen with 1 token for tg tests; compare-llama-bench.py compares AVG(avg_ts) only. What we do now: an A/A run of every config before an A/B sweep to measure its noise floor, at least 3 invocations per arm interleaved A B A B, one throwaway invocation after each model load, and a hierarchical bootstrap of the median ratio (resample invocations, then repetitions) with a minimum effect size, instead of mean +- stddev from one run. With only 3 vs 3 identical runs, one split still showed +9.9% (interval 1.003-1.187) because the slow first run landed on one side, which is why the throwaway run matters.

• https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md
• https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp
• https://github.com/ggml-org/llama.cpp/blob/master/scripts/compare-llama-bench.py

Next Run the same A/A test on dual R9700s for the Flash lane and on one R9700 for the 35B lane to get the real Radeon noise floor, then use it as the minimum effect size for the Q4/Q5/Q6 dual-GPU sweep.

Method: Open WebUI driven through its API against a logging mock backend to capture exactly what it sends, then those task prompts replayed against a real llama-server (llama.cpp b11912 CPU build, Qwen3.5-0.8B Q8_0, a hybrid Gated DeltaNet model like the Flash-class models) for a 5-turn, ~11.1K-token chat. Measured on an 8-vCPU cloud Xeon, not a Radeon card; the re-processed token counts do not depend on hardware, the seconds do. Captured calls: task prompts start with '### Task:' and carry a truncated history (follow-ups: first user message plus last 4, about 1,000 characters each); title max_tokens is 1000, follow-up and tags have no limit, and by default all run on the chat's own model. llama.cpp's cache entry for the conversation was 150.6 MiB of state, 208.4 MiB with its 3 context checkpoints, and saving plus restoring took 209 ms; each task prompt adds its own 27-30 MiB entry to the same cache. Auto slots (4 slots, unified KV, the llama-server default when -np is not set) and -np 2 also kept reuse at about 70 tokens; -np 2 without unified KV halves each slot's context.

• https://github.com/open-webui/docs/blob/main/docs/features/administration/task-models.mdx
• https://github.com/open-webui/open-webui/issues/638
• https://github.com/ggml-org/llama.cpp/pull/16391
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp
• https://github.com/ggml-org/llama.cpp/releases/tag/b11912

Next Keep --cache-ram on and larger than one long conversation plus its checkpoints (check the server log's 'total state size' line), point Open WebUI's task model at a small non-reasoning model or turn follow-ups and tags off, and cap task max_tokens. Next: measure the cache entry size and restore time on a Radeon R9700 lane at 60K and 200K tokens.

Method: templates read from the GGUF headers and from upstream at each commit, rendered with llama.cpp's Jinja engine (common/jinja at master 4244ad7) and with Python Jinja2 3.1.6. Single-threaded, measured on an 8-vCPU cloud Xeon, not the R9700 desktop, so absolute times will differ; the ratio is the point. Long agent sessions: 60 turns x 4 calls 176 vs 81 ms; 30 x 16 910 vs 127 ms; 10 x 64 4,356 vs 175 ms. llama.cpp's engine was about 5x slower than Jinja2 on the same template. The 2026-09-04 upstream 'early break' change alone gave no speedup. 200 randomized conversations (shuffled, missing and duplicate tool ids, multi-turn) rendered byte-identical. Side finding on Qwen templates: the Unsloth GGUF templates for Qwen3.6-35B-A3B and Qwen3.8-Flash-Next accept a developer role, two leading system messages and a conversation with no user message, where upstream raises an error; but a system message after the first turn makes the Flash-Next GGUF template raise and the 35B GGUF template silently drop it. Keep the GGUF Qwen templates and avoid late system messages.

Model

GLM-5.3-Flash (also Qwen3.6-35B-A3B, Qwen3.8-Flash-Next templates)

Quant

UD-Q6_K_XL header (template is quant-independent)

Engine

llama.cpp Jinja engine + Python Jinja2

Engine version

llama.cpp master 4244ad7; Jinja2 3.1.6

GPUs

none (CPU-only template rendering)

Context

n/a

Flags

--chat-template-file

• https://huggingface.co/zai-org/GLM-5.3-Flash/discussions/63
• https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF
• https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/chat_template.jinja
• https://github.com/ggml-org/llama.cpp/tree/4244ad7acd98328c1f04f1c2201d571f643370bb/common/jinja

Next Time /apply-template on the R9700 desktop with both templates and fold --chat-template-file into the GLM preset if it saves time at a normal tool fan-out.

Snapshot taken Oct 26, 2026 around 4 PM CT from vga.live's public card pages, before tax. ASRock Creator, ASUS Turbo and Sapphire are $1,799.99 at Micro Center; PowerColor is only in stock on Amazon at $2,680.81. Newegg (ASRock, $1,699.99) and B&H (Gigabyte, $1,899.95) showed out of stock. A two-card setup (64 GB VRAM) costs $3,399.98 before tax today, $200 above the recent low. Prices move daily; check before buying.

GPUs

Radeon AI PRO R9700 32 GB (6 board-partner models)

Region

US, new condition, in stock

• https://vga.live/gpus/radeon-ai-pro-r9700
• https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation
• https://vga.live/cards/xfx-amd-radeon-ai-pro-r9700-single-fan-32gb-gddr6
• https://vga.live/cards/asrock-amd-radeon-ai-pro-r9700-creator-single-fan-ai-workstation
• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html

Next Track the cheapest new price daily and flag when it returns to the $1,599.99 floor.

Estimate, not a measurement: if vLLM matched llama.cpp's ~100.8 tok/s single-card decode for the 35B, an 80-100% efficient TP2 split would land near 110-128 tok/s, so TP2 is mainly a capacity tool, not a big speed win, for models that already fit on one card. RCCL's low-latency LL protocol (auto-picked for AllReduce up to 16 KiB per rank on gfx120x) is what deadlocks on RDNA4 (ROCm rocm-systems #5480, open); the NCCL_PROTO=Simple workaround avoids it but likely costs small-message latency. AMD's patch to force Simple by default was closed unmerged on 2026-10-20 after its 4-GPU hang turned out to be a hipBLASLt issue. Published R9700 TP numbers (e.g. Qwen3.6-35B-A3B-FP8 TP2 17.4 tok/s) were run with --enforce-eager. Two platform traps: with iommu=pt, cross-process peer IPC on PCIe RDNA4 returns success but reads 0xFF and drops writes; and RCCL collectives currently fail on an R9700 attached through the chipset ('AQL dispatch failed!', rocm-systems #13616, open).

Model

Qwen3.6-35B-A3B / Qwen3.8-Flash-Next

Quant

n/a (communication analysis)

Engine

vLLM (main, 2026-10-26 source) + RCCL

Engine version

vLLM main 7f56f45

GPUs

2x Radeon AI PRO R9700

Context

batch 1 decode

Flags

tensor-parallel-size 2, NCCL_PROTO=Simple

• https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/platforms/rocm.py
• https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/config/parallel.py
• https://github.com/vllm-project/vllm/pull/19744
• https://github.com/ROCm/rccl-tests/issues/162
• https://github.com/ROCm/rocm-systems/issues/5480
• https://github.com/ROCm/rocm-systems/pull/13151
• https://github.com/ROCm/ROCR-Runtime/issues/355
• https://github.com/ROCm/rocm-systems/issues/13616
• https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504

Next Measure 2-card RCCL all-reduce latency (Simple vs shared-memory path vs a time-limited LL run) and vLLM TP2 decode with HIP graphs vs eager on the local desktop.

Method: listed every GGUF with its LFS sha256 at the commit and at its parent via the Hugging Face API, then parsed the GGUF headers of both versions. GLM-5.3-Flash UD-Q6_K_XL shard 1: 40 architecture keys renamed with the architecture, 1 key added, tensor data shards unchanged; the pre-change file's sha256 matches revisions from 2026-08-29 to 2026-09-06. The old glm5next name came from llama.cpp PR #27752, which was closed unmerged; PR #27773 merged support as glm5-next. Flash-Next MTP heads (BF16, Q4_K_M, Q8_0; same size and same 34 tensors): qwen4exp.attention.compress_ratios entry 48 changed 0 -> 4. We have not measured what the old value does to MTP acceptance. Qwen3.6-35B-A3B GGUFs were last re-uploaded 2026-04-20. We wrote a read-only checker that compares a local GGUF's size and header with the current Hugging Face file (or its full sha256 with --full) and labels it current, stale or corrupt; it correctly flagged an old GLM shard, a current shard and a shard with one flipped byte.

Model

GLM-5.3-Flash, Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (Unsloth GGUF)

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header diff on UD-Q6_K_XL shard 1 and MTP Q8_0)

Engine

llama.cpp (arch names from master)

Engine version

master, 2026-10-26

GPUs

n/a (file analysis)

• https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/commit/a38483c8cd5df544f53d70fb281afe97369d5ab6
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/commit/efc50b7c3195a10eebd8d3e32f83b27b2aeb239d
• https://github.com/ggml-org/llama.cpp/pull/27773
• https://github.com/ggml-org/llama.cpp/pull/27752
• https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/commits/main

Next Check our own GLM-5.3-Flash and Flash-Next MTP copies with the checker; if stale, swap the 9 MiB shard and A/B MTP acceptance with the old vs new head.

Size cost from the Hugging Face file listings: the MTP files are 470 MiB (UD-Q4_K_XL), 540 MiB (UD-Q5_K_XL) and 732 MiB (UD-Q6_K_XL) larger than the plain files. The MTP block has its own attention layer, so its KV cache adds about 512 MiB at f16 or 272 MiB at q8_0 for 262K context (K/V width 512 from the header). Our estimate for UD-Q5_K_XL with MTP at 262K and f16 KV is about 30.9 GiB before compute buffers, which is tight or over on a 32 GiB card with ECC on (about 29.9 GiB visible); quantizing only the draft cache or capping context near 128K should fit (estimate, not measured). Other data points: a 7900 XTX on ROCm gained 1.70x at 16K falling to 1.35x at 196K and ran out of memory at 245K; Unsloth reports 1.15-1.2x on average for this MoE at 2 draft tokens, with acceptance falling from 83% to 50% at 4; the community DSpark drafter beat MTP on code (+12.7%) but lost on prose (-4.8%) on non-Radeon hardware. A later re-run of the 3090 study found the slowdown tracks low real acceptance plus draft overhead.

Model

Qwen3.6-35B-A3B (MTP GGUF)

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header check); Q4_K_M and UD-Q4_K_XL in cited speed tests

Engine

llama.cpp

Engine version

per source

GPUs

1x R9700 (cited); RTX 3090 and 7900 XTX for comparison

Context

short to 245K

Flags

--spec-type draft-mtp --spec-draft-n-max 2, -np 1

• https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/
• https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/discussions/14
• https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090
• https://gist.github.com/AshyIsMe/ef9dc0e57d7af5626f3935f55c38543a
• https://unsloth.ai/docs/models/tutorials/qwen3.6
• https://github.com/ggml-org/llama.cpp/discussions/26230
• https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF

Next Measure MTP at 1, 2 and 3 draft tokens on our single-R9700 UD-Q5_K_XL lane at 128K, then test whether 262K fits with a q8_0 draft cache.

Windows HIP, -sm layer (2x R9700, 7900 XTX + 9070 XT and others): a staged GPU-to-GPU copy raced in-flight compute; fixed in ROCm CLR PR #10553 and confirmed by AMD on a current Lemonade build, but older Windows ROCm releases (e.g. b10767) still corrupt. Building llama.cpp with -DGGML_CUDA_NO_PEER_COPY=ON (a cmake option; an environment variable does nothing) also fixes it; one mixed-card test went from 0/7 to 7/7 exact answers at about -4.5% prefill and -7.7% decode. Windows HIP -sm tensor: open (Lemonade llamacpp-rocm #148); root cause is hipMemcpy2DAsync from memory-mapped model pages, and --load-mode none avoids it. Even when correct, Windows exposes no GPU peer access, so a second R9700 adds memory, not speed (open driver request ROCm/rocm-systems #13715). Linux, 2x RX 9060 XT on separate root ports (ROCm 7.2.4): HSA_ENABLE_PEER_SDMA=0 did not fix it; the no-peer-copy build did. Quick check: ask a few one-answer questions at temperature 0 on one card and then split across both; answers should match exactly.

Model

Qwen3.8-27B Q4_K_M (Windows speed test); various (corruption reports)

Quant

>=Q4 in all cited tests

Engine

llama.cpp (HIP and Vulkan), Strata

Engine version

per source

GPUs

2x R9700; 7900 XTX + 9070 XT; 2x 9060 XT; others

Context

short

Flags

-sm layer / -sm tensor, -DGGML_CUDA_NO_PEER_COPY=ON, --load-mode none

• https://github.com/ggml-org/llama.cpp/issues/28676
• https://github.com/lemonade-sdk/llamacpp-rocm/issues/75
• https://github.com/ROCm/rocm-systems/pull/10553
• https://github.com/lemonade-sdk/llamacpp-rocm/issues/148
• https://github.com/ROCm/rocm-systems/issues/13715
• https://github.com/ROCm/ROCm/issues/6342

Next Run the same exact-answer check on our dual-R9700 Linux launchers (Vulkan and HIP, layer and tensor) and make it a required step before every benchmark.

Source read at llama.cpp master c6fb3b5: the slot file carries the KV payload, then a checkpoint appendix, then (since PR #30574, b11669) the draft/MTP state, so speculative decoding keeps its acceptance after a restore (37% to 49% in the PR's Qwen3.8-27B test). Older files still load; mismatched draft data is dropped instead of crashing. Testing trap: the RAM prompt cache (--cache-ram) can hide a broken disk restore, so test with --cache-ram 0. Estimate from the Qwen3.6-35B-A3B config (not measured): about 20 KiB per token of f16 KV plus about 63 MiB of recurrent state, so roughly 1.2 GiB at 58K tokens and 5.1 GiB at 262K. A test on dual R9700 is next.

Model

Qwen3.5-35B-A3B (published test); Qwen3.6-35B-A3B (size estimate)

Quant

UD-Q4_K_XL (published test)

Engine

llama.cpp llama-server

Engine version

>= b11507 (checkpoints), >= b11669 (draft state)

GPUs

Radeon 8050S (published test)

Context

58K tokens

Flags

--slot-save-path DIR, --ctx-checkpoints, --cache-ram 0 for testing

• https://github.com/ggml-org/llama.cpp/pull/26004
• https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5055464245
• https://github.com/ggml-org/llama.cpp/pull/30574
• https://github.com/ggml-org/llama.cpp/issues/28194
• https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json

Next Measure save, restart and restore times on dual R9700 for the 35B and the Flash llama.cpp lane, then wire save-on-stop and restore-on-start into the server units.

In llama.cpp master (2026-10-26) the per-request budget overrides the --reasoning-budget server flag unless it is -1, the sampler re-arms on every new think block, and nothing happens without --jinja because the think tags come from the chat template. Qwen's documented stop sentence is: 'Considering the limited time by the user, I have to give the solution based on the thinking directly now.' Arithmetic at measured decode speeds: 2,048 thinking tokens take about 63 s on the Flash model at 32.6 tok/s, about 128 s on a 16 tok/s fallback, and about 20 s on a 35B-A3B at 100.8 tok/s on one R9700. A 20-second deadline allows about 650 thinking tokens on the Flash model vs about 2,000 on the 35B. Whether the Strata engine honours these request fields is not yet checked.

Model

Qwen3.8 Flash (UD-Q6_K_XL) and Qwen3.6-35B-A3B (UD-Q5_K_XL)

Engine

llama.cpp server (source review), Strata (speed only)

Engine version

llama.cpp master 8b28c23

GPUs

2x Radeon AI PRO R9700 (Flash); 1x R9700 (35B)

• https://github.com/ggml-org/llama.cpp/pull/20297
• https://github.com/ggml-org/llama.cpp/pull/23971
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder/commit/59aa5b59f6ecd7a7fa9cc191e1e40d064ad2b2e7
• https://arxiv.org/abs/2505.09388

Next Measure natural thinking length and capped accuracy on real agent turns for both models, then pick a default cap or deadline.

Practical rules for unattended serving: probe with a tiny real completion and a wall-clock timeout instead of trusting /health; scan the kernel log for PSP/IP-block resume failures and stop (don't restart) when they appear; cap restarts (for example 3 per 30 minutes) so a failing load can't loop. amdgpu.cwsr_enable=0 is circulating as a fix but one report saw it turn clean faults into 'GPU Hang' on 3 of 4 R9700s. In one controlled A/B the same llama.cpp workload that wedged on ROCm 7.14 passed on Vulkan RADV, so a Vulkan build is a reasonable fallback service. The hipBLASLt page fault behind many training-time resets is fixed by rocm-libraries PR 8909 (in ROCm 10.2 nightlies as of 2026-09-21). We wrote a stdlib watchdog that encodes these rules and passes 8 of 8 simulated cases; it has not yet been run against real hangs.

Engine

llama.cpp / vLLM (published reports)

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://github.com/ROCm/legacy-rocm-build/issues/6630
• https://github.com/ROCm/ROCm/issues/6413
• https://github.com/ROCm/TheRock/issues/9271
• https://github.com/lemonade-sdk/lemonade/issues/2609
• https://github.com/ROCm/ROCm/issues/6127
• https://docs.kernel.org/gpu/amdgpu/debugfs.html

Next Run the watchdog in log-only mode for three nights on the dual-R9700 desktop, then turn on automatic restart, fallback and stop-and-alert.

Sources: llama.cpp master server source (public endpoints list) and server/RPC READMEs; CVE records for CVE-2026-107183 (fixed b11393, PR #29942; Debian's sid package 0.7.0 still listed vulnerable), CVE-2026-43632 (tokenizer-endpoint use-after-free when --sleep-idle-seconds is set), CVE-2026-21869 (negative n_discard with --context-shift), CVE-2026-34159 (RPC GRAPH_COMPUTE RCE). vLLM's own security docs say --api-key guards only /v1, /v2, /inference and /cohere; /invocations, /pause, /abort_requests and /metrics stay open. A September 2026 internet census found 36,769 exposed self-hosted AI endpoints, of which 2.02% returned an auth challenge (llama.cpp 69 found, 0 authenticated; vLLM 4,880 found, 3 authenticated). Our posture for a dual-R9700 box: keep every server on localhost and reach it through one reverse proxy or VPN tunnel; if a LAN bind is needed use --api-key-file, set --cors-origins to the frontend, and turn off /slots; run rpc-server bound only to the inter-machine link address. We also wrote a read-only probe that checks which routes answer without a key and whether CORS reflects arbitrary origins; a bare TCP check on the RPC port can false-positive, since other services may use 50052.

Engine

llama.cpp llama-server, ggml RPC, vLLM

Engine version

llama.cpp master 9125a2e (release v0.7.0)

GPUs

2x Radeon AI PRO R9700 (applies to any GPU)

• https://www.cve.org/CVERecord?id=CVE-2026-107183
• https://www.cve.org/CVERecord?id=CVE-2026-34159
• https://github.com/ggml-org/llama.cpp/security/advisories?state=published
• https://github.com/advisories/GHSA-3f2h-96vw-c84x
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md
• https://docs.vllm.ai/en/latest/usage/security/
• https://www.mysteriumvpn.com/blog/data-and-research/we-exposed-ai-supply-chain

Next Run the read-only probe against every listener on our own box and from a second LAN machine, then pin all llama.cpp builds at b11393 or newer before the two-box RPC test.

The 1.7B Base GGUFs are Q4_K_M 0.965 GiB, Q8_0 1.721 GiB and bf16 3.234 GiB, plus a 0.416-0.623 GiB mmproj, all at or above 4-bit. A review of the PR found that code2wav pads every decode batch to 72 frames, so shrinking batches for faster first audio detunes the voice (speaker similarity 0.817 vs 0.862); a not-yet-merged patch with a 12/24/48/72 frame ramp cut time to first audio from about 1.8 s to 0.4 s on Vulkan. Fixed seeds are not reproducible on the server yet. Estimate only (bandwidth roofline from the published per-frame weight reads): on an R9700 every quant is far faster than real time (RTF about 0.07-0.23), while on CPU with dual-channel DDR5 at a measured 55 GB/s Q8_0 lands near RTF 1.05 and Q4_K_M near 0.60, and that traffic competes with any MoE experts offloaded to the same DDR5.

Model

Qwen3-TTS-12Hz 1.7B and 0.6B

Engine

llama.cpp llama-server /tts (b11781+); custom HIP port; PyTorch reference (published)

GPUs

1x Radeon AI PRO R9700

• https://github.com/ggml-org/llama.cpp/pull/26603
• https://github.com/ggml-org/llama.cpp/pull/26254
• https://github.com/ggml-org/llama.cpp/tree/master/tools/tts
• https://huggingface.co/ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF
• https://flynnbody.com/blog/running-qwen3-tts-on-amd-r9700/
• https://github.com/piedshag/qwen3-tts-hip

Next Measure llama-server /tts on our own R9700s (Vulkan vs HIP vs CPU), then again while the Flash model decodes, to see what speaking costs the LLM.

Wan 2.2 T2V A14B Q6_K GGUF at 720p (33 frames, 4 steps) peaks at 31.8 of 31.86 GiB on one card, 27.7 GiB at 480p, so video cannot share a card with any resident LLM layers; even SDXL peaks at 12.87 GiB. Video runs drew 500-568 W at 97-104 C junction vs 342-503 W for image runs (same source). Cold starts dominate short jobs: Qwen-Image 4-step 33.4 s cold vs 12.4 s warm. The flag A/B is one warm run per config, so differences under ~2% are noise; the video slowdown is consistent in sign across Wan 2.2 I2V/T2V and HunyuanVideo 1.5 I2V/T2V. Qwen-Image-Edit 20-step was 7.0% faster with the flags. All weights cited are 4-bit or higher (FP8, Q6_K, BF16).

Model

SDXL 1.0, Qwen-Image-2512 FP8, Wan 2.2 A14B (FP8 and Q6_K), HunyuanVideo 1.5

Engine

ComfyUI (published)

Engine version

per source (ROCm 7.1-7.2.4)

GPUs

1x Radeon AI PRO R9700

• https://kyuz0.github.io/amd-r9700-comfy/
• https://kyuz0.github.io/amd-r9700-comfy/benchmark_results.json
• https://brookesia.app/boards/ai-hardware/76
• https://brookesia.app/boards/ai-hardware/78
• https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html

Next Measure an SDXL baseline, an ABBA test of the two flags, and two ComfyUI instances (one per card) on our own pair of R9700s.

llama.cpp serves both: --embedding --pooling last for Qwen3-Embedding and --reranking for /v1/rerank. Bidirectional rerankers such as bge-reranker-v2-m3 must fit the whole prompt in --ubatch-size; causal Qwen3 rerankers do not have that limit. Qwen reports that adding a query-side instruction ('Instruct: ...\nQuery: ...') improves retrieval by about 1-5%. Scores are vendor-reported (Qwen model cards), not measured here; reranker scores are on the top-100 candidates from Qwen3-Embedding-0.6B. Whether the small pair is fast enough on CPU, and how much a GPU embedder slows a co-resident Flash model, is still to be measured.

Model

Qwen3-Embedding 0.6B/4B/8B, Qwen3-Reranker-0.6B, bge-reranker-v2-m3

Quant

Q8_0 / Q5_K_M

Engine

llama.cpp server (docs at master d854d98bae)

GPUs

Radeon AI PRO R9700 (planned placement)

• https://huggingface.co/Qwen/Qwen3-Embedding-8B
• https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF
• https://huggingface.co/Qwen/Qwen3-Reranker-0.6B
• https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md

Next Measure CPU vs one-R9700 placement for the 0.6B pair and the 4B embedder, including the decode drop on a co-resident Flash model.

gpt-oss-120b MXFP4 (59 GiB, two R9700s): 98.0 vs 56.6 tok/s decode, 1956 vs 720 tok/s pp512. Qwen3.6-35B-A3B UD-Q4_K_XL (one R9700): 108.3 vs 60.4 tok/s decode; at 64K depth 82.4 vs 43.2 tok/s decode and 2048 vs 548 tok/s prefill. Strix Halo runs Qwen3.5-122B-A10B UD-Q5_K_XL (85.6 GiB) at 22.3 tok/s and Nemotron-3-Super-120B-A12B UD-Q4_K_XL (78 GiB) at 14.9 tok/s; those exceed 64 GB of R9700 VRAM and need DDR5 expert offload on a desktop. Caveats: different llama.cpp builds (b8990 vs b9187/b9193) and OS versions; best backend chosen separately per platform; Q1-Q3 rows excluded.

Model

8 overlapping models (gpt-oss-120b, Qwen3.6-35B-A3B, Gemma 4, others)

Quant

Q4+

Engine

llama.cpp llama-bench (published grids)

Engine version

b8990 (R9700), b9187/b9193 (Strix Halo)

GPUs

1-2x Radeon AI PRO R9700 vs Ryzen AI Max+ 395 128 GB

Context

0, 32K and 64K depth

Flags

fa on, ngl 99, best of ROCm and Vulkan backends

• https://kyuz0.github.io/amd-r9700-ai-toolboxes/
• https://kyuz0.github.io/amd-strix-halo-toolboxes/
• https://github.com/lhl/strix-halo-testing/tree/main/llm-bench

Next Measure Qwen3.5-122B-A10B UD-Q5_K_XL on two R9700s with experts in DDR5, with the same tests, against Strix Halo's 22.3 tok/s.

Read from llama.cpp master d854d98bae (2026-10-25): server_lru_sched::has_capacity() compares the number of running models with models_max, and a sleeping model still counts as running, so it holds a slot after its memory is freed. --sleep-idle-seconds (PR #18228) unloads the model and KV cache after the idle time; GET /health, /props, /models and /metrics neither wake it nor reset the timer, so monitoring doesn't keep the cards busy. Presets accept any CLI flag as an INI key plus load-on-startup and stop-timeout (default 10 s). Router mode only runs llama.cpp and can't express rules like 'these two single-card models may coexist, nothing else may'. llama-swap v266 runs any OpenAI-compatible server as a child process and has three routing engines: group (one model at a time by default), matrix (expressions listing allowed combinations, evicting by evict_cost) and capacity (declared pools such as gpu0, gpu1 and ram, with a requirement per model). The capacity engine does no automatic measurement, so per-model footprints have to be measured first. Load, swap and wake times on R9700s are not measured yet.

Engine

llama.cpp master d854d98bae and llama-swap v266 (source and docs read)

Hardware

planned for 2x Radeon AI PRO R9700 + DDR5

• https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/server-models.cpp
• https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/README.md
• https://github.com/ggml-org/llama.cpp/pull/18228
• https://github.com/mostlygeek/llama-swap/blob/v266/docs/config.example.yaml
• https://github.com/mostlygeek/llama-swap/blob/v266/README.md

Next Measure each preset's per-card VRAM and DDR5 footprint, cold-load, swap-in and wake times on 2x R9700, then decide whether the always-on agent model goes behind the swap endpoint.

Read from llama.cpp master d854d98bae (2026-10-25): common/sampling.cpp logs "backend sampling is not compatible with grammar, disabling" and also disables it when a reasoning budget is active; rejection sampling was restored in PR #18137 because grammar sampling is widely used and slow. Published -bs results are CUDA-only: no change at 1 slot and +48% at 32 slots (issue #27050), about 8% with MTP speculative decoding on an RTX 5090 (PR #25532); PR #17004 recommends CUDA for now. In MaskBench (128K-token vocab, one core), llama.cpp's native grammar mask averaged 16.6 ms vs 64 us for LLGuidance, and llama.cpp handled 6,505 of 11,306 schemas vs 8,909. Building llama.cpp with LLGuidance helps JSON Schema requests only; tool-call grammars from the chat parser stay GBNF. Qwen3.6 has a 248,320-token vocab, about 1.9x the benchmark's. Not yet measured on R9700s; the cost should show most on fast models (~10 ms per token) and speculative lanes, least on DDR5-hybrid Flash.

Engine

llama.cpp master d854d98bae (source read)

Models

Qwen3.6 tokenizer (248,320 vocab)

• https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/sampling.cpp
• https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/chat-auto-parser-generator.cpp
• https://github.com/ggml-org/llama.cpp/pull/18137
• https://github.com/ggml-org/llama.cpp/pull/17004
• https://github.com/ggml-org/llama.cpp/pull/25532
• https://github.com/ggml-org/llama.cpp/issues/27050
• https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md
• https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json

Next Measure plain vs tools (auto and required) vs JSON-schema decode speed on a fast single-R9700 model and on dual-R9700 Flash, with and without -bs.

Cited results, not yet measured here. Strix Halo (gfx1151) on an 11 s clip: large-v3-turbo 0.84 s on Vulkan vs 0.63 s on ROCm/HIP. A single RX 9070 tester ran whisper.cpp on Vulkan and a ROCm diarizer on the same card at once without meaningful contention, and saw large-v3-turbo crash with a UTF-8 decode error (large-v3 was fine). whisper-server (master 231638a) turns flash attention on by default, picks the card with -dev N or WHISPER_ARG_DEVICE, and has built-in VAD. llama.cpp's own /v1/audio/transcriptions route (PR #21863) is for audio-capable LLMs, not Whisper models. Model sizes: large-v3-turbo f16 1.513 GiB, q8_0 0.814, q5_0 0.535; large-v3 f16 2.882, q5_0 1.007. On a dual-R9700 Flash setup those GiB compete with experts kept in VRAM, so the CPU lane is worth testing first.

Engine

whisper.cpp master 231638a (docs/source), llama.cpp PR #21863

Models

Whisper large-v3 / large-v3-turbo ggml (f16, q8_0, q5_0)

• https://github.com/ggml-org/whisper.cpp/pull/3823
• https://pypi.org/project/babelscribe/
• https://github.com/collabora/WhisperLive/issues/520
• https://github.com/maherr/onnxruntime-migraphx-rdna4/blob/main/witness/README.md
• https://github.com/ggml-org/whisper.cpp/blob/master/examples/server/server.cpp
• https://github.com/ggml-org/llama.cpp/pull/21863
• https://huggingface.co/ggerganov/whisper.cpp

Next Measure HIP, Vulkan and CPU-only Whisper lanes on a dual-R9700 box, and how much a loaded speech model slows Flash decode.

Estimate, not a measurement: at a 205 tok/s hybrid prefill a default-cap 1080p screenshot needs about 0.9 s of encoding plus about 10 s of prefill; at a 1,024-token cap about 5.3 s. llama.cpp allows 8 to 4,096 tokens per Qwen-VL image by default and warns that grounding tasks want at least 1,024; Strata caps pictures at 1,024 by default. Strata's GPU reader reserves about 1.4 GiB of VRAM (fewer cached experts, a few percent slower text) and can start on demand and stop after 120 s idle. llama.cpp can pin the encoder to one card with --mmproj-device or keep it on the CPU with --no-mmproj-offload. On a DGX Spark, GGUF plus projector scored 58/60 on a 60-image synthetic set, the same as NVFP4; both misses were output-length truncations.

Model

Qwen3.8-Flash-Next GGUF + mmproj-BF16/F16 (Unsloth)

Engine

llama.cpp master d854d98bae (mtmd), Strata v0.2.6 docs

• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/clip.cpp
• https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp
• https://github.com/Niko1221/Strata/blob/main/docs/AMD_HIP.md
• https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md
• https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md
• https://0xbakeer.com/flash-next/vision

Next Measure time to first token, VRAM per card and text decode with the encoder on and off on two R9700s at Q5 and Q6, at 1,024 and 4,096 image-token caps.

Check the llama-bench device line: RADV prints '(RADV GFX1201) (radv)' with shared memory 65536, AMDVLK prints '(AMD open-source driver)' with 32768; pin AMD_VULKAN_ICD=RADV. The dmesg line 'SMU driver if version not matched' (driver 0x2E vs card 0x32/0x33) shows up on every R9700 and AMD staff call it harmless. AMD says MES firmware 0x8B fixes a stuck-busy idle state (100% busy, ~100 W at idle), but one dual-R9700 owner saw about 10.8% lower decode on that package. On a three-card host, one card's SMU hung while idle and only a reboot cleared it; health checks that skip unreadable sensors go blind, so treat an unreadable sensor as an alarm. Docked or external cards can pick the wrong runtime power method; amdgpu.runpm=0 fixed one case. Our own measured single-card Qwen3.6-35B Vulkan decode (~101 tok/s) shows the decode-cliff report does not apply to our current build. Mesa and kernel deltas are single-tester results.

Model

various (per source)

Quant

per source

Engine

llama.cpp Vulkan (RADV / AMDVLK), amdgpu kernel driver

Engine version

per source

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://github.com/ggml-org/llama.cpp/discussions/21043
• https://github.com/ggml-org/llama.cpp/issues/22375
• https://github.com/ggml-org/llama.cpp/issues/26679
• https://github.com/ggml-org/llama.cpp/issues/26795
• https://github.com/ROCm/ROCm/issues/6101

Next Fingerprint the stack (kernel, Mesa, ICD, firmware, llama.cpp build) on both desktops and record it with every benchmark; A/B RADV vs AMDVLK only if AMDVLK is installed.

Sharded TP gives each card half of every active expert, so the work is equal every token. EP gives each card full copies of the experts it owns, and the all-reduce waits for the card that drew more of the token's 10 experts. Analytic check: E[max(X,10-X)]/5 with X~Binomial(10,0.5) = 1.246. Round-robin placement and EPLB do not help at batch 1 because the imbalance is per-token chance, not a hot-expert bias; EPLB also costs VRAM. Estimate: routed experts are roughly 40% of Flash-Next's per-token weight reads by parameter count, so the single-stream penalty would be about 10% on 2 cards and about 27% on 4, unless the half-width sharded expert kernel is much slower on RDNA4. This is a model, not a measurement; a TP vs TP+EP benchmark is planned.

Model

Qwen3.8-Flash-Next (config), 256-expert top-8 class for comparison

Engine

vLLM (behavior per docs and AMD blog)

GPUs

2x Radeon AI PRO R9700 (also modeled 4x)

Flags

-tp 2 vs -tp 2 --enable-expert-parallel

• https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html
• https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/
• https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py
• https://0xsero.github.io/blackwell-gpu-wiki/interconnect/moe-parallelism/

Next Benchmark plain TP2 vs TP2+EP in vLLM at 1, 4, 16 and 32 concurrent streams on a 4-bit-or-better MoE that fits two R9700s.

Shape method: active = total - 3 x hidden x moe_intermediate x experts x MoE layers x (1 - k/E), counting MTP layers as MoE layers. Checks against official active counts: DeepSeek-V3 37B (shortcut 21.4, shape 40.1), Qwen3-30B-A3B 3.3B (1.91 vs 3.35), Qwen3-235B-A22B 22B (14.7 vs 22.2), GLM-4.5-Air 12B (6.9 vs 15.0; its official count appears to leave out MTP and embeddings). For file sizes, the Hugging Face model API returns per-file sizes only when called with ?blobs=true. Split GGUFs must have their shards summed per quant folder, and helper files must be skipped: Qwen3.8-Flash-Next's GGUF repo carries a 3.9 GiB MTP draft head named mtp-...-Q4_K_M.gguf that a filename match reads as the model's Q4_K_M (the real UD-Q4_K_XL is 103.7 GiB). Checkpoints with FP4-packed experts also undercount parameters in the safetensors metadata: MiniMax-M3.1-Flash estimates to 124.5 GiB at Q4 but the measured UD-Q4_K_XL is 232.3 GiB. With measured sizes and a rough rule (2x 32 GB VRAM less ~6 GiB for KV/compute, experts in DDR5 with 16 GB left for the OS), 5 Flash-class MoE models fit two R9700s plus 128 GB DDR5 as a hybrid and 5 more need 256 GB; dense models of that size are not worth running from DDR5. A corrected /new-models data script is prepared.

Engine

Hugging Face Hub API + config.json

GPUs

fit tiers computed for 2x Radeon AI PRO R9700 (32 GB each) + DDR5

• https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json
• https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/config.json
• https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/config.json
• https://huggingface.co/zai-org/GLM-4.5-Air/blob/main/config.json
• https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json
• https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF?blobs=true
• https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF

Next Ship the corrected script, then show measured vs estimated sizes and two DDR5-hybrid fit columns (128 GB, 256 GB) on /new-models.

llama.cpp (decode ~1K/36K/163K; prefill 36K/163K; KLD text/code; tools): UD-Q5_K_XL 147 GB 17.6/17.1/14.2, 226/195, 0.036/0.019, top-1 92.2%/96.3%, 24/26. UD-Q4_K_XL 104 GB 21.2/19.9/16.0, 314/273, 0.060/0.034, top-1 90.1%/95.1%, 24/26. AP-Q5_K_M 104 GB 20.1/20.3/16.5, 330/301, 0.065/0.033, 24/26. AP-Q4_K_XL 94 GB 22.2/21.8/17.6, 416/359, 0.070/0.038, 24/26. Strata UD-Q5: 25/26 tools with thinking on; greedy exact match vs Q8_0 37.5% (llama.cpp UD-Q5 43.8%). vLLM community MXFP4: best tools 25/26 but lowest exact match 18.8%; KLD run failed, not speed-benched. KLD = llama-perplexity KL vs Q8_0 on held-out post-Oct-1 corpus (152K text / 51K code tokens); 26-case tool-call eval. Q3 estimate: not applicable for this project.

For MoE models it fills the last GPU with dense-only layers (experts in system RAM), then converts layers to full GPU layers front-to-back, letting parts of a layer overflow to the next GPU or RAM. Default margin is 1024 MiB per device and can be set per device (-fitt 512,1024). llama-bench now accepts -fitt and -fitc, so fitted and hand-written placements can be compared in one tool; llama-fit-params prints the chosen -c/-ngl/-ts/-ot. It is not available with -sm tensor. Integrated GPUs are eligible targets, so a desktop CPU's iGPU that shows up as a ROCm or Vulkan device should be excluded with --device. A user report on two Arc Pro B70s found an explicit -ts 1,1 silently spilling to one card plus RAM with fit left on (about 10x slower) until -fit off. Upstream's own CUDA table shows a second GPU can lower prefill while the model is still RAM-bound (Qwen3 Next q8_0 pp4096 904 to 574 t/s, tg 24.3 to 28.3). Fit measures memory before warm-up, so leave headroom for long prompts.

Engine

llama.cpp

Engine version

master 6050b467 (2026-10-25)

GPUs

2x Radeon AI PRO R9700 (applies to any multi-GPU + RAM setup)

• https://github.com/ggml-org/llama.cpp/discussions/18049
• https://github.com/ggml-org/llama.cpp/pull/18679
• https://github.com/ggml-org/llama.cpp/issues/22592
• https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md

Next Measure fitted vs hand-tuned placement for Flash-class Q4/Q5/Q6 on two R9700s with DDR5 experts, at 0 and 32K depth.

From llama.cpp master source (2026-10-25): the only graph compatibility check left is the MUL_MAT_ID sync fallback, and batch-1 quantized MoE decode takes the graph-safe path. Each GPU split gets its own graph, which needs two identical calls to warm up and resets on any shape change, so with experts in DDR5 a GPU's work is cut into many small graphs and the gain there is unmeasured. gfx1201 has no VMM, so llama.cpp uses its legacy pool, and every pool flush forces all graphs to be captured again. GGML_CUDA_GRAPH_CHUNK_SIZE (default 512 nodes) landed 2026-10-23 with +1-2% decode on CUDA; GGML_CUDA_GRAPH_OPT=1 (concurrent streams) has been allowed per split on multi-GPU since 2026-09-03, +3.4-3.7% MoE decode on CUDA. GGML_CUDA_GRAPH_DEBUG=1 with verbose logging prints warm-up, reset and reuse events. Build trap: CUDA/HIP builds fail between the chunked-graphs merge and its fix (commit 9ff0d7e90f), so don't pin a commit in that window. Graphs only exist in the HIP/CUDA backend; Vulkan launchers are unaffected. No R9700 measurement yet; an A/B is planned.

Engine

llama.cpp (source read)

Engine version

master 6050b467 (2026-10-25)

GPUs

Radeon AI PRO R9700 (gfx1201)

Flags

GGML_HIP_GRAPHS=ON (default)

• https://github.com/ggml-org/llama.cpp/pull/22254
• https://github.com/ggml-org/llama.cpp/issues/23579
• https://github.com/ggml-org/llama.cpp/pull/30804
• https://github.com/ggml-org/llama.cpp/pull/31053
• https://github.com/ggml-org/llama.cpp/pull/28198
• https://github.com/ggml-org/llama.cpp/pull/30327
• https://github.com/ggml-org/llama.cpp/issues/25082
• https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-cuda/ggml-cuda.cu

Next A/B on R9700: default vs graphs off vs chunk 0/128 vs GRAPH_OPT=1, single-GPU MoE and the dual-R9700 Flash hybrid, with graph event counts.

Ollama's Linux ROCm v7 list includes the R9700, but its Windows ROCm list has no RDNA4 cards, so Windows R9700 users run on Ollama's Vulkan backend. The mixed-card MoE crash ('no kernel image is available') was reproduced on Ollama 0.32.14 through 0.34.1 with R9700 + RX 9060 XT and R9700 + RX 7900 XTX; two identical R9700s are not affected. The rocBLAS fix (rocm-libraries #4781) merged 2026-03-02. On kernels without the gfx12 MES idle-power fix (in 6.12+ stable), ROCm runtimes can keep a card 100% busy at up to ~300 W between prompts until the model unloads (reported on dual R9700 in LM Studio). LM Studio's ROCm runtime has an open report of a 27B dense model hanging at 99% load on an R9700 where the Vulkan runtime loads it. llama.cpp's own multi-GPU doc marks -sm row deprecated and -sm tensor experimental with no performance guarantees outside CUDA, so layer split remains the safe default. Not measured on our hardware yet.

Engine

Ollama / LM Studio / Lemonade (llama.cpp-based)

Engine version

per source

GPUs

2x Radeon AI PRO R9700

• https://docs.ollama.com/gpu
• https://docs.ollama.com/faq
• https://github.com/ollama/ollama/issues/18162
• https://github.com/lemonade-sdk/llamacpp-rocm/issues/96
• https://lemonade-server.ai/docs/dev/backends-reference/
• https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1991
• https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1916
• https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md

Next Check whether the iGPU is visible as a device on our dual-R9700 desktop, and measure Ollama against raw llama.cpp on the same 35B Q5 file.

Rename old files (device_name=AMD_Radeon_AI_PRO_R9700 -> AMD_Radeon_R9700) and confirm the server log says 'Using configuration from' instead of 'Using default MoE config'. Flash-Next keys (int4_w4a16, the group size is not part of the key): tensor parallel 2 -> E=512,N=320; TP2 with expert parallel -> E=256,N=640; TP4 -> E=512,N=160; TP4 with EP -> E=128,N=640. On gfx1201, compressed-tensors W4A16 int packs reach the Triton MoE kernel that uses these files (the RDNA3 HIP kernel is gfx1100-only and Marlin is CUDA-only). benchmark_moe.py has no Qwen4Exp branch and stops with 'has no MoE experts'; a small wrapper that reads num_experts/moe_intermediate_size from text_config fixes it, and tuning needs only config.json. Payoff on other R9700 shapes from the cited PRs: 1.05-1.46x per MoE layer (int4, M=1-32), -4.2% ms/token single stream and +13.5% tok/s at 4 concurrent; FP8 TPOT -8% to -24%. Flash-Next numbers are not measured yet.

Model

Qwen3.8-Flash-Next (W4A16 int4 packs)

Engine

vLLM main (source read 2026-10-25)

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://github.com/vllm-project/vllm/pull/38086
• https://github.com/vllm-project/vllm/pull/36659
• https://github.com/vllm-project/vllm/pull/60893
• https://github.com/vllm-project/vllm/issues/50436
• https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py
• https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py

Next Tune the four Flash-Next keys on R9700 and measure default vs tuned per-layer time.

The slow starts showed thousands of EAGAIN retries of AMDKFD_IOC_ALLOC_MEMORY_OF_GPU (about 3,770 in one 198.6 s start). The reporter's unconfirmed explanation is that 2 MB huge-page faults trigger compaction, which invalidates the HMM walk so ROCr retries the whole range. Workaround: a tiny launcher that calls prctl(PR_SET_THP_DISABLE) and execs the server (no root); allocating in 2 GiB pieces was also fast in the good state (3.6 s). Separately, because RDNA GPUs have no XNACK, reclaim or compaction of these pageable pages evicts all of the process's GPU queues; an open Strata PR that mlocks every GPU-mapped host range reported no stalls in 30 minutes of heavy memory pressure (twice) versus stalls after 103 s and 63 s without it, and needs LimitMEMLOCK=infinity. The older HSA_USERPTR_FOR_PAGED_MEM=0 workaround avoids the stall but stops ROCr pinning the arena and roughly halved prompt reads in that test. AMD's IOMMU guidance is to keep the default remapping mode on Radeon GPUs that use peer-to-peer; iommu=pt is recommended for Instinct and for Radeon without P2P. THP itself is a small speed lever for hybrid inference; the risk it brings here is start time and stalls.

Engine

Strata (HIP build)

Engine version

0.1.44.1 (start test), 0.1.48.1 + patch (stall test)

GPUs

2x RX 7900 XTX (start-time test); 1x RX 7900 XTX (stall test)

Flags

THP enabled=madvise, defrag=madvise

Rocm

7.2.4 (start test)

• https://github.com/Niko1221/Strata/issues/2850
• https://github.com/Niko1221/Strata/pull/3913
• https://man7.org/linux/man-pages/man2/PR_SET_THP_DISABLE.2const.html
• https://rocm.docs.amd.com/en/docs-6.3.2/conceptual/iommu.html
• https://lists.openwall.net/linux-kernel/2026/06/25/875

Next A/B the same THP-off launcher on dual R9700 for the Q6 and Q5 Flash hybrid lanes (start time, decode and prefill), then a memlock pressure test.

Measurements are from vLLM issue #57838 (gemma-4-12B FP8 compressed-tensors, AMD rocm/vllm ROCm 10.1 image, vLLM 0.29.1.dev, fp8 KV, single stream, eight boots 38.43-38.54). The fix is not merged as of 2026-10-25; open PR #61091 adds a native Triton per-token FP8 backend (+14-17% per-request output rate vs RowWise at TP1, GSM8K -0.3 pt). Block-scaled FP8 checkpoints (128x128 weight blocks) are not affected by this kernel choice; they use AITER Triton blockscale for shapes in the gfx1201 tuned list (PR #43615) and generic Triton otherwise. MXFP4 MoE on RDNA4 runs through Triton's tl.dot_scaled decomposed onto 16-bit WMMA (PR #37826). llama.cpp's HIP backend runs GGUF k-quants on INT8 WMMA (PR #17156). No mainstream LLM engine uses the INT4 matrix rate yet; it needs W4A4 rotation and calibration, not a normal Q4 file. Separately, AITER unified attention on gfx12 gave silently wrong image answers (text fine) in the same issue thread; use --attention-backend TRITON_ATTN for vision models.

Quant

FP8 W8A8 per-token; MXFP4; GGUF k-quants (all >= 4-bit)

Engine

vLLM (ROCm) and llama.cpp HIP, cited

Engine version

vLLM 0.28-0.29.1.dev

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html
• https://rocm.docs.amd.com/projects/rocWMMA/en/develop/api-reference/data-type-support.html
• https://github.com/vllm-project/vllm/issues/57838
• https://github.com/vllm-project/vllm/pull/61091
• https://github.com/vllm-project/vllm/pull/43615
• https://github.com/vllm-project/vllm/pull/37826
• https://github.com/ggml-org/llama.cpp/pull/17156

Next Check the startup log of the dual-R9700 vLLM Flash lane for which FP8 and MXFP4 kernels it selects, and A/B any slow pick.

Kernel numbers are test-backend-ops MUL_MAT at n=512 on an RX 9070 XT from the PR author; the author's 2026-09-09 re-run on master shows a pure Q6_K Llama 8B at pp2048 2,320 -> 3,891 t/s (ub 512) and 2,425 -> 4,422 (ub 2048), Q4_K_M and Q5_K_M +10% (their Q6_K tensors), Q8_0, Q4_0, IQ4_NL and IQ4_XS flat. An R9700 user on ROCm 7.2.4 measured Qwen3.6-27B Q6_K pp512 684 -> 935 t/s (+36.7%) and a 35B MoE Q6_K 2,153 -> 2,757 (+28.1%). The merged RDNA4 MMQ tuning (PR #26199, July) predates those master numbers, and llama.cpp master as of 2026-10-25 has no fix. The Flash pack estimate uses real per-type tensor sizes from the GGUF headers and covers routed-expert matmuls only (attention and PCIe copies excluded); offloaded layers that are PCIe-bound will hide part of the gap. Practical takeaway: on the HIP backend prefer Q5 packs for Flash until #25940 merges, and don't assume HIP's measured Q4_K_M prefill lead over Vulkan carries over to Q6. Vulkan has a smaller Q6_K penalty (open PR #22970 adds +15% on Q6_K). Strata uses its own kernels and is not covered.

Model

Qwen3.8-Flash-Next (estimate); Llama 8B, Qwen3.6-27B, 35B MoE (cited)

Quant

Q4_K / Q5_K / Q6_K / Q8_0 and UD/AP packs, all >= 4-bit

Engine

llama.cpp HIP (ROCm)

Engine version

master vs PR #25940

GPUs

RX 9070 XT and Radeon AI PRO R9700 (gfx1201)

Context

pp512-pp4096

Flags

-ngl 99, -ub 512 / 2048

• https://github.com/ggml-org/llama.cpp/pull/25940
• https://github.com/ggml-org/llama.cpp/pull/25587
• https://github.com/ggml-org/llama.cpp/issues/27181
• https://github.com/ggml-org/llama.cpp/pull/26199
• https://github.com/ggml-org/llama.cpp/pull/22970
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Next Measure HIP master vs master + PR #25940 on Flash-Next UD-Q6_K_XL and UD-Q5_K_XL hybrid prefill on 2x R9700 (decode expected flat).

B70 notes: Mesa version is a 2x lever (Vulkan MoE decode 39.0 on Mesa 25.3.6 vs 86.1 on Mesa 26.2.5); SYCL is far faster for long prompts; upstream vLLM XPU serves FP16 only and has no BF16 path, with INT4/FP8/GPTQ via Intel's llm-scaler container. With MTP depth 2 on dense 27B a comparison video reports 46.2 (R9700) vs 28.4 (B70), consistent with the R9700's higher matrix throughput helping multi-token verification. A single-B70 Flash-Next result of 41.7 tok/s exists but uses a 3.05-bit pack, so it is excluded from this 4-bit-and-up comparison. Power: 230 W TBP (B70) vs 300 W (R9700). Newegg listings seen 2026-10-25: R9700 $1,749-$1,999.99, B70 $1,299.99-$2,878, so the B70's launch-price edge has mostly closed at that store. Cost per decode tok/s on two cards for Flash-Next 4-bit GGUF works out to roughly $55 (R9700) vs $78-117 (B70); this is an estimate built from different testers, engines and hosts, so treat all ratios as rough.

Model

Qwen3.6-35B-A3B, Qwen3.8-27B, Qwen3.8-Flash-Next

Quant

Q4 to Q5 GGUF, MXFP4 (all >= 4-bit)

Engine

llama.cpp Vulkan/SYCL/ROCm, Strata, vLLM (per source)

Engine version

per source

GPUs

1-2x Radeon AI PRO R9700 vs 1-2x Arc Pro B70

Context

short unless noted

Flags

per source

• https://riceharvest.github.io/awesome-b70/performance.html
• https://www.pugetsystems.com/labs/articles/intel-arc-pro-b70-multi-gpu-ai-inference-performance/
• https://github.com/0xSero/qwen38-b70
• https://www.intelinside.ai/runtimes/llamacpp
• https://www.youtube.com/watch?v=pNYNRUOnPeU
• https://github.com/ggml-org/llama.cpp/pull/25051
• https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md
• https://www.newegg.com/p/pl?N=100008333+601191745

Next Watch for a 4-bit-or-higher dual-B70 Flash-Next result on vLLM or llm-scaler and refresh prices before turning this into a full comparison page.

Header-only read of unsloth's UD-Q5_K_XL: per_layer_token_embd.weight is 50.66 GiB of 147.41 GiB total and the only tensor above the 4 GiB lazy threshold; UD-Q4_K_XL's table is 26.82 GiB (IQ4_NL). Current master: --lazy-mode auto reads tensors > 4 GiB on demand and still uses mmap even with --load-mode none; --load-mode auto means mmap on discrete GPUs (only iGPUs opt out), so dual R9700 setups get the lazy path by default. Cited costs, NVIDIA hosts: UD-Q4_K_XL, 24,576-token varied prompt, table in RAM 2825 tok/s prefill and 90.0 tok/s decode vs cold mmap 1293 and 85.9. A low-RAM box that can never keep the table cached saw cold 584 vs warm 2019 tok/s at 3.3K tokens, with the gap shrinking to almost nothing past 130K. Benchmark trap: prompts built from repeated filler text touch only a few table rows and always measure the warm case. The direct-read PRs (#28136, then #29030 '--lazy-mode on-direct') were both closed without merging. Flag change: #20834 folded the old flags into --load-mode (none = old --no-mmap, dio = old -dio) and #28334 removed the old flags on 2026-09-09. Not yet measured on R9700; a cold / warm / lazy-off A/B is planned.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL

Engine

llama.cpp (cited results on NVIDIA; source read on master b3fa69955e)

Engine version

master 2026-10-25

GPUs

cited: RTX PRO 6000; target: 2x Radeon AI PRO R9700

Context

24,576-token prompt (cited)

Flags

--lazy-mode auto vs off; --load-mode

• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q5_K_XL
• https://github.com/ggml-org/llama.cpp/pull/28136
• https://github.com/ggml-org/llama.cpp/pull/29030
• https://github.com/ggml-org/llama.cpp/pull/26081
• https://github.com/ggml-org/llama.cpp/pull/20834
• https://github.com/ggml-org/llama.cpp/pull/28334
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model-loader.cpp

Next A/B on 2x R9700 at Q5: cold table vs pre-warmed vs --lazy-mode off vs --lazy-mode off with --load-mode none, using varied real prompts at 3K/13K/30K tokens.

Deep prefill (pp131072, ROCm 7.14): defective card junction 110 C, edge 62 C, gap 47 C (peak 62), ~229-230 W at ~2790-2867 MHz, 297-300 t/s; healthy card gap ~38 C, ~297 W, ~3050 MHz. Swapping PSU cables between cards left the deficit with the card. After warranty service the same card measured a 37.4 C gap, 297 W, 3057 MHz and 323.2 t/s (+8.4% vs its own earlier run). The author treats the pair's tp=2 prefill of ~1841-1894 t/s as a lower bound next to a ~1965 t/s community 2x R9700 run, and gives ~37 C as healthy and a drift toward ~47 C as a warning. On Navi 48, fan control lives in gpu_od/fan_ctrl/fan_curve (five hotspot-temperature points, 25% minimum); staged points read back as if applied but only take effect after writing 'c' to commit. For scale, single-card sustained loads in reviews: 3,200-3,400 rpm and 57.9 dB(A) (igor'sLAB), 99 C hotspot without throttling in a 7.7-minute FurMark run, 98 C junction on a 32-minute agentic 35B run, and a 210 W cap cutting 97 C to 87 C for a 1.2% token-generation loss. Practical takeaway: before trusting any dual-R9700 benchmark, log each card's junction, edge, power and clock under a long load and compare the cards; a gap of 45 C or more, or one card stuck well below its power cap at a hot junction, points at the cooler rather than the software. Our own earlier note: hybrid DDR5-offload decode draws only ~180-190 W per card, so this mostly bites long prefill and fully in-VRAM tensor-parallel runs.

Model

per source (vLLM FP8 models; 35B MoE GGUF)

Quant

per source

Engine

vLLM ROCm tp=2, llama.cpp (published)

Engine version

ROCm 7.14, amdgpu DKMS 6.19.4 (thermal investigation)

GPUs

2x Radeon AI PRO R9700 (investigation); 1x in reviews

Context

pp131072 deep prefill

Flags

stock fan curve, 300 W cap

• https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-mem-vendor-bios-variance.md
• https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit
• https://docs.kernel.org/gpu/amdgpu/thermal.html
• https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/
• https://pendakwah.tech/reviews/r9700ai/
• https://branetechnologies.com/blog/local-35b-model-gpu-thermals-waterblock
• https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/

Next Baseline both of our R9700s under a 20-minute in-VRAM prefill load (per-card gap, power, clock, plus single-card runs) and re-check monthly.

Idle: ROCm #5706 single R9700 (MES 0x84, llama.cpp ROCm image) 92-95 W idle with a model loaded; GPU_MAX_HW_QUEUES=1 brought it to 3% / 21-22 W, while --no-warmup, --no-mmap and smaller context changed nothing. PyTorch resident processes stayed pinned even with =1. On ROCm 7.14 (MES 0x8b) a smaller residual remains: HIP 18-19 W vs Vulkan 12-13 W per card on the same 3x R9700 box (ROCm #6453, open). Speed: vLLM on one R9700 was bimodal per spawn (33.1-33.2 vs 25.7-26.2 tok/s); with =1, 8 of 8 cold spawns landed at 33.43-33.52 and the KFD queue count dropped 7 to 3. Two or more streams with pending hipStreamWaitEvent waits added ~17 us to every small kernel (gone at 2 or fewer queues; gfx1030 unaffected). Strata reported its MTP draft step bimodal per start (about 6% of decode on PCIe x16, up to ~25% on a chipset x4 link); =2 fixed every start on both links, =1 was slow on x4, =16 always slow (default 4). Estimate: a 24/7 resident two-card server stuck in the idle-busy state uses ~1,660 kWh/yr of card power (~$264/yr at the July 2026 Texas residential average) vs ~385 kWh/yr (~$61) with the cap. Single-GPU reports only so far; whether a dual-R9700 layer split shows a slow-start mode is untested, so we pin =2 before further A/B runs.

Model

various (Qwen3-14B FP8, Gemma 4 12B FP8, Qwen3.x 35B GGUF, Flash-Next via Strata)

Quant

per source

Engine

llama.cpp HIP, vLLM ROCm, Strata (published)

Engine version

ROCm 7.2.x to 10, per source

GPUs

1-3x Radeon AI PRO R9700

Context

per source

Flags

GPU_MAX_HW_QUEUES unset vs 1 vs 2

• https://github.com/ROCm/ROCm/issues/5706
• https://github.com/ROCm/ROCm/issues/6453
• https://github.com/ROCm/legacy-rocm-build/issues/6347
• https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a

Next Spawn-lottery A/B on 2x R9700 (unset vs 1 vs 2, 8 cold starts each) for llama.cpp UD-Q5 (then Q4, Q6) and Strata Q6, plus idle watts with the model resident.

Pinned host memory is also needed: in the PR's own tests the default mmap load gave no gain (2029 to 2088 tok/s) while --load-mode none went from 2705 to 5486 on Qwen3.5-35B-A3B with all experts in RAM. Merged PR #30449 splits CUDA-family pinned buffers into 4 GiB pieces so 50+ GiB of experts can be pinned. Other requirements: -sm layer, all layers offloaded, op offload and KQV offload on, and async plus events on every device; the Vulkan backend reports both, but no Vulkan result exists yet. CUDA results from the PR author on 3 GPUs with Qwen3.8-Flash-Next Q8_0: 52K-prompt prefill 1009 to 1808 tok/s and pp32768 1334 to 2474, with tg64 46.2 to 47.1. For Flash-Next on two cards (48 layers, 0-23 on card 1), a spread list that keeps the high-imatrix-energy layers 36-47 in VRAM also balances VRAM per card, so -ts 1,1 should fit. Not yet measured on R9700; expect decode within 2% and prefill gains that grow with prompt length.

Model

Qwen3.8-Flash-Next (layout analysis); DeepSeek-V4-Flash and Qwen3.5-35B-A3B (cited tests)

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (analysis); MXFP4, Q8_0 (cited)

Engine

llama.cpp PR #29963 (head 3ca7eed, open)

GPUs

2x Radeon AI PRO R9700 (analysis); 4x MI50 ROCm 7.14 and 3x RTX PRO 4000 (cited)

Context

131072-262144

Flags

-sm layer, experts in RAM via -ot spread over both cards, --load-mode none

• https://github.com/ggml-org/llama.cpp/pull/29963
• https://github.com/ggml-org/llama.cpp/pull/30449
• https://github.com/ggml-org/llama.cpp/pull/24156

Next A/B on 2x R9700 with HIP: master --n-cpu-moe vs the PR with a spread override list and --load-mode none, Q4 then Q5 then Q6, pp8192/pp32768/tg64.

Estimated decode ceilings, ECC on vs off (capacity only / capacity plus +9% GPU bandwidth): UD-Q4_K_XL 43.2 vs 46.1 / 48.5 tok/s, UD-Q5_K_XL 32.1 vs 34.2 / 35.4, UD-Q6_K_XL 28.7 vs 29.9 / 30.9. GPU-resident MoE layers go from 29/22/20 to 32/25/22 of 48. Our earlier dual-R9700 ceilings (45/34/30) assumed the full 32 GiB, so with ECC left on they are about 3-5% too high. The +9% figure is one source (Qwen3.6-35B-A3B Q4_K, 101 to 110 tok/s on one R9700 in a Vulkan engine) and is not yet reproduced. The trade-off: VRAM bit flips go undetected, so stress-test first and keep ECC on if the kernel log ever reported ECC errors. Check the state with cat /sys/module/amdgpu/parameters/ras_enable (-1 is the default, ECC on) and dmesg | grep GECC.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL

Engine

llama.cpp (model, not measured)

GPUs

2x Radeon AI PRO R9700

Context

262144, f16 KV

Flags

layer split, experts of the first N layers in DDR5 (-ncmoe), no MTP

• https://github.com/ROCm/ROCm/issues/5832
• https://lunnova.dev/articles/amdgpu-disabling-ecc/
• https://forum.level1techs.com/t/sapphire-radeon-r9700-not-32gb/244649
• https://zolotukhin.ai/zinc/docs/rdna4-tuning/
• https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md

Next Measure ECC on vs off with llama-bench on two R9700s at Q4 and Q5, 32K depth, with a soak test before keeping it off.

Two concrete facts from the llama.cpp source: --tensor-type / --tensor-type-file take regexes and the first match wins, and Flash-Next's expert down tensors have 640 columns, so 256-block k-quants silently fall back (Q4_K to Q5_0, Q5_K to Q5_1, Q6_K to Q8_0). That is why Unsloth's UD files use Q5_1/Q8_0 downs, and why a hand-made 'Q4_K everywhere' recipe still gets 5.5-bit downs; IQ4_NL or Q4_0 are the only 4.5-bit options. Unsloth's own Q8_0-down layer picks (2, 4, 30, 46, 47) do not follow imatrix energy, so energy is a proxy, not a sensitivity measure. Keeping stock UD-Q4's Q8_0-down layers on GPU with -ot is a slight loss in the model (45.4 vs 46.1 tok/s). All speeds are roofline estimates (GPU at 75% of 640 GB/s, 55 GB/s DDR5, 262K f16 KV, layer split), not benchmarks.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / custom mix (all >=4-bit)

Engine

llama.cpp (model)

GPUs

2x Radeon AI PRO R9700

Context

262K

Flags

layer split, --n-cpu-moe, f16 KV

• https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://unsloth.ai/docs/models/qwen3.8-next

Next Build the mix from BF16 with Unsloth's imatrix and measure held-out KLD and decode against UD-Q4 and UD-Q5 on the local desktop; adopt only if KLD is within 15% of UD-Q5 at 10%+ more speed.

Bandwidth estimate for 2x R9700 with experts in DDR5 (UD-Q4, 45 tok/s at short context): about 35 tok/s at 131K and 29 at 262K with dense attention and f16 KV, versus about 44 and 43 if the sparse path and pooled cache both apply. The f16 KV cache at 262K is only about 6 GiB across both cards (q8_0: 3.2 GiB), so the VRAM it would free is worth a few percent at most. Sparse flash attention (PR #28770) was built for CUDA; whether HIP batch-1 decode reaches it on gfx1201 is not yet verified. A patched CUDA fork went from 10.2 to 33.7 tok/s at 250K on 6x RTX 3090, which shows how much is at stake. The k-pool indexer layout (#29751) doubles the indexer cache and mirrors it on every GPU under tensor split (issue #30005).

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL

Engine

llama.cpp (HIP)

Engine version

per source

GPUs

2x Radeon AI PRO R9700 (estimate); cited runs on gfx1151, gfx1201, A100, RTX 3090

Context

0-262K

Flags

layer split, -fa on, --parallel 1

• https://github.com/ggml-org/llama.cpp/issues/27856
• https://github.com/ggml-org/llama.cpp/issues/28012
• https://github.com/ggml-org/llama.cpp/pull/28699
• https://github.com/ggml-org/llama.cpp/pull/28770
• https://github.com/ggml-org/llama.cpp/issues/28734
• https://github.com/ggml-org/llama.cpp/issues/27796
• https://github.com/ggml-org/llama.cpp/issues/30005

Next Measure decode at 0, 32K, 128K and 200K on 2x R9700 with f16 and q8_0 KV on a current HIP build. If 128K decode is below 75% of short-context decode, look at bringing the sparse decode path to HIP.

Same model as our earlier roofline: GPU at 75% of 640 GB/s, measured 55 GB/s DDR5 read, 2.5 GiB reserve per card, 262K f16 KV, layer split, no MTP. These are upper-bound estimates, not benchmarks; llama.cpp has measured 55-66% of the ceiling on Flash-Next. GLM-5.3-Flash is attention-heavy (6.15 GiB of attention weights read every token at UD-Q4; 12 MLA + 34 linear layers). DeepSeek-V4-Flash and MiniMax-M3.1-Flash store experts as native 4-bit (MXFP4 / NVFP4) in every Unsloth file, so their Q8 files only upgrade the dense part and there is no Q5/Q6 choice. MiniMax keeps full KV in 60 layers (about 30 GiB at 262K, estimate). DeepSeek-V4.1-Flash-1017 is 466 GiB at UD-Q4, too big for 256 GB RAM plus 64 GB VRAM, and its architecture is not in llama.cpp master yet.

Model

Qwen3.8-Flash-Next, GLM-5.3-Flash, DeepSeek-V4-Flash, MiniMax-M3.1-Flash

Quant

UD-Q4_K_XL to UD-Q6_K_XL (UD-Q8_K_XL where experts are native 4-bit)

Engine

bandwidth model (llama.cpp layout)

GPUs

2x Radeon AI PRO R9700

Context

262144

Flags

layer split, f16 KV, experts spilled to DDR5, no MTP

• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF
• https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF
• https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF
• https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp

Next Measure DeepSeek-V4-Flash UD-Q4 against GLM-5.3-Flash UD-Q5 on the local desktop to check the ~1.6x estimate; Qwen3.8-Flash-Next stays the main dual-R9700 lane.

Open caveat (issue #28867): the new dispatch sends small multi-token batches (speculative-decoding verify batches, merged parallel slots) to the WMMA kernel once n_q x gqa_ratio_eff > 16. One R9700 owner measured MTP decode on Qwen3.8-27B (24 Q / 4 KV heads) fall from 51.7 to 41.4 tok/s, recovered by raising the threshold to 64 with no prefill loss; another owner on a tuned fork saw the opposite. Reading the quoted gate, Flash-Next (24 Q / 2 KV heads, effective GQA factor 4) keeps plain decode and single-stream MTP with up to 3 draft tokens on the tile kernel, so it should not hit this; two or more parallel slots with MTP would. This is a reading of quoted source, not a measurement. The old rocWMMA path also crashed around 106K tokens of prefill with HIP graphs on gfx1201 (#24961; GGML_CUDA_DISABLE_GRAPHS=1 avoided it), another reason to move to a post-#28102 build instead of reverting.

• https://github.com/ggml-org/llama.cpp/pull/28102
• https://github.com/ggml-org/llama.cpp/issues/26220
• https://github.com/ggml-org/llama.cpp/issues/28867
• https://github.com/ggml-org/llama.cpp/issues/24961

Next Measure Flash-Next UD-Q5_K_XL prefill at 0/32K/128K on two R9700s: pre- vs post-#28102 ROCm builds vs Vulkan, with a kernel trace to confirm which attention kernel runs (its sparse-attention indexer path is unverified).

Estimates use a bandwidth roofline (75% of 640 GB/s per card, 55 GB/s measured DDR5 read), 2-4 network round trips per token, and a ~3% per-token RPC software tax reported for a 1 local + 1 RPC GPU layer split. Do not put only the experts on the remote box: that layout makes two round trips per layer per token. In vLLM, keep tensor parallel inside each box and use pipeline parallel across boxes; tensor parallel across a network link all-reduces every layer. llama.cpp -sm tensor currently asserts on Qwen3 MoE with 2x R9700 on ROCm, so use -sm layer. Not yet measured on hardware.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimate)

Engine

llama.cpp RPC, -sm layer (estimate)

GPUs

2 desktops x 2 Radeon AI PRO R9700

Context

262K, f16 KV, single request

• https://github.com/torvalds/linux/commit/1881f2efbf7f78dc0a79a387b29fde6ff56d3731
• https://qiita.com/faliye/items/2a2d8def5298f6c1c10e
• https://github.com/kjaiswal/llama-cpp-distributed-benchmarks
• https://github.com/ggml-org/llama.cpp/pull/24122
• https://github.com/ggml-org/llama.cpp/issues/22307
• https://docs.vllm.ai/en/stable/serving/parallelism_scaling/
• https://jingchaozhang.github.io/LLM-Inference-Across-NVLink-and-InfiniBand/
• https://www.asrock.com/MB/AMD/X870%20Taichi%20Creator/

Next Measure it: iperf3 and RTT on the 10GbE link, then llama-bench with --rpc at Q4, Q5 and Q6 once the second desktop is built.

Estimate from measured GGUF header sizes, keeping the n-gram embedding table in system RAM, 262K f16 KV, 2.5 GiB reserve per card, 75% of 640 GB/s, 55 GB/s DDR5. GPU bytes needed: UD-Q4_K_XL 82.9 GiB, UD-Q5_K_XL 102.8, UD-Q6_K_XL 113.0, versus about 59 free on 2 cards, 88.5 on 3 and 118 on 4. Layer-split decode ceilings with no MTP: Q4 45 tok/s on 2 cards vs ~74 resident on 3; Q5 34 vs ~69 resident on 4; Q6 30 vs ~67 on 4 (only ~5 GiB slack). Once resident, the Q4/Q5/Q6 ceilings converge, so a 4-card box would favor Q5 or Q6. Layer split only removes the DDR5 bottleneck; the much larger 219 tok/s figure comes from 4-way tensor parallel. Stock vLLM requires the tensor-parallel size to divide the attention-head count and recommends pipeline parallel for uneven GPU counts, so 3-way TP likely needs a non-stock engine. A cited 3x R9700 tensor-parallel run (~182-191 tok/s) used a fine-tune with an unreported quant, so it is not counted here. Different rigs; not a measurement on our hardware.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (model); MXFP4 experts + FP8 dense (cited)

Engine

llama.cpp layer split (model); vLLM TP4 (cited)

GPUs

2-4x Radeon AI PRO R9700

Context

262K (model); 32K (cited)

• https://amd-pro-bench.com/gpus/r9700/4-gpus
• https://amd-pro-bench.com/gpus/r9700/3-gpus
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://docs.vllm.ai/en/stable/serving/parallelism_scaling/
• https://github.com/vllm-project/vllm/issues/11797

Next Measure a 3-card resident UD-Q4_K_XL run and a 4-card vLLM TP4 run if extra cards are pooled into one box.

HRX early data is from RDNA3/RDNA3.5: AMD claims about 30-50% prefill uplift and parity to +15% non-MTP decode vs the faster of Vulkan and HIP. Community Strix Halo (gfx1151) runs on Qwen3-30B-A3B Q4_K_M show tg128 85.1 vs 68.0 tok/s for HIP (+25%), but another run had Vulkan ahead (89.5 vs 85.3); HRX could not chunk prefill past 1024 tokens, and one development branch produced garbage text while llama-bench still reported healthy speed, so check output, not just tok/s. Lemonade ships an HRX engine recipe (merged 2026-10-02) without MTP. The ik_llama.cpp HIP repair also fixed AMD GPUs being detected as NVIDIA (NaNs in MoE matmul) and a software byte-permute that made 4-bit table quants about 15x slower on HIP (IQ4_XS 17.16 to 266.31 tok/s on a small model, RDNA3). Its maintainer says ROCm has not been a project goal and he cannot test it, so expect breakage. Takeaway for dual-R9700 owners today: stay on mainline llama.cpp, Strata or vLLM; ik_llama.cpp is the one alternative worth a gated try for the CPU-expert path.

Engine

ggml-hrx draft, ik_llama.cpp, KTransformers (per sources)

GPUs

Radeon AI PRO R9700 (gfx1201); sources cover gfx1100/1101/1151

• https://github.com/ggml-org/llama.cpp/discussions/27219
• https://github.com/ggml-org/llama.cpp/pull/27218
• https://github.com/lemonade-sdk/lemonade/pull/3723
• https://github.com/ikawrakow/ik_llama.cpp/pull/2339
• https://github.com/ikawrakow/ik_llama.cpp/discussions/562
• https://ktransformers.net/en/docs/hardware-platforms/cpu-gpu-requirements

Next Gated test of ik_llama.cpp (HIP) vs mainline on 2x R9700 with a Q5 Flash model: build, correctness check, then speed with identical expert placement. Watch for HRX gfx1201 support.

The same comparison saw a failure that BFCL missed: one engine printed a tool call into its reply text instead of dispatching it, in a real two-turn agent job. Tool-call checks should count calls leaked into text separately. A pruned community GGUF of the same model also reports a run-to-run band of about 3 cases out of 40 at temperature 0 across backends. Sample sizes are from Connor's approximation (alpha 0.05 two-sided, 80% power) and are estimates; repeats of the same case are not independent, so real needs are higher. We built a 40-case paired tool-call harness for OpenAI-compatible servers (llama.cpp, Strata, vLLM) and will run Q4/Q5/Q6 on two R9700s.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL (published BFCL run); UD-Q4/Q5 KLD from Unsloth

Engine

llama.cpp (published)

GPUs

published run on one RTX 6000 Blackwell, not R9700

Context

131K served

Flags

temp 0.6, 3 repeats, Q8_0 KV, MTP 3 draft tokens

• https://www.demietrich.com/blog/q3.8fn_engine_comparison/
• https://unsloth.ai/docs/models/qwen3.8-next
• https://huggingface.co/Cyronius/Qwen3.8-Flash-Next-131B-A6B-GGUF

Next Run the paired 40-case tool-call harness on Q4, Q5 and Q6 Flash on two R9700s with identical server flags, 5 repeats each, and publish pass rates, leaked-call counts and McNemar results.

API: OpenRouter qwen/qwen3.8-flash on 24 Oct 2026 lists $0.15 per million input, $0.47 output and $0.016 cached input; the weighted average actually paid for input is $0.0278 because of caching. Electricity: EIA July 2026 residential averages, Texas 15.88 and US 18.31 cents per kWh. Power is an estimate, not measured on our machine: 315 W for both cards (Puget Systems measured 310-315 W for two R9700s at one user under vLLM TP=2 on a 27B model) plus 150 W for a consumer desktop or Puget's ~300 W workstation overhead. Prefill is where local wins: about $0.005-0.014 per million input tokens versus $0.15 uncached, though cached API input ($0.016) is close. If batching scaled the way Puget measured on the 27B model (also an estimate, not measured on Flash), 4-8 streams would bring local to about $0.05-0.09 per million output tokens. Card prices: $1,699.99-$1,999.99 per R9700 on Newegg in October 2026. Idle draw matters for an always-on server: a ROCm 7.14 issue reports 18-19 W per idle R9700 versus 12-13 W on Vulkan.

Model

Qwen3.8 Flash

Quant

MXFP4-FP8-GPTQ (vLLM) / UD-Q5_K_XL (Strata)

Engine

vLLM TP2, Strata

GPUs

2x Radeon AI PRO R9700, local desktop

Context

36K test

• https://openrouter.ai/qwen/qwen3.8-flash
• https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a
• https://energyfactbook.com/electric-power-monthly/
• https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/
• http://www.nowinstock.net/computers/videocards/amd/aipror9700/
• https://github.com/ROCm/ROCm/issues/6453

Next Measure GPU and CPU power on our machine during vLLM and Strata runs, plus vLLM aggregate speed at 2, 4 and 8 streams, to replace the estimates.

Sourced data points: llama.cpp PR #28123 (merged 2026-09-01) gave Flash-Next native recurrent rollback; with an MTP head at n-max 3 on an RTX PRO 6000 it went from 108 tok/s (no draft) to 183 on code and 144 on prose, vs 123 / 83 before. Open PR #28118 measured the host checkpoint at about 600 ms of an 825 ms round on Strix Halo (Vulkan, Flash-Next Q4_K_M, MTP): 32.4 tok/s without drafts, 6.2 with host checkpoints, 41.5 with on-device checkpoints. Open issue #28019: ngram-map-k4v with 3 parallel slots on a GB10 gave about 50 tok/s aggregate vs 66-70 with speculation off, and reports a recurrent-rollback test mismatch on master that affects MTP when slots are reused across conversations. Elsewhere, ngram-mod took a 177B MoE from 22.09 to 77.56 tok/s on a file edit (90.3% accepted) with no change on prose. Our estimate for UD-Q5_K_XL on 2x R9700 (bandwidth model, uniform routing, single slot): rewrite turns 0.87x at defaults, about 1.9x with GGML_OP_OFFLOAD_MIN_BATCH=256 if checkpoints are cheap, about 1.0x if they cost 600 ms; mixed agent turns 0.47x at defaults. Not measured on R9700 yet.

Model

Qwen3.8-Flash-Next

Quant

UD-Q5_K_XL (model), Q4-Q6 lane

Engine

llama.cpp

Engine version

master 71255fe (source read)

GPUs

2x Radeon AI PRO R9700

Context

262K

Flags

-sm layer, routed experts partly in DDR5, -np 1

• https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp
• https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp
• https://github.com/ggml-org/llama.cpp/pull/28123
• https://github.com/ggml-org/llama.cpp/pull/28118
• https://github.com/ggml-org/llama.cpp/issues/28019
• https://ai-muninn.com/en/blog/qwen4exp-177b-three-2080ti
• https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md

Next Measure on 2x R9700: baseline vs ngram-mod (defaults, MIN_BATCH=256, n-max 30) on rewrite, agent and prose turns, plus the checkpoint cost. MTP with native rollback is the likelier win once a Q4-or-better MTP head is confirmed.

Pack revision 6fe3e74: experts MXFP4 at 4.25 bpw, n-gram table int6, MTP head 4.25 bpw, so every tensor is 4-bit or higher. 262K context, fp8 KV, card-default max-num-batched-tokens 4096. VRAM 27.55 / 27.40 GB, system RAM 90 GB including a 38.7 GB pinned n-gram table. Startup takes about 10 minutes. MTP acceptance 51.8%. Tool calls parse cleanly and reasoning comes back in a separate 'reasoning' field. 36K prefill runs were 3809 / 3804 / 3789 tok/s on a clean relaunch. An earlier run right after a 110 GB download got only about 1,030 tok/s prefill and logged a 'couldn't lock n-gram table' warning; page-cache pressure is the likely cause but this is unproven, so relaunch on a quiet system before benchmarking. Failed: max-num-batched-tokens 8192 at 262K, 16384 at 131K and 8192 at 131K all fail KV-cache sizing, and no flag to cap it was found. Quality (KL divergence vs a reference) of this pack has not been measured yet. For reference, a CUDA vLLM fork with an expert cache reports 85-90 tok/s on two 3090s, but with a W4A16 fine-tune of the model, so it is not a like-for-like comparison.

Model

Qwen3.8 Flash

Quant

MXFP4-FP8-GPTQ (tcclaviger), rev 6fe3e74

Engine

vLLM

Engine version

0.29.0.dev0 (tcclaviger/vllm:latest)

GPUs

2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)

Context

262K

Flags

tensor parallel 2, fp8 KV, MNBT 4096, MTP

Next Measure KL divergence / top-token match of the MXFP4 pack against a Q8_0 reference before calling it the default.

Qwen3.8 Flash UD-Q5_K_XL. The n-gram table (per_layer_token_embd, Q8_0, 50.66 GiB) stays in the DDR5 page cache, with only small buffers on the GPUs, so it costs no VRAM.

Model

Qwen3.8 Flash

Quant

UD-Q5_K_XL

Engine

Strata

Engine version

0.2.3

GPUs

2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)

Context

36K test

Flags

layer split, MTP

Next No further Strata tuning planned; vLLM MXFP4 is now faster.

Warm story/code decode, then 36K decode/prefill, then hit rate: fit only 18.2/18.2, 17.0/231; +6544 MiB 22.0/19.6, 18.0/201, 79.75%; +12000 MiB 23.4/21.0, 19.4/183, 88.74%; -cmoe +40000 MiB 23.5/21.5, 19.6/162, 93.72%. Q6 at the -cmoe +40000 setting: 22.3/20.4, 18.8/147. --moe-cache-mib is a total split across both GPUs, and it can't be combined with tensor split ('MoE cache requires a GPU backend').

Model

Qwen3.8 Flash

Quant

UD-Q5_K_XL (Q6 for comparison)

Engine

llama.cpp (HIP, dsv4 build)

Engine version

dsv4 build

GPUs

2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)

Context

262K

Flags

f16 KV, --moe-cache-mib 6544 / 12000 / 40000 with -cmoe

Next Use a modest cache size if prefill matters; the prompt-batch hit rate is the limiter.

q8 KV, ubatch/batch 2048, -t 16, 262K context. VRAM 24.8 / 27.8 GB. Failed: Vulkan with ubatch 4096 hit an amdgpu ring timeout (device lost) at about 29K tokens; stay on HIP or a smaller ubatch.

Model

GLM 5.3 Flash

Quant

Q5_K_M (bartowski), 227 GiB

Engine

llama.cpp (HIP, dsv4 build)

Engine version

dsv4 build

GPUs

2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)

Context

262K

Flags

q8 KV, -ub 2048 -b 2048, -t 16

Next Revisit if Strata or vLLM adds GLM 5.3 support.

Same build and settings as the GLM run, 262K context for the main numbers. VRAM 29.1 / 29.0 GB.

Model

DeepSeek V4 Flash

Quant

UD-Q4_K_XL (unsloth), 144.4 GiB

Engine

llama.cpp (HIP, dsv4 build)

Engine version

dsv4 build

GPUs

2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)

Context

262K (1M also loads)

Flags

q8 KV, -ub 2048 -b 2048, -t 16

Next None planned.

Read from llama.cpp tools/llama-bench/llama-bench.cpp at master df9beac (2026-10-24): each row has 43 fields plus per-run samples_ns/samples_ts; model_type comes from llama_model_desc (e.g. 'qwen3next 80B.A3B Q5_K - Medium'), so the quant can be checked without the GGUF; test_time is UTC. For like-for-like comparisons, rows must match on model_type, n_prompt, n_gen, n_depth, type_k/type_v, flash_attn and n_ubatch. We wrote a converter that keeps only setup fields, reduces the path to the file name, rejects quants below Q4 and flags noisy rows (over 5% spread), fewer than 3 runs and missing long-context rows; it will back a result-submission form when the site's next build lands.

Engine

llama.cpp llama-bench

Engine version

master df9beac

• https://github.com/ggml-org/llama.cpp/blob/df9beac/tools/llama-bench/llama-bench.cpp

Next Suggested sharing command: llama-bench -m MODEL.gguf -fa 1 -p 512 -n 128 -d 0,8192,32768 -r 3 -o jsonl

Prefill is the exception: on one R9700, HIP leads at long prompts (pp2048 4635 vs 3675, pp4096 4325 vs 3386, Qwen3-30B-A3B UD-Q4_K_XL), but offloaded-expert prefill is PCIe-bound, so that lead should shrink in hybrid runs (to be measured). Vulkan tensor split (llama.cpp PR #25051) is still open: on 2x R9700 it reaches 45.0 tok/s on dense Qwen3.8-27B Q4_K_M (ROCm tensor 40.2, layer split 27), but for a MoE that fits one card it is slower than a single GPU (144.8 vs 165.2). Whether tensor split works with CPU-resident experts for Flash is untested. A real multi-agent workload on one R9700 also found HIP 1.3-1.4x slower at every concurrency level. The hybrid percentages are estimates from our bandwidth model (GPU share of each decode step 37-64%), not measurements.

Model

Qwen3.6-35B-A3B / Qwen3.8-27B (published); Qwen3.8-Flash-Next (estimate)

Quant

Q4_K_M (published); Q4-Q6 (estimate)

Engine

llama.cpp

Engine version

master + PR #25051 (published)

GPUs

1-2x Radeon AI PRO R9700

Context

tg128 / pp512-4096

Flags

-fa 1

• https://github.com/ggml-org/llama.cpp/pull/25051
• https://github.com/ggml-org/llama.cpp/discussions/22463
• https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html
• https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/

Next Same-commit Vulkan vs HIP A/B on Flash-Next Q5 with experts in DDR5 on the local desktop; test Vulkan tensor split with CPU-resident experts once the PR merges.

Source reading (llama.cpp master 3642238, 2026-10-24): for MUL_MAT_ID the offload batch size is op->ne[2], the tokens in the ubatch; sched_copy_experts() then uploads only the experts the router used (MoE-cache hits come from VRAM), after a synchronous readback of expert ids per layer. Estimates below are bandwidth-only, assume uniform routing (the least overlap between slots, so real sharing should be better), use our measured 55 GB/s DDR5 read and ~23 GB/s achieved on one PCIe 5.0 x8 link, and ignore CPU compute limits, which may cap the CPU path past about 8-16 tokens per step. Qwen3.8-Flash-Next UD-Q5_K_XL, layer split, 23 expert layers in RAM: 34 tok/s at 1 slot, 47 total at 4 slots, 52 at 8, 67 at 31, then about 31 at 32 tokens per step with the default threshold. AP-Q4_K_XL: 60, 97 and 111 at 1, 4 and 8 slots. So parallel slots add only about 1.4-1.5x total throughput at 4-8 slots in this worst-case routing model, and per-agent speed drops fast. Not yet measured on hardware.

Model

Qwen3.8-Flash-Next

Quant

UD-Q5_K_XL, AP-Q4_K_XL (estimates)

Engine

llama.cpp (source reading + bandwidth model)

Engine version

master 3642238

GPUs

2x Radeon AI PRO R9700

Context

262K f16 KV pool, 8K fill per slot

Flags

-sm layer, -ncmoe per quant, -np N

• https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-cuda/ggml-cuda.cu
• https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-vulkan/ggml-vulkan.cpp
• https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/src/llama-context.cpp
• https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/tools/batched-bench/README.md

Next Measure with llama-batched-bench at 1-48 parallel sequences, default threshold vs GGML_OP_OFFLOAD_MIN_BATCH=256, Q5 first.

Why it matters: with Flash-class experts in DDR5, prefill is PCIe-bound and Q5/Q6 decode is DDR5-bound, so a silently halved link or a BIOS that fell back to 3600 MT/s costs real speed. The R9700 is a two-slot blower (about 267 x 111 x 39-40 mm), so slot spacing is a non-issue on any board with two CPU-wired x16 slots. Ryzen 9000 has 28 PCIe 5.0 lanes with 4 to the chipset, so x8/x8 is the AM5 maximum; boards like the Strix X870E-E have only one CPU x16 slot. The Apex is 2-DIMM (128 GB max), too small for 147-158 GiB Q5/Q6 Flash GGUFs. TRX50 with Threadripper 9000 runs four channels of DDR5-6400 RDIMM (204.8 GB/s theoretical) and x16/x16 Gen5. Estimated 2x R9700 decode ceilings (layer split, no MTP, 262K f16 KV, 8K fill), all from a bandwidth model calibrated to a measured 55 GB/s read on 4x64 GB DDR5-5200 and scaled at the same efficiency, not measured on the other platforms: UD-Q5_K_XL 27 / 34 / 37 / 52 / 63 tok/s and AP-Q4_K_XL 52 / 60 / 64 / 77 / 85 tok/s for AM5 at 3600 / 5200 / 6000, TRX50 and WRX90. Check with lspci -vv under load that both cards show Width x8 at 32GT/s.

Model

Qwen3.8-Flash-Next GGUFs

Engine

bandwidth model (estimate) + manufacturer specs

GPUs

2x Radeon AI PRO R9700

Context

262K

• https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x870e-creator-wifi/techspec/
• https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM5/ROG_CROSSHAIR_X870E_APEX/E25780_ROG_CROSSHAIR_X870E_APEX_EM_WEB.pdf
• https://rog.asus.com/ea/motherboards/rog-crosshair/rog-crosshair-x870e-dark-hero/spec/
• https://rog.asus.com/us/motherboards/rog-strix/rog-strix-x870e-e-gaming-wifi7-neo/wtb/
• https://www.sapphiretech.com/en/commercial/radeon-ai-pro-r9700
• https://en.wikipedia.org/wiki/Template:AMD_Ryzen_9000_Series
• https://www.tweaktown.com/news/105825/msi-releases-amd-agesa-1-2-0-3e-bios-officially-supports-4-64gb-sticks-up-to-256gb-ram/index.html
• https://www.kingston.com/en/memory/server-memory/memory-population-rules-wrx90-trx50

Next Verify both cards train at x8 Gen5 under load and DDR5 runs at its set speed on our dual-R9700 machine.

Bytes that matter for decode (everything except the n-gram table): UD-Q4_K_XL 76.8 GiB, AP-Q5_K_M 76.7, UD-Q4_K_XL with IQ4_NL downs (bitlamas DN4) 68.6, AP-Q4_K_XL 67.3, UD-Q5_K_XL 96.7, UD-Q6_K_XL 106.8. Two R9700s hold 64 GiB, so every option spills some experts to DDR5. On Unsloth's own KL table, Q4 to Q5 cuts mean KL by 35% (0.0469 to 0.0304), and Q5 to Q6 cuts it by only 11% more (0.0271). AP-Q5_K_M keeps its trunk at Q6_K instead of Q8_0, so its always-read dense part is about 0.9 GiB smaller. Our bandwidth model puts its decode ceiling on 2x R9700 at about 50 tok/s, against 45 for UD-Q4_K_XL, 34 for UD-Q5_K_XL and 30 for UD-Q6_K_XL. These are estimates only: layer split, no MTP, no expert cache, 55 GB/s measured DDR5. agentionai warns that this model's n-gram table has memorised wikitext, which pushes top-1 agreement 6.9 to 8.6 points higher than on held-out text. That means wikitext KL figures understate how much a quant hurts. Agentionai's lower tiers (AP-Q4_K_M and below) use IQ3 or IQ2 experts and fall below a 4-bit floor. Quality numbers from different publishers use different references and can't be compared with each other.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4/Q5/Q6_K_XL, AP-Q4_K_XL, AP-Q5_K_M, AP-Q5_K_XL, Q4_K_XL-DN4

Engine

llama.cpp (GGUF header audit + bandwidth model)

GPUs

2x Radeon AI PRO R9700 (modeled)

Context

262K f16 KV (modeled)

Flags

layer split, n-gram table in system RAM

• https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF
• https://unsloth.ai/docs/models/qwen3.8-next
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://huggingface.co/bitlamas/Qwen3.8-Flash-Next-Q4_K_XL-DN4

Next Measure held-out KL and decode speed for AP-Q5_K_M, AP-Q4_K_XL, UD-Q4_K_XL and UD-Q5_K_XL on 2x R9700 with identical flags.

Per Radiance's docs the 2-bit draft head is only a coarse drafting pass: candidates are re-scored against bf16 weights and the target verifies every token, so output should not change. Turning speculation off doesn't help, because the 2-bit tensor is still in the file. The publisher's 2x R9700 TP2 recipe for the vLLM pack claims 100 tok/s single-request decode and 323 tok/s at 16 concurrent, using 82 GiB of system RAM, with experts that don't fit VRAM held in RAM and the n-gram table streamed from NVMe. Radiance's card implies about 160 tok/s (15 ms per step at about 2.4 accepted tokens). These speeds come from the publishers and we haven't checked them. The quality figures use different references and can't be compared directly: the vLLM pack scores mean KL 0.055 and 95.2% top-1 against BF16, and the Radiance pack 0.079 overall (0.024 on text turns) and 92.9% top-1.

Model

Qwen3.8-Flash-Next

Quant

MXFP4 experts / int6 n-gram / FP8 attention (vLLM); w4 codebook + int8 (Radiance)

Engine

vLLM (tcclaviger image), Radiance

Engine version

HF revisions 6fe3e74 and 8267a67

GPUs

2x Radeon AI PRO R9700

Context

262K / 200K

Flags

header audit only, no inference run

• https://huggingface.co/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ
• https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe
• https://hub.docker.com/r/stilldeadcode/vllm-radiance/
• https://codeberg.org/StillDeadcode/radiance

Next Run the vLLM pack's TP2 recipe on two R9700s and measure decode, prefill and memory against our GGUF Q5/Q6 results.

Found by a new release-radar script that scans llama.cpp, vLLM, ROCm, Lemonade and related repos daily for RDNA4/HIP/ROCm changes. If you build llama.cpp with HIP for an R9700 against a fresh toolchain, update to b11847 or newer first. Prebuilt Vulkan builds are not affected by this compile error.

Engine

llama.cpp (HIP backend)

Engine version

b11847

GPUs

Radeon AI PRO R9700 (gfx1201)

• https://github.com/ggml-org/llama.cpp/pull/30885
• https://github.com/ggml-org/llama.cpp/issues/30839
• https://github.com/ggml-org/llama.cpp/releases/tag/b11847

Next Use b11847 or later for the dual-R9700 HIP test lane.

From llama.cpp master (commit 8e5a58df87, 2026-10-24): the scheduler offloads big-batch ops on CPU-resident weights to the first GPU that accepts them, so with -sm layer on two cards all expert copies use one card's PCIe link while the other link idles (source reading, not yet measured). Under mmap the loader replaces pinned host buffers with plain CPU memory; PR #28223 (keep explicit host overrides) was closed unmerged and PR #26659 is still open, so on stock master only --load-mode none pins the experts. That costs RAM for the whole file (UD-Q4/Q5/Q6_K_XL are 103.7 / 147.4 / 157.5 GiB, including the 27-51 GiB n-gram table), which fits on a 256 GB DDR5 machine. Published evidence: on an RX 9070 XT (gfx1201, Vulkan) with Qwen3.6-35B-A3B Q4_K_M offloaded, going from -b 1024 -ub 512 to -b 8192 -ub 2048 took 48.5K prefill from 705 to 1,566 tok/s, while -ub 4096 was slower (808); on CUDA, pinned experts took Flash-Next UD-Q6_K_XL prefill from 166 to 330-379 tok/s at 26K with decode flat; on a 3090, gpt-oss-120b went from 380 to 2,091 tok/s between -ub 512 and 8192 with decode down 7%. Short prompts pay the full copy too (about 2 s for ~46 GiB of Q5 experts), and raising GGML_OP_OFFLOAD_MIN_BATCH to 2048 cut one 300-token prompt's time to first token from 29 s to 7.7 s on a box with 135 GB of experts. Our ceilings are estimates from measured GGUF byte counts, using UD-Q5_K_XL expert sizes as a stand-in for Q5_K_M.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimates); baseline Q5_K_M (measured)

Engine

llama.cpp master source + llama.cpp Vulkan baseline

GPUs

2x Radeon AI PRO R9700 (PCIe 5.0 x8 each)

Context

262K planned, 36K baseline

Flags

-sm layer, --n-cpu-moe 24-28, -ub 512/2048/4096

• https://github.com/ggml-org/llama.cpp/issues/25859
• https://inovello.dev/writeups/pinned-host-experts-under-mmap/
• https://github.com/ggml-org/llama.cpp/pull/28223
• https://github.com/ggml-org/llama.cpp/pull/26659
• https://github.com/ggml-org/llama.cpp/pull/19976
• https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend.cpp
• https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide
• https://github.com/gamozolabs/llama.cpp
• https://bittide.aicompass.dev/article/c9df2d89-8522-46ec-aa3f-fc1aed5b17d3
• https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html

Next Measure on our two R9700s, Q5 first: -ub 512 vs 2048 vs 4096 with -b 8192, then --load-mode none, RADV's transfer queue, HIP vs Vulkan, a higher offload threshold for short prompts, and whether only one card's link carries the copies.

Each Flash-Next checkpoint costs 112.6 MiB, plus 2,072 bytes per prefix token when MTP is on (about 555 MiB at 224K). Current llama.cpp flags are -ctxcp/--ctx-checkpoints (default 32), -cms/--checkpoint-min-step (default 8192) and -cram/--cache-ram (default 8192 MiB); --checkpoint-every-n-tokens no longer exists. Hybrid checkpoints can still be invalidated by upstream issue #24055; the fix, PR #25592, was unmerged at last check, and users who cherry-picked it report 35-41 s re-prefills dropping to 0.1-1.3 s. On the client side, anything that changes the start of the prompt (a per-minute timestamp, a mid-session model switch) forces a full re-read; Hermes Agent made its timestamp date-only for this reason. Strata has a separate host-RAM conversation cache: two alternating 57K-token chats went from 9.2 s to 2.1 s prefill on re-entry, at about 1.5 GB of RAM per conversation. These are cited numbers from other rigs; our own dual-R9700 measurement is still to come.

Model

Qwen3.8-Flash-Next

Engine

llama.cpp server / Strata (published reports)

Flags

--ctx-checkpoints, --cache-ram, --conversation-cache-mib

• https://kksoftwareag.github.io/strix-halo-qwen-3.8-flash-next/memory.html
• https://github.com/ggml-org/llama.cpp/issues/18497
• https://github.com/ggml-org/llama.cpp/pull/25592
• https://particula.tech/blog/prompt-reprocessing-swa-hybrid-models-kv-cache
• https://github.com/NousResearch/hermes-agent/pull/27675
• https://thomaswildetech.com/software-development/ai-agents/strata/

Next Measure per-turn cache hit and turn latency for Flash on our dual R9700s, Strata versus llama.cpp with and without checkpoint flags.

Plain banded expert offload at UD-IQ4_XS on 2x 3090 gives 35-38 tok/s. Excluded from the comparison: Strata IQ3_XXS 111 tok/s, tqLLM 77 tok/s at 128K (IQ3_S, one 3090, CUDA only, no multi-GPU yet), EXL3 2.5 bpw 38.6 tok/s, and 150-160 tok/s posts with no quant named. On the 3090 box the cache hit rate depended on slot count, not quant (Q4 and Q6 both 84-85% at 135 slots); Q4 won only by fitting more slots, and -ub 512 freed about 50 slots of VRAM versus 2048. Two R9700s have 64 GB of VRAM against 48 GB; our estimate is about 302 / 236 / 213 cache slots per layer at Q4 / Q5 / Q6 with MTP, against 150 on the 3090 pair (estimate, R9700 compute buffers unmeasured). Reading the merged llama.cpp expert-cache code (PR #29887): every selected expert gets its own real slot, batches up to 32 tokens use the cache, and the draft model's context has it off, so the duplicate-id bug reported on the older draft cache PR (#27861) should not apply to MTP + cache on master. Also from the 3090 write-up: q8_0 KV was slower (-4.8% short, -18% at 131K), MTP acceptance collapsed to ~20% at 131K depth where an n-gram draft won, and DIMM thermal throttling halved decode after a few minutes until a fan was added.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL

Engine

llama.cpp with expert cache + MTP

GPUs

2x RTX 3090 (comparison, published)

Context

261888, f16 KV

Flags

experts in host RAM, -ub 512, MTP draft n-max 3

• https://inovello.dev/writeups/qwen3-flash-next-2x3090-q4-mtp/
• https://www.reddit.com/r/LocalLLM/comments/1w069fh/qwen38flashnext_on_2x_rtx_3090s_3343_toks/
• https://x.com/geldeki/status/2095560770163458426
• https://x.com/coffeecup2020/status/2113265219283660816
• https://github.com/r0b0tlab/qwen38-flashnext-exl3
• https://github.com/ggml-org/llama.cpp/pull/29887
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-moe-cache.cpp

Next Run the same recipe on two R9700s with llama.cpp master (expert cache sized per card, -ub 512, MTP on and off), Q5 first. The target is beating 41 tok/s at full 262K context with Q5.

Also seen: a Ryzen iGPU visible to ROCm makes llama.cpp split onto it and segfault (hide it or pin the device); an open ROCm 7.14 report shows a llama.cpp generation hang that ends in an MES queue failure and a GPU reset failure until reboot, while Vulkan passes on the same machine; a missing TensileLibrary_lazy_gfx1201.dat means an incomplete rocBLAS install. For context, Level1Techs measured Qwen3.6-35B-A3B FP8 decode at 17.4 tok/s with tp=2 and 15.7 with tp=4 on R9700s (vLLM 0.20.2, NCCL_PROTO=Simple), so more cards did not speed up a small-active MoE at batch 1. Whether ROCm 7.14's newer RCCL avoids the deadlock without the env var is not confirmed, so we keep NCCL_PROTO=Simple pinned. We wrote a read-only pre-flight script that checks a machine for all of these; our own A/B of GPU_MAX_HW_QUEUES on decode speed is still to come.

Engine

vLLM ROCm images, llama.cpp HIP and Vulkan

Engine version

per source

GPUs

2-4x Radeon AI PRO R9700 (gfx1201), published reports

• https://github.com/vllm-project/vllm/issues/40980
• https://github.com/ROCm/rocm-systems/issues/5480
• https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504
• https://github.com/ROCm/ROCm/issues/5706
• https://github.com/lemonade-sdk/llamacpp-rocm/issues/96
• https://github.com/ROCm/legacy-rocm-build/issues/6630
• https://www.reddit.com/r/LocalLLaMA/comments/1vjtokl/dual_radeon_ai_pro_r9700_server_much_slower_than/

Next Measure GPU_MAX_HW_QUEUES=1 against unset on our dual-R9700 Flash config for decode speed, and run a small tp=2 vLLM smoke test with NCCL_PROTO=Simple before the vLLM lane.

Per token, time = bytes read from VRAM / GPU bandwidth + expert bytes read from DDR5 / DDR5 bandwidth. Dense weights read every token (attention and DeltaNet, lm_head, router, shared expert): about 4.46 GiB at UD-Q5_K_XL; routed experts: 10 of 512 per layer. With experts in DDR5, DDR5 is 57-77% of each token's time at Q5/Q6, so ideal tensor split helps Q4 by about 39% but Q5/Q6 only by 23-27%. Q4 to Q5 costs about 24% of ceiling, Q5 to Q6 about 13%. If every Q5 expert fit in VRAM (four cards), the ceiling would be about 69 tok/s on R9700 vs about 101 on RTX 3090: the 3090's bandwidth edge only shows once DDR5 is out of the loop. Assumptions: GPU decode at 75% of peak bandwidth, 2.5 GiB compute reserve per card, 262K f16 KV read to an 8K fill, tensor split modeled with no all-reduce cost. It ignores CPU compute limits, kernel launch overhead and PCIe, so real numbers land below these ceilings.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL

Engine

model (bandwidth roofline), checked against measured llama.cpp Vulkan and Strata 0.2.3 runs

GPUs

1-2x Radeon AI PRO R9700

Context

262K f16 KV, 8K fill

Flags

DDR5 4x64 GB at 5200, measured 53.8-56.2 GB/s read

• https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622

Next Check the model's slope with a llama.cpp --n-cpu-moe and thread sweep, and run Strata Q5 with MTP on vs off to split MTP gains from expert-cache gains.

The 0.50 GB token embedding table stays in system RAM (llama.cpp's default for input-layer tensors), so it doesn't count against VRAM. KV size: 10 full-attention layers x 2 KV heads x (256 + 256) dims x 2 bytes x 262,144 tokens = 5.00 GB. GB here means GiB. Also useful for anyone building tools: Hugging Face allows browser CORS Range requests on GGUF files (the resolve URL permits the Range header and its CDN returns 206 with an open CORS header), so a web page can read these headers without downloading weights or running a proxy.

Model

Qwen3.6-35B-A3B

Quant

Unsloth UD-Q5_K_XL

Engine

llama.cpp (Vulkan)

GPUs

1x Radeon AI PRO R9700 (32 GB)

Context

262144

Flags

f16 KV, all layers on GPU

• https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Next Measure actual VRAM in use on the card to pin the compute-buffer number exactly, and use it as the default in amdai.net's fit calculator.

Baseline: llama.cpp v0.6.0 Vulkan, Q5_K_M, --n-cpu-moe 28, about 23 GB VRAM per card; short-context decode 16.9-20.3 tok/s, at 36K decode 20.0 / prefill 205. Strata 0.2.3.1 (engine 0.1.48 HIP), UD-Q6_K_XL, MTP, int8 KV, 256K: warm decode 31-57 (median ~40); at 36K decode 22.8-51.1 / prefill 1204-1833; at 131K decode 35.0-47.4 / prefill 1242-1864; VRAM 29.8/29.6 GB, RAM 126.9 GB. Strata 0.2.3, UD-Q5_K_XL, MTP, int8 KV, 256K: story ~52, code ~65 (48-68); at 36K decode 43.9-52.4 / prefill 2043-2514; at 131K decode 47.6-54.3 / prefill 2519-2882; VRAM 29.8/29.6 GB, RAM ~112 GB, MTP acceptance 70-75%, swap flat. --tune quick kept the defaults. Q5 also used about 15 GB less system RAM than Q6 (about 112 vs 126.9 GB). Command: strata launch --model ud-q5_k_xl --family unsloth --context 256k --kv int8 --mtp on --gpu 0,1 --parallel 1 --set STRATA_ROUTE_TAIL_SKIP=0

Model

Qwen3.8 Flash

Quant

UD-Q5_K_XL, UD-Q6_K_XL (baseline Q5_K_M)

Engine

Strata 0.2.3 / 0.2.3.1 (engine 0.1.48 HIP); baseline llama.cpp v0.6.0 Vulkan

GPUs

2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write)

Context

256K (Strata)

Next Measure Q4 (UD-Q4_K_XL) on the same Strata setup to confirm the 2-7% Q5 penalty we expect from published numbers.

Thread count matters: without MTP, -t32 gave only 8.9-9.3 tok/s because the CPU was oversubscribed; -t16 gave 20.3-20.5. With MTP at -t16: story 22-26, code 32, VRAM 29.9/28.2 GB. At 140K context: 36K decode 22.8 / prefill 175; 131K decode 16.2 / prefill 144. Strata at the same 131K depth: decode 47.6-54.3 / prefill 2519-2882.

Model

Qwen3.8 Flash

Engine

RDNA4 llama.cpp fork, commit 185252d, ROCm 7.2.0 (gfx1201)

GPUs

2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write)

Context

140K

Flags

-sm layer -ts 1,1, q8 KV, FA, MTP shared Q8_0 head n-max 3, -t16

Next Use -t16 (physical cores minus headroom), not all threads, for hybrid CPU-expert runs.

Undervolt pays only when the card is actually power-limited (sustained prefill near 300 W). -150 mV was stable at 300 W but crashed with a page fault at 330 W on a 32K prompt after a 2K prompt passed, so short benchmarks don't prove stability. Verify memory clocks with pp_dpm_mclk, not OD_MCLK, which reads back the requested value even when it is ignored. The common 210 W cap with about -75 mV is reported as ~30% lower peak power for a few percent speed loss (community self-reports). The same pattern shows on datacenter GPUs: on H200, decode never hit a power cap and clock locking saved up to 32% decode energy at minimal throughput loss. PCIe ASPM performance mode was reported +10.8% decode on a dense 27B and ~0% on MoE. Not yet measured on our dual-R9700 hybrid-offload setup.

Model

Qwen3.6-27B Q4_K_M (undervolt and memory-clock tests)

Engine

llama.cpp (published)

GPUs

Radeon AI PRO R9700 (published single-card tests)

• https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md
• https://www.reddit.com/r/ROCm/comments/1vuu58x/
• https://www.reddit.com/r/ROCm/comments/1vr5j5e/
• https://github.com/ggml-org/llama.cpp/discussions/21043
• https://arxiv.org/abs/2605.11999

Next A/B test power caps, a locked memory clock, COMPUTE profile and PCIe ASPM on dual R9700s with Flash experts in DDR5, where idle gaps between expert steps may make clock ramp-up matter more.

Headline dual-R9700 Flash results of 216-272 tok/s (vLLM + Paiton W3A8) and 130 tok/s (SGLang, EXL3 3.05 bpw) are 3-bit, so they are excluded from our comparisons, which only cover Q4 and up. Other useful reports: splitting a model that fits on one card across two R9700s cost about 7% decode in llama.cpp (75.4 to 70.3 tok/s, Qwen3.6-35B-A3B Q4_K_M), and a chipset PCIe 4.0 x4 slot cost about 12% (66.2 tok/s). For reference, our own measurement on one R9700 plus DDR5 is 32.6 tok/s for Qwen3.8 Flash UD-Q6_K_XL on Strata versus 16 on llama.cpp. We plan to add a 4-bit native lane (vLLM MXFP4 at TP2 and Radiance W4) to the dual-R9700 sweep after confirming every expert tensor is at least 4-bit.

Model

Qwen3.8-Flash-Next

Quant

W4 / MXFP4-FP8 (4-bit class)

Engine

Radiance 1.4.0; vLLM + r9700-stack (as reported)

GPUs

2x and 4x Radeon AI PRO R9700 (as reported)

Context

up to 256K (as reported)

• https://x.com/ECander84852/status/2110182801639408108
• https://x.com/dad__vibes/status/2111137765396210053
• https://x.com/dad__vibes/status/2111573866996977673
• https://x.com/dad__vibes/status/2111908353802998061
• https://x.com/sincoice/status/2113225429679210835

Next Confirm Radiance and r9700-stack expert formats are at least 4-bit, then measure both at TP2 against the best GGUF Q5 result.

Estimate, not measured: two R9700s at 262K context with f16 KV leave about 46.8 GiB for experts (assuming 3 GiB per card for compute buffers). An offline LRU trace study posted on the PR suggests that cached share would give roughly 97.6% / 95.2% / 94.1% hits for UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL, all in the range where the cache paid off on an R9700. On ROCm, tensor mode's all-reduce is host-staged; a community direct-P2P patch measured +7.5% to +11.2% decode on 2x R9700 with dense Q8_0 models. Linux R9700 pairs report about 28 GB/s GPU-to-GPU and 21.5 GB/s CPU-to-GPU copy bandwidth.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (planned)

Engine

llama.cpp master (HIP)

GPUs

2x Radeon AI PRO R9700 (planned target)

• https://github.com/ggml-org/llama.cpp/pull/29887
• https://github.com/ggml-org/llama.cpp/pull/30112
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp
• https://github.com/ggml-org/llama.cpp/pull/25028
• https://github.com/JohnTDI-cpu/llama-hip-p2p-allreduce
• https://forum.level1techs.com/t/does-the-lrlink-pcie-bridge-fix-rccls-broken-p2p-on-dual-r9700-update-yes-it-does/249623

Next Measure UD-Q5_K_XL at 262K on two R9700s: the current -ncmoe baseline vs --fit with a partial cache vs all experts cached, with real hit rates, then Q4 and Q6 and a tensor-split check.

llama.cpp master tags per_layer_token_embd as an input-layer get-rows tensor, so it stays on the CPU like token_embd. Where Strata puts it is not yet checked. Estimated -ncmoe starting points at 262K context with f16 KV: 17 for Q4, 24 for Q5, 26 for Q6 (assumes 3 GiB compute reserve per GPU and KV from the model card's 12 sparse-attention layers, about 6.75 GiB at 262K). These start points are estimates until measured.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL

Engine

GGUF header analysis (no inference)

GPUs

2x Radeon AI PRO R9700 (planned target)

• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp

Next Check n-gram table placement in Strata and run an -ncmoe sweep around the estimated start points on the local desktop.

Stock llama.cpp on one R9700 with --n-cpu-moe manages only 18-20 tok/s at Q4 on the same model (published). llama.cpp lacks a VRAM expert cache on HIP; the moe-cache work is CUDA-only and unmerged.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL

Engine

llamacpp-club3090 moe-cache (3090) vs Strata (R9700)

Engine version

per sources

GPUs

2x RTX 3090 vs 2x Radeon AI PRO R9700

Context

204,800 (3090) / 40K (R9700)

Flags

per sources

• https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md
• https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md
• https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622

Next Reproduce on the local desktop with Strata 0.2.3 and the RDNA4 Flash-Next llama.cpp fork.

Qwen3.8-Flash-Next file sizes: UD-Q4_K_XL 103.7 GiB, UD-Q5_K_XL 147.4 GiB, UD-Q6_K_XL 157.5 GiB. All fit in 64 GiB VRAM plus DDR5 with experts spilled to RAM. GLM-5.3-Flash Q6_K_XL (271.8 GiB) is too large for comfort, so Q5 is its ceiling. DeepSeek-V4-Flash has no Q5/Q6 GGUF; UD-Q4_K_XL is the pick. Published numbers come from an EPYC 8-channel DDR4 host, not this rig.

Model

Qwen3.8-Flash-Next

Quant

UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL

Engine

Strata (published)

Engine version

per source

GPUs

2x Radeon AI PRO R9700

Context

40K

Flags

layer split, 8-bit KV, MTP, every expert in RAM

• https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md
• https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
• https://unsloth.ai/docs/models/qwen3.8-next

Next Run the ordered Q5 benchmark sequence on the local desktop, with Q4 and Q6 as secondary runs.