{ "as_of": "2026-11-02T07:14:30-06:00", "entries": [ { "id": "2026-11-02-vulkan-tensor-parallel-merged", "date": "2026-11-02T07:10:00-06:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp's Vulkan tensor parallel is now in master: a second R9700 makes a dense 27B ~50% faster, but one card still wins for Qwen3.6-35B-A3B", "summary": "PR #25051 (Vulkan all-reduce for -sm tensor) merged on 2026-11-02. On a maintainer's 2x R9700 system, dense Qwen3.8-27B Q4_K_M decodes at 45.0 tok/s with Vulkan tensor split vs 30.0 on one card (+50%) and 40.2 on ROCm tensor split. For Qwen3.6-35B-A3B Q4_K_M one card is still faster (165.2 vs 144.8 tok/s); tensor split only adds ~10% prefill. So for the 35B, run one full copy per card.", "details": "Tensor mode is still marked experimental. It needs flash attention (turned on automatically), falls back to CPU sampling, and builds its split device from GPUs only. Earlier versions of the PR showed problems you should check for yourself: decode dropping from 43.6 to 9 tok/s at a 164K context (2x W7800), a second server request slowing to 2 tok/s on R9700s, and garbled output on some MoE quants. No long-context R9700 result has been posted for the merged code yet, so benchmark at your real context and do a greedy-output diff against -sm layer before switching. Leave GGML_VK_COMM_D2D unset; in June it was 45-90% slower on R9700s. Whether DDR5 expert offload (--n-cpu-moe) works together with -sm tensor is untested.", "setup": { "model": "Qwen3.8-27B / Qwen3.6-35B-A3B / Llama-3.1-8B", "quant": "Q4_K_M (Q8_0 for 8B)", "engine": "llama.cpp Vulkan (RADV) vs ROCm (published)", "engine_version": "PR #25051 head; merged as dcbb091", "gpus": "2x Radeon AI PRO R9700", "context": "pp512 / tg128", "flags": "-sm none | layer | tensor, -fa 1" }, "metrics": [ { "label": "Dense 27B Q4_K_M decode, 1 card Vulkan", "value": 30.0, "unit": "tok/s" }, { "label": "Dense 27B Q4_K_M decode, 2 cards Vulkan tensor", "value": 45.0, "unit": "tok/s" }, { "label": "Dense 27B Q4_K_M decode, 2 cards ROCm tensor", "value": 40.2, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode, 1 card Vulkan", "value": 165.2, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode, 2 cards Vulkan tensor", "value": 144.8, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M prefill pp512, 1 card vs tensor", "value": "4444 vs 4887", "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/25051", "https://github.com/ggml-org/llama.cpp/commit/dcbb0916772195bee457d23751edf8c1cbdc707e" ], "next": "Measure -sm tensor on two R9700s at 32K-256K context, and test whether Flash-class models with DDR5 expert offload can use it." }, { "id": "2026-11-02-moe-cache-cpu-share", "date": "2026-11-02T05:10:00-06:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp's MoE-cache CPU share: pin the experts first; the share helps small caches, not a two-R9700 Flash setup", "summary": "Open llama.cpp PR #30966 lets the CPU compute a share of expert-cache misses from system RAM instead of uploading them over PCIe. Published results: big caches pick share 0 (the author's Flash-Next run recommends a 24000 MiB cache with share 0, and one single-R9700 PCIe 5.0 x16 user saw about 15% slower decode with any share), while 8-16 GiB cards running a 35B-A3B gain 29-57% at share 0.95. Our estimate for Qwen3.8-Flash-Next at Q4-Q6 on two R9700s (~48 GiB cache) says the share adds little once experts are pinned; pinning is the bigger lever.", "details": "The PR's docs say experts are pinned only with -lm none, mlock or dio (not the default mmap), and pinned uploads run at full link speed, so the CPU share often does not help. Estimate (bandwidth model, not a benchmark; ~45 GB/s DDR5, PCIe 5.0 x8 per card): two R9700s with experts pinned, Q4 gains nothing and Q5/Q6 at most +12-19% if the CPU part overlaps GPU work; unpinned uploads (~10 GiB/s assumed) would cost Q6 roughly 51 to 39 tok/s. With Flash on one R9700 (~18 GiB cache at 262K context), the share is worth an estimated +6-23% even when pinned. Caveats from the PR thread: any share above zero kept all 8 CPU cores busy in one R9700 report; Vulkan on RADV needs open PR #31139 for caches with banks over 4 GiB; early Vulkan gibberish reports were traced to a stale build.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimate)", "engine": "llama.cpp PR #30966", "engine_version": "head 8e876eb (open)", "gpus": "2x Radeon AI PRO R9700 (and 1x)", "context": "262K", "flags": "-cmoe, --moe-cache-mib, --moe-cache-cpu-share" }, "metrics": [ { "label": "Author Flash-Next IQ4_NL, 19392 MiB cache, share 0 / 0.95", "value": "65.5 / 54.1", "unit": "tok/s" }, { "label": "5060 Ti 16 GiB, 35B UD-Q6_K_XL, share 0 / 0.95", "value": "38.8 / 61.1", "unit": "tok/s" }, { "label": "Single R9700 PCIe5 x16, any share > 0", "value": "-15", "unit": "% decode" }, { "label": "Est. 2x R9700 Flash-Next UD-Q5 pinned, best share gain", "value": "0-12", "unit": "%" }, { "label": "Est. 1x R9700 Flash-Next UD-Q5 pinned, best share gain", "value": "6-22", "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/30966", "https://github.com/ggml-org/llama.cpp/pull/31233", "https://github.com/ggml-org/llama.cpp/pull/31286", "https://github.com/ggml-org/llama.cpp/pull/31139" ], "next": "Measure pinned vs mmap experts first, then the PR's tuner, on two R9700s at Q5 and Q6, and check long-context output stays coherent." }, { "id": "2026-11-02-r9700-gaming-vs-9070xt", "date": "2026-11-02T04:20:00-06:00", "project": "idea-lab", "kind": "result", "title": "The R9700 games like an RX 9070 XT, but not while Qwen3.6-35B is loaded", "summary": "Independent reviews put the Radeon AI PRO R9700 within about 3% of an RX 9070 XT in games (PC Games Hardware's four-resolution index), with Cyberpunk 2077 at 1440p max settings running 127.3 vs 128.3 FPS (Back2Gaming), on AMD's regular Adrenalin driver. The extra 16 GB doesn't help games: Cyberpunk used about 8.4 GB. With Qwen3.6-35B-A3B (UD-Q5_K_XL) loaded on the same card, an estimated 1.8 GiB stays free at 262K context and only about 6.4 GiB even at 16K, so stop the model server before gaming.", "details": "Weights take 24.25 GiB and the f16 KV cache 5.00 GiB at 262,144 tokens (linear in context); 1 GiB for compute buffers and driver reserve is an estimate, not a measurement. AMD lists the R9700 as compatible with Adrenalin 26.10.1 and offers both Adrenalin and the PRO Edition driver for it. Back2Gaming notes the blower cooler is louder than most open-air 9070 XT models and that the card shouldn't be bought for gaming first. On Newegg on Nov 2, 2026, the lowest new RX 9070 XT was $919.99, while the only R9700 listing seen was a marketplace offer at $2,779.97.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "UD-Q5_K_XL", "engine": "llama.cpp (any backend)", "gpus": "1x Radeon AI PRO R9700 (32 GB)", "context": "16K-262K", "flags": "f16 KV" }, "metrics": [ { "label": "RX 9070 XT lead over R9700, 4-resolution game index (PCGH)", "value": 3, "unit": "%" }, { "label": "Cyberpunk 2077 1440p max, R9700", "value": 127.3, "unit": "FPS" }, { "label": "Cyberpunk 2077 1440p max, RX 9070 XT", "value": 128.3, "unit": "FPS" }, { "label": "Free VRAM with the 35B at 262K (estimate)", "value": 1.75, "unit": "GiB" }, { "label": "Free VRAM with the 35B at 16K (estimate)", "value": 6.44, "unit": "GiB" }, { "label": "Lowest new RX 9070 XT on Newegg, 2026-11-02", "value": 919.99, "unit": "USD" } ], "sources": [ "https://www.back2gaming.com/review/amd-radeon-ai-pro-r9700-32gb/", "https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/2/", "https://www.amd.com/en/resources/support-articles/release-notes/RN-RAD-WIN-26-10-1.html", "https://www.newegg.com/p/pl?d=rx+9070+xt&N=100007709&Order=1" ], "next": "Buy the RX 9070 XT if gaming comes first; buy the R9700 if you want the 35B fully on one card at 256K context and also game on it." }, { "id": "2026-11-02-vulkan-splitk-rdna4-35b", "date": "2026-11-02T03:10:00-06:00", "project": "idea-lab", "kind": "result", "title": "New RDNA3/4 Vulkan split-K patches won't speed up Qwen3.6-35B-A3B on one R9700", "summary": "Two open llama.cpp Vulkan PRs retune split-K for RDNA3/4: #30280 (up to +21-25% on some q4 matmuls on a 64-CU 9070 XT, +5-10% pp512 on 4B-12B models) and #30146 (+12-20% prefill at ubatch 13-32 and +13-14% MTP decode at draft max 8/16 on RDNA3). Replaying the split-K heuristic for every dense matmul in Qwen3.6-35B-A3B on a 64-CU R9700 shows #30280 changes none of them at -ub 512, 1024 or 2048, because the model's 2048/4096/8192-wide projections fill whole waves of workgroups. Keep your current build for this model.", "details": "Dense matmuls are about 59% of the 35B's active matmul weights per token; the routed experts (MUL_MAT_ID) are untouched by both PRs. #30280 only affects a prompt's last partial batch of 513-640 tokens (about 23% of that batch's dense work, once per prompt). #30146 changes about half of the dense work only at 9-32 tokens per batch, which matters for speculative decoding with large drafts, not for plain decode or normal prefill. Models with roughly 2,200-2,700-wide projections (the 4B class, hidden size 2560) are the ones that gain at -ub 512. This is a replay of the published heuristic, not a benchmark; the model reproduces the PR author's which-shapes-gain table.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "any (dense projection shapes)", "engine": "llama.cpp Vulkan (RADV, coopmat)", "engine_version": "master 2026-11-02 + PR #30280 / #30146 diffs", "gpus": "1x Radeon AI PRO R9700 (64 CUs)", "context": "n/a", "flags": "-ub 512 / 1024 / 2048" }, "metrics": [ { "label": "35B dense split decisions changed by #30280 at -ub 512/1024/2048", "value": 0, "unit": "%" }, { "label": "35B dense share of active matmul weights", "value": 59, "unit": "%" }, { "label": "Dense work changed by #30146 at 9-32 tokens/batch", "value": 50, "unit": "%" }, { "label": "Author best case, q4_K 2560x10240 on 9070 XT (#30280)", "value": 24.9, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/30280", "https://github.com/ggml-org/llama.cpp/pull/30146", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-vulkan/ggml-vulkan.cpp", "https://huggingface.co/Qwen/Qwen3.6-35B-A3B" ], "next": "Optional A/B with llama-bench on one R9700 to confirm; revisit #30146 if a 35B speculative-decoding setup with draft max 8+ becomes common." }, { "id": "2026-10-31-hip-q8-kv-tile-fa", "date": "2026-10-31T23:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "On ROCm, a q8_0 KV cache costs the R9700 ~40% decode at 128K for Qwen3.6-35B; an open llama.cpp PR flips it to slightly faster than f16", "summary": "With llama.cpp's ROCm/HIP backend today, quantizing the KV cache to q8_0 to save VRAM makes Qwen3.6-35B-A3B decode slower the deeper the context goes: on the PR author's R9700 it falls from about 61 tok/s with f16 KV to 37 tok/s at 128K. An open pull request moves quantized KV into the tile flash-attention kernel, and the same card then decodes 68 tok/s at 128K with q8_0. Until it merges, ROCm users should keep f16 KV for long context; the Vulkan backend already reads q8_0 blocks directly.", "details": "On master, HIP decode with quantized KV either uses the vector kernel, which re-reads the cache once per query head, or converts the whole cache to f16 every step. PR #31234 adds quantized K/V support to the tile kernel (head sizes 64/128/256, the decode shapes the 35B uses with 16 query and 2 KV heads). Author's R9700 numbers for Qwen3.6-35B-A3B Q4_K_M, q8_0 KV, tg128, master vs PR: 78.9 vs 83.5 at 8K, 64.4 vs 80.7 at 32K, 51.4 vs 75.3 at 64K, 36.8 vs 67.9 at 128K tok/s. The f16 baseline (61.2 tok/s at 128K, 70.3 at 64K) comes from the same author's earlier, closed PR #30803 on the same card at tg64, so the q8_0-vs-f16 comparison is approximate. Prefill (pp512 at 32K) and f16 KV are unchanged, and the author reports all 4117 flash-attention backend tests passing on the R9700. At the model's full 262,144-token context, q8_0 KV is 2.66 GiB instead of 5.00 GiB (from the GGUF header: 10 attention layers, 2 KV heads, head size 256), the headroom that lets a Q6_K 35B hold full context on one 32 GB card. Gemma 4's 512-wide global heads are not covered yet. The PR is open and not yet reviewed. We have not measured it ourselves yet.", "setup": { "engine": "llama.cpp (ROCm/HIP backend)", "engine_version": "open PR #31234 vs master (author's benchmark)", "gpus": "Radeon AI PRO R9700 (RDNA4), one card, 300 W" }, "metrics": [ { "label": "35B-A3B Q4_K_M decode at 128K, q8_0 KV, master vs PR", "value": "36.8 / 67.9", "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode at 128K, f16 KV (earlier PR, tg64)", "value": "61.2", "unit": "tok/s" }, { "label": "35B-A3B KV cache at 262K, f16 vs q8_0 (header-derived)", "value": "5.00 / 2.66", "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/31234", "https://github.com/ggml-org/llama.cpp/pull/30803" ], "next": "Measure HIP master vs the PR vs Vulkan with f16 and q8_0 KV at 32K, 128K and 250K on one R9700, then test a Q6_K 35B with q8_0 KV at full context." }, { "id": "2026-10-31-vulkan-batched-slot-save", "date": "2026-10-31T21:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "A Vulkan patch cuts llama-server's idle-slot save on an R9700 from ~300 ms to ~25-43 ms for Qwen3.6-35B", "summary": "By default llama-server copies each idle slot's cache to system RAM when a new request arrives, and other streams pause while it does. Earlier this week llama.cpp added a batched copy that fixes this on CUDA and Metal only. An open Vulkan pull request now implements it too, and its author measured an R9700 running Qwen3.6-35B-A3B Q4_K_M: the save dropped from 286-332 ms to 25-43 ms. The ROCm/HIP build still lacks the faster gather path.", "details": "PR #31418 adds Vulkan set/get/cpy_tensor_n for the batched tensor copy API from PR #30881 (merged Oct 29). The author's table covers multi-slot unified-KV prompt saves: R9700 Llama-3.1-8B Q6_K (64 MiB per slot) 871-950 ms to 24-48 ms, R9700 Qwen3.6-35B-A3B Q4_K_M (73 MiB per slot) 286-332 ms to 25-43 ms, with similar gains on a Radeon 8060S and an NVIDIA GB10. The PR is still in review. From the model's config, a Qwen3.6-35B-A3B slot holds about 63 MiB of fixed DeltaNet state plus 20 KiB per token of f16 KV cache, so 73 MiB is a slot with only ~500 cached tokens (our estimate). Coding-agent slots are much bigger: about 220 MiB at 8K tokens and 700 MiB at 32K, and the default 8 GiB host prompt cache holds about 11 slots at 32K. Save times at those sizes have not been published for either build yet. The CUDA gather follow-up (PR #31391) is disabled on HIP builds. Single-user setups are barely affected because empty slots are skipped; the stall hits multi-agent and router setups.", "setup": { "engine": "llama.cpp (Vulkan backend)", "engine_version": "open PR #31418 (head 8a20245)", "gpus": "Radeon AI PRO R9700 (RDNA4), author's benchmark plus source review" }, "metrics": [ { "label": "35B-A3B Q4_K_M slot save on R9700, master vs PR", "value": "286-332 / 25-43", "unit": "ms" }, { "label": "Llama-3.1-8B Q6_K slot save on R9700, master vs PR", "value": "871-950 / 24-48", "unit": "ms" }, { "label": "Estimated 35B-A3B slot size at 32K tokens (f16 KV)", "value": "~700", "unit": "MiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/31418", "https://github.com/ggml-org/llama.cpp/pull/30881", "https://github.com/ggml-org/llama.cpp/pull/31391", "https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json" ], "next": "Measure save time and worst stream pause at 8K and 32K agent contexts on one R9700, master vs the PR, once it is close to merging." }, { "id": "2026-10-31-gdn-chunked-prefill-rdna4", "date": "2026-10-31T19:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama.cpp's new chunked DeltaNet prefill skips the R9700 on ROCm; a Vulkan port needs Mesa 26.3", "summary": "Qwen3.6-35B-A3B and Qwen 3.8 Flash-Next are hybrid models with Gated DeltaNet layers. llama.cpp merged a chunked tensor-core prefill kernel for them on Oct 14 (+11-21% prompt speed on RTX 4090/5090/PRO 6000 for 35B-A3B), but on HIP it is enabled only for RDNA3.5, and an AMD HIP maintainer found it slower than the existing kernel on RDNA4 because of register spills. A draft Vulkan port has an RDNA4-specific path that runs the DeltaNet op about 1.7-1.8x faster on an R9700 for the 35B shape, but its author says the gain doesn't show at the model level yet, and it needs a RADV extension that first ships in Mesa 26.3.", "details": "In llama.cpp master (a91eab6, Oct 31), the HIP build picks the chunked kernel only on RDNA3.5 GPUs and only for batches of 2048+ tokens; RDNA4 always uses the scan kernel. PR #31327 enables it on CDNA (MI100 +19-25%) and reports RDNA4 and RDNA3 were slower than the scan kernel. Draft PR #30207 (Vulkan) adds a wave64 cooperative-matrix path for RDNA4; the author's R9700 op chart shows 96-102 GB/s (scan) vs 173-177 GB/s (chunked) for the 35B shape across 2K-32K token batches, and about 1.25x for a 27B shape. Vulkan's existing scan kernel is already about twice as fast as CUDA's, so there is less to gain than on NVIDIA. The path needs VK_EXT_cooperative_matrix_maintenance1, which RADV gained in Mesa main in August; it is in 26.3.0-rc3 and not in 26.2.6. The chunked path is also skipped whenever MTP or draft speculation is enabled, so for now it doesn't stack with MTP. Decode speed is unaffected; this is a prefill-only change. We have not measured the model-level effect on an R9700 yet.", "setup": { "engine": "llama.cpp (HIP and Vulkan backends)", "engine_version": "master a91eab6; draft PR #30207", "gpus": "Radeon AI PRO R9700 (RDNA4), source review" }, "metrics": [ { "label": "35B-A3B prompt speed gain from chunked kernel, RTX 4090 / 5090", "value": "11 / 16-18", "unit": "%" }, { "label": "DeltaNet op throughput on R9700 Vulkan, scan vs chunked (35B shape)", "value": "96-102 / 173-177", "unit": "GB/s" }, { "label": "First Mesa release with the needed RADV extension", "value": "26.3", "unit": "" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/29353", "https://github.com/ggml-org/llama.cpp/pull/31327", "https://github.com/ggml-org/llama.cpp/pull/30207", "https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43730" ], "next": "Measure prompt speed for the 35B at -ub 512 and 2048 on one R9700 with Mesa 26.3, master vs the Vulkan draft PR, plus a KL-divergence check." }, { "id": "2026-10-31-idle-slot-save-stall", "date": "2026-10-31T17:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Default llama-server pauses every stream to save idle slots, and the new fix is NVIDIA-only", "summary": "Since April 2026, llama-server with no -np runs 4 slots with a unified KV cache and saves every idle slot to a RAM prompt cache whenever a new request starts, which pauses other streams (llama.cpp issue #30252, still open). A batched copy API (#30881, merged Oct 29) and a gather kernel (#31391, open) cut the save from about 140 ms to 7 ms on an RTX PRO 6000, but the Vulkan backend has no batched hook and the gather kernel is disabled on HIP, so Radeon cards keep the slow path.", "details": "Reported numbers: a ~300-token save took 0.7-0.9 s and new requests waited up to 2.1 s on an RTX 3090 (issue #30252). With #31391 on an RTX PRO 6000 (Qwen3-30B-A3B, 8 slots, unified KV, 8 streams) the worst stream freeze fell from 700 to 200 ms, time to first token from 285 to 170 ms, and throughput rose 19%; on the hybrid Qwen3.8-Flash-Next the save time halved but streams barely changed. A single user with one conversation is mostly unaffected because the other slots are empty and are skipped. For one-user Radeon servers, passing -np 1 avoids the multi-slot path; --no-cache-idle-slots turns the save off. We have not measured the stall on an R9700 yet.", "setup": { "engine": "llama.cpp llama-server", "engine_version": "master ddf48f8 (b12024)", "gpus": "Radeon (Vulkan and HIP backends), source review" }, "metrics": [ { "label": "Idle-slot save, before gather kernel (RTX PRO 6000)", "value": 140, "unit": "ms" }, { "label": "Idle-slot save, with gather kernel (RTX PRO 6000)", "value": 7, "unit": "ms" }, { "label": "Worst stream freeze, before / after", "value": "700 / 200", "unit": "ms" }, { "label": "Throughput change with gather kernel", "value": 19, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/issues/30252", "https://github.com/ggml-org/llama.cpp/pull/20993", "https://github.com/ggml-org/llama.cpp/pull/30881", "https://github.com/ggml-org/llama.cpp/pull/31391" ], "next": "Measure the stall on one R9700 with the Vulkan 35B at 1 and 3 concurrent agents, comparing the default against -np 1 and --no-cache-idle-slots." }, { "id": "2026-10-31-vllm-segmented-attn-opt-in", "date": "2026-10-31T15:30:00-05:00", "project": "dual-r9700", "kind": "result", "title": "vLLM 0.32 on two R9700s: the fast RDNA attention backend is opt-in, so add --attention-backend ROCM_SEGMENTED_ATTN (it doesn't cover Qwen 3.8 Flash-Next)", "summary": "vLLM 0.32.0 ships ROCM_SEGMENTED_ATTN, a Triton attention backend for RDNA3/3.5/4. On two R9700s with tensor parallelism 2 (Qwen3.8-27B-FP8, FP8 KV cache, multi-turn agent workload), its author measured per-request output speed up 56% at 1 stream and 176-181% at 4-8 streams over the default ROCM_ATTN, with end-to-end latency down 39-70%. It is not picked automatically: in 0.32.0 the ROCm auto order starts with ROCM_ATTN and the new backend isn't in the list, so you must pass the flag. Qwen 3.8 Flash-Next doesn't benefit, because its sparse-attention layers on AMD use their own fixed Triton backend and require a BF16 KV cache.", "details": "With a DFlash2 draft model (7 speculative tokens, 4 concurrent requests) on the same two R9700s, aggregate output went from 102 to 254 tok/s on GSM8K and from 4.1 to 132 tok/s with 32K-token inputs; draft acceptance at 16K-32K recovered from about 1.03 to about 3.9 tokens per round, which suggests the old backend was breaking speculative decoding at long context (the author notes this doesn't isolate the cause). Optional autotuning added another 2-6% but about 8 minutes to the first startup. Limits: FP8 KV works only on gfx12 (RDNA4); head sizes 64/128/256 and GQA ratios 1-16. These are the PR author's single-run numbers on a dense 8-bit 27B model; compare the ratios, not the absolute per-request rates. Separately, 0.32.0's 1.45-1.68x Flash-Next projection-merge speedup comes with tuned GEMM plans for NVIDIA only; no AMD gain has been measured.", "setup": { "model": "Qwen3.8-27B-FP8 (PR author's test)", "quant": "FP8 weights, FP8 KV", "engine": "vLLM", "engine_version": "0.32.0 (PR #59132)", "gpus": "2x Radeon AI PRO R9700, TP2", "context": "2K prompt + 512 output per turn, 8 turns; 16K/32K inputs for the drafter test", "flags": "--attention-backend ROCM_SEGMENTED_ATTN --tensor-parallel-size 2 --kv-cache-dtype fp8" }, "metrics": [ { "label": "Per-request output speed, 1 stream (ROCM_ATTN to segmented)", "value": "+56", "unit": "%" }, { "label": "Per-request output speed, 8 streams", "value": "+181", "unit": "%" }, { "label": "DFlash2 aggregate output, 32K input", "value": "4.08 to 131.54", "unit": "tok/s" }, { "label": "Extra first-start time with autotune", "value": "8.3", "unit": "min" } ], "sources": [ "https://github.com/vllm-project/vllm/pull/59132", "https://github.com/vllm-project/vllm/releases/tag/v0.32.0", "https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/platforms/rocm.py", "https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/models/qwen4_exp/amd/qsa.py", "https://github.com/vllm-project/vllm/pull/59533" ], "next": "Measure ROCM_ATTN vs ROCM_SEGMENTED_ATTN on a 4-bit Qwen3.6-35B-A3B at TP2 and compare against one llama.cpp copy per card." }, { "id": "2026-10-31-avx512-not-a-decode-lever", "date": "2026-10-31T12:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "An AVX-512 / Zen 5 llama.cpp build won't speed up DDR5-offload decode: the K-quant decode kernels are AVX2-only and already outrun DDR5", "summary": "In llama.cpp master (0ec4027, 2026-10-31) the x86 decode kernels for Q4_K, Q5_K and Q6_K have only AVX2/AVX code paths, and the Q4_K repack GEMV is AVX2-only; AVX-512 appears only in the batched (prefill) repack GEMMs. Running those AVX2 kernels in a harness, one core processes about 18 GB/s of Q4_K and 26 GB/s of Q6_K weights from cache. Sixteen such cores would chew roughly 290-420 GB/s, 5-8x more than dual-channel DDR5 delivers (about 55 GB/s), so the CPU side of expert-offload decode is memory-bound and rebuilding for AVX-512 is not a decode speed lever.", "details": "Measured on an 8-vCPU AVX-512 Xeon cloud VM with the kernels copied verbatim from llama.cpp; the Ryzen 9950X figures are an estimate that assumes a Zen 5 core runs this AVX2 code at least as fast. In-cache throughput scaled linearly to 8 threads (155 GB/s Q4_K, 205 GB/s Q6_K). Single-thread streaming from DRAM reached only 4.7-9.9 GB/s, so what matters is enough threads to keep DDR5 busy, not wider vectors. Q6_K moves more bytes per second than Q4_K because its scale unpacking is cheaper per byte, but per weight Q4_K is still faster. Pitfall for anyone writing kernel benchmarks: random fp16 scales (NaN/denormal values) made the same kernels 4-8x slower. AVX-512 still helps CPU prompt processing (ik_llama.cpp targets that), which matters only if prefill runs on the CPU.", "setup": { "model": "synthetic Q4_K / Q6_K rows (2048 columns)", "quant": "Q4_K, Q6_K", "engine": "llama.cpp x86 AVX2 kernels (master 0ec4027) in a standalone harness", "engine_version": "0ec4027", "gpus": "none (CPU kernel test)", "context": "n/a", "flags": "gcc -O3 -mavx2 -mfma -mf16c, OpenMP 1-8 threads" }, "metrics": [ { "label": "Q4_K in-cache throughput, 1 core", "value": 17.99, "unit": "GB/s" }, { "label": "Q6_K in-cache throughput, 1 core", "value": 26.28, "unit": "GB/s" }, { "label": "Q6_K in-cache throughput, 8 threads", "value": 205.01, "unit": "GB/s" }, { "label": "Est. 16-core kernel headroom over DDR5 (Q4_K to Q6_K)", "value": "5-8", "unit": "x (estimate)" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cpu/arch/x86/quants.c", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cpu/arch/x86/repack.cpp", "https://github.com/ggml-org/llama.cpp/commit/94de74e7b1b9ee6404b54ebb0df273a3ef35a555", "https://github.com/ikawrakow/ik_llama.cpp/pull/710" ], "next": "Confirm on the 9950X with test-backend-ops perf on the CPU backend and an AVX2-only vs native A/B on Flash UD-Q5 decode." }, { "id": "2026-10-31-local-llm-build-cost", "date": "2026-10-31T10:25:00-05:00", "project": "idea-lab", "kind": "result", "title": "What a local-LLM PC on AMD Radeon costs today: ~$3,890 for one R9700, ~$6,440 for two, and DDR5 is now a big slice", "summary": "Store prices read Oct 31, 2026 (before tax): one Radeon AI PRO R9700 build that holds Qwen3.6-35B at 256K context totals $3,891.61; a two-R9700 + 96 GB build for Qwen 3.8 Flash Q4 totals $6,444.61. RAM is 31% of the first build's non-GPU parts (32 GB DDR5-6000 CL30 at $659.99) and 50% of the second's (96 GB DDR5-5600 at $1,479.99).", "details": "One-card build: R9700 from $1,750 (cheapest new card sold by a store itself), ASRock X870 Taichi Creator $319.99, Ryzen 7 9700X $339.00, G.SKILL Flare X5 2x16 GB DDR5-6000 CL30 $659.99, Corsair RM850x $146.99, Samsung 990 PRO 2 TB $452.75, Thermalright Peerless Assassin 120 SE $47.90, Fractal Meshify 2 $174.99 (+$9.99 shipping). Two-card build swaps in a second R9700, G.SKILL Flare X5 2x48 GB DDR5-5600 CL40 and a Thermaltake Toughpower GT 1200W ($129.99). Newegg had no R9700 of its own in stock (Sapphire sold out, ASRock Creator out of stock, Gigabyte AI TOP via marketplace at $2,799.99). Non-GPU prices are Newegg search-listing prices; seller not verified for every item. Prices change daily.", "setup": { "gpus": "1x or 2x Radeon AI PRO R9700", "context": "256K", "model": "Qwen3.6-35B-A3B (one card); Qwen 3.8 Flash Q4 (two cards)" }, "metrics": [ { "label": "One-R9700 build total", "value": 3891.61, "unit": "USD" }, { "label": "Two-R9700 + 96 GB build total", "value": 6444.61, "unit": "USD" }, { "label": "32 GB DDR5-6000 CL30 kit", "value": 659.99, "unit": "USD" }, { "label": "96 GB DDR5-5600 kit", "value": 1479.99, "unit": "USD" } ], "sources": [ "https://vga.live/gpus/radeon-ai-pro-r9700", "https://www.newegg.com/p/pl?d=AMD+Radeon+AI+PRO+R9700&N=100007709", "https://www.newegg.com/p/pl?d=F5-6000J3038F16GX2-FX5", "https://www.newegg.com/p/pl?d=F5-5600J4040D48GX2-FX5&N=8000", "https://www.newegg.com/p/pl?d=ASRock+X870+Taichi+Creator", "https://www.newegg.com/p/pl?d=AMD+Ryzen+7+9700X", "https://www.newegg.com/p/pl?d=CP-9020270-NA", "https://www.newegg.com/p/pl?d=PS-TPT-1200FNFAGU-3", "https://www.newegg.com/p/pl?d=MZ-V9P2T0BW", "https://www.newegg.com/p/pl?d=Thermalright+Peerless+Assassin+120+SE", "https://www.newegg.com/p/pl?d=Fractal+Design+Meshify+2" ], "next": "Turn this into a dated build page with one store button per part, refreshed with a seller check before it ships." }, { "id": "2026-10-31-cheap-third-card-expert-shelf", "date": "2026-10-31T09:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "A cheap 16 GB third card for Flash on two R9700s: ~16-21% faster at Q4 (estimate), and a faster 16 GB card barely helps", "summary": "Estimate from GGUF tensor sizes and measured DDR5 bandwidth: a 16 GB RDNA4 card holding whole expert layers takes 9 (Q4), 7 (Q5) or 6 (Q6) layers out of system RAM, raising the decode ceiling ~16-21% at Q4, ~12-15% at Q5 and ~10-12% at Q6. A 640 GB/s RX 9070 XT is only ~2 tok/s faster than a 320 GB/s RX 9060 XT here, because the layers left in DDR5 still dominate. A third R9700 costs about the same per added tok/s at Q4 but takes Q4 fully off system RAM (+48-62%).", "details": "Model: Qwen3.8-Flash-Next UD packs at 262,144 context, f16 KV, llama.cpp-style layer split, GPU decode at 75% of spec bandwidth. Prices are the lowest new prices on 2026-10-31: R9700 $1,750, RX 9060 XT 16GB $500, RX 9070 $750, RX 9070 XT $880. On Q4 that is about $53-62 per added tok/s of ceiling for the 9060 XT, $78-88 for the 9070 XT and $63-74 for a third R9700; at Q5 and Q6 the third R9700 is the better value. Practical notes: never make the extra card device 0 (big prefill copies go through device 0's link), put whole layers on it rather than only the experts of layers whose attention lives elsewhere, add gfx1200 to the HIP build targets for a 9060 XT (the 9070 cards are gfx1201 like the R9700), and expect the third slot on most x8/x8 boards to be a chipset x4 link. These are ceilings, not measurements; real llama.cpp speed has run at 55-66% of this ceiling.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp layer split (modeled)", "gpus": "2x Radeon AI PRO R9700 + optional 16 GB RX 9060 XT / 9070 / 9070 XT or a third R9700", "context": "262144", "flags": "f16 KV, experts not on a GPU in DDR5, no MTP" }, "metrics": [ { "label": "Q4 ceiling, 2x R9700 + 2-stick DDR5-5600 (estimate)", "value": 49.4, "unit": "tok/s" }, { "label": "Q4 ceiling, + RX 9060 XT 16GB shelf (estimate)", "value": 57.4, "unit": "tok/s" }, { "label": "Q4 ceiling, + RX 9070 XT 16GB shelf (estimate)", "value": 59.4, "unit": "tok/s" }, { "label": "Q4 ceiling, + third R9700 (estimate)", "value": 73.0, "unit": "tok/s" }, { "label": "Q5 gain from a 16 GB shelf (estimate)", "value": "12-15", "unit": "%" }, { "label": "RX 9060 XT 16GB lowest new price, 2026-10-31", "value": 500, "unit": "USD" } ], "sources": [ "https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9060xt.html", "https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html", "https://github.com/ggml-org/llama.cpp/issues/23855", "https://github.com/ggml-org/llama.cpp/discussions/22183", "https://www.reddit.com/r/LocalLLaMA/comments/1t2l8ri/3xr9700_for_semiautonomous_research_and/", "https://vga.live/gpus/radeon-rx-9060-xt-16gb" ], "next": "Measure the cost of holding expert layers on a second GPU with hardware already in hand (a 35B model with 10 expert layers on the second card vs in DDR5) before recommending a third card." }, { "id": "2026-10-31-ncpumoe-two-cards-placement", "date": "2026-10-31T02:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Two R9700s: --n-cpu-moe frees VRAM only on the first card; per-card placement est. +25-41% Flash decode", "summary": "In llama.cpp, --n-cpu-moe N moves the experts of layers 0..N-1 to system RAM. With the default layer split those layers all live on GPU 0, so the second card can't shed anything until N passes half the layers. Our estimate for Qwen3.8-Flash-Next at 262K on two R9700s: --n-cpu-moe alone needs 33/36/37 expert layers in DDR5 at UD-Q4/Q5/Q6 and leaves ~25 GiB of VRAM idle, while per-card placement needs 17/24/26 layers, an estimated +41%/+28%/+25% decode. llama.cpp's --fit (on by default) already places per card, but it silently does nothing if you pass -ngl, -ts, -ot or --n-cpu-moe.", "details": "Checked in llama.cpp master source (Oct 29, 2026): common/arg.cpp defines --n-cpu-moe as the first N layers; common/fit.cpp gives each device its own run of CPU-expert layers and aborts when -ngl, -ts or tensor overrides are set by the user. A public A-B-A sweep on two R9700s saw the same thing in VRAM (all freed memory came off GPU 0) and saw SMT threads (-t above physical cores) more than halve generation; its model was a Q3 quant, so we don't reuse its speeds. Separately, op offload uploads only the experts the router picked for each micro-batch (llama_context::sched_copy_experts), and llama-bench's prompt test uses random tokens, which likely explains why its prompt numbers imply more PCIe bandwidth than a Gen5 x8 link has. Speed numbers here are model estimates from measured GGUF header sizes (GPU at 75% of 640 GB/s, 55 GB/s DDR5 read, 2.5 GiB reserve per card, 60 us per CPU/GPU hand-off), not measurements.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (source review + estimate)", "engine_version": "master, Oct 29 2026", "gpus": "2x Radeon AI PRO R9700", "context": "262144", "flags": "-sm layer, f16 KV; A: --n-cpu-moe only, B: --fit per-card placement" }, "metrics": [ { "label": "Q5 expert layers in DDR5, --n-cpu-moe alone (est.)", "value": 36, "unit": "layers" }, { "label": "Q5 expert layers in DDR5, per-card (est.)", "value": 24, "unit": "layers" }, { "label": "Q5 decode, --n-cpu-moe alone (est.)", "value": 23.8, "unit": "tok/s" }, { "label": "Q5 decode, per-card (est.)", "value": 30.5, "unit": "tok/s" }, { "label": "Idle VRAM with --n-cpu-moe alone, Q5 (est.)", "value": 24.9, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/common/fit.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp", "https://github.com/getsamsultan/rdna4-local-ai-lab/blob/main/data/2026-10-31-qwen38-system-ram-sweep.md", "https://github.com/ggml-org/llama.cpp/discussions/31450" ], "next": "Measure --fit placement against --n-cpu-moe on two R9700s with real-text 8K prompts at Q4, Q5 and Q6, and keep -t at or below physical cores." }, { "id": "2026-10-31-x86-q6k-repack-flash-offload", "date": "2026-10-31T00:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp's pending x86 Q6_K repack won't speed up Q6 Flash models with experts in DDR5", "summary": "Open llama.cpp PR #19706 adds AVX2/AVX-512 repacking for Q6_K. Its own Ryzen 5 9600X numbers show prompt processing +17% (AVX2) to +54% (AVX-512) but decode about 3% slower. With two R9700s and experts in system RAM, llama.cpp sends any batch of 32+ tokens to the GPU anyway, so the CPU only handles decode, where this PR doesn't help.", "details": "Checked at llama.cpp master 1052f7e (2026-10-31): x86 repacking still covers only Q4_0, Q4_K, IQ4_NL and MXFP4; Q5_K and Q6_K repack is ARM-only. In the PR (rebased 2026-10-29) the batch-1 decode kernel (GEMV) has only an AVX2 path, so AVX-512 doesn't reach decode. Repacked weights also can't sit in the GPU's pinned host buffer, so using them means --no-host, which upstream measured at 0.64x prompt speed. An open July report shows an assert with a DeepSeek-V4-Flash GGUF. Estimate for a UD-Q6 Flash pack with experts in DDR5: 0 to -3% decode, no prefill gain. Also re-checked six other pending PRs that matter for dual R9700 (Vulkan tensor-parallel allreduce #25051, host-RAM expert pipeline parallelism #29963, RDNA4 HIP matmul #25940, hybrid checkpoint fixes #25592/#28118, slot save/restore #29738): none merged yet.", "setup": { "model": "Qwen3.8-Flash-Next (target)", "quant": "UD-Q6_K_XL", "engine": "llama.cpp", "engine_version": "master 1052f7e + PR #19706 head e744a1e", "gpus": "2x Radeon AI PRO R9700 (target)", "context": "n/a", "flags": "source review, no run" }, "metrics": [ { "label": "PR #19706 pp512 gain, AVX-512 (9600X, 7B Q6_K)", "value": 54, "unit": "%" }, { "label": "PR #19706 tg128 change (9600X, 7B Q6_K)", "value": -2.9, "unit": "%" }, { "label": "Batch size llama.cpp sends to GPU", "value": 32, "unit": "tokens" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/19706", "https://github.com/ggml-org/llama.cpp/blob/1052f7e1d62611d328ccc65d38c77cd194ee59fc/ggml/src/ggml-cpu/repack.cpp", "https://github.com/ggml-org/llama.cpp/pull/25051", "https://github.com/ggml-org/llama.cpp/pull/29963" ], "next": "Re-check weekly; test only if a Q6_K AVX-512 decode kernel with positive decode numbers appears." }, { "id": "2026-10-30-dwarfstar-rocm-dual-r9700-status", "date": "2026-10-30T22:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "DwarfStar (ds4) can't split Flash models across two R9700s yet", "summary": "DwarfStar's expert tensor parallelism is CUDA only. Its ROCm backend accepts one GPU per process, has no Qwen3.8 Flash Next graph, and doesn't build for the R9700 (gfx1201) upstream; three fix PRs are still open. For two R9700s at Q4 or above, llama.cpp and Strata remain the options.", "details": "Read at antirez/ds4 main c97964b (2026-10-30). ds4_gpu_init_multi rejects more than one GPU on ROCm, and the ROCm tensor-parallel stubs print 'tensor parallelism is Metal-only'. ROCm tensor parallel exists only as an open PR chain for GLM 5.3 across two networked Strix Halo machines (#1257). The Makefile targets gfx1151 by default; the R9700 build fails on an RDNA3-only WMMA builtin (issue #521), with fixes #558, #599 and #744 unmerged. DwarfStar also doesn't compute experts on the CPU: past VRAM it streams experts from page cache or SSD. For reference, DwarfStar's own Qwen3.8 Flash Next Q4 on one DGX Spark decodes 21.2 tok/s at 8K depth (prefill 513-755 tok/s). One idea worth borrowing: it keeps Flash Next's n-gram embedding table on SSD and reads rows on demand, so its Q4 pack needs 69.74 GiB resident.", "setup": { "model": "Qwen3.8-Flash-Next, DeepSeek V4 Flash, GLM 5.3 Flash", "quant": "Q4 and above only", "engine": "DwarfStar (antirez/ds4)", "engine_version": "main c97964b", "gpus": "2x Radeon AI PRO R9700 (target)", "context": "n/a", "flags": "source review, no run" }, "metrics": [ { "label": "GPUs per process on ROCm", "value": 1, "unit": "" }, { "label": "Qwen3.8 Flash Next Q4 decode, DGX Spark, 8K depth (DwarfStar docs)", "value": 21.2, "unit": "tok/s" }, { "label": "Qwen3.8 Flash Next Q4 resident weights with n-grams on SSD", "value": 69.74, "unit": "GiB" } ], "sources": [ "https://github.com/antirez/ds4", "https://github.com/antirez/ds4/blob/c97964b8aa77034a8e17afb10ca7a686c40ab511/docs/QWEN38_FLASH_NEXT.md", "https://github.com/antirez/ds4/blob/c97964b8aa77034a8e17afb10ca7a686c40ab511/docs/CUDA_MULTI_GPU.md", "https://github.com/antirez/ds4/issues/521", "https://github.com/antirez/ds4/pull/599", "https://github.com/antirez/ds4/pull/1257" ], "next": "Re-check when a gfx1201 build fix merges; separately test whether llama.cpp with mmap keeps Flash Next's n-gram table out of RAM." }, { "id": "2026-10-30-ollama-default-context-vram-tier", "date": "2026-10-30T19:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Ollama picks your context from total VRAM: one 32 GB R9700 starts at 32K, not 256K", "summary": "In Ollama v0.40.13 the default context comes from total VRAM summed across all GPUs: under 23 GiB gets 4K, 23 to 47 GiB gets 32K, and 47 GiB or more gets 256K. Ollama's FAQ still says 4096. One R9700 holds Qwen3.6-35B Q4_K_M at 256K, so set OLLAMA_CONTEXT_LENGTH=262144 instead of buying a second card.", "details": "The rule is in server/routes.go (thresholds 47/23 GiB to allow for reported-size differences); when the automatic value doesn't fit, the scheduler steps down 256K to 32K to 4K. So a 16 GB RX 9070 XT or 9060 XT starts at 4K, a 24 GB RX 7900 XTX or a 32 GB R9700/W7800 at 32K, and a 48 GB W7900 or two R9700s at 256K. Ollama's library carries the 35B only as q4_K_M (23-24 GB), q8_0 (39 GB) and bf16, so Q5/Q6 need a Hugging Face pull. Fit at 256K is an estimate: about 25.6 GB for Q4_K_M with f16 KV plus roughly 1 GB for the vision projector. Check with ollama ps that PROCESSOR shows 100% GPU and CONTEXT shows 262144. Not measured inside Ollama.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M (Ollama library tag)", "engine": "Ollama (source read)", "engine_version": "v0.40.13", "gpus": "1x or 2x Radeon AI PRO R9700", "context": "4K / 32K / 256K by VRAM tier", "flags": "OLLAMA_CONTEXT_LENGTH=262144" }, "metrics": [ { "label": "Default context, 1x R9700 (32 GB)", "value": 32768, "unit": "tokens" }, { "label": "Default context, 2x R9700 or 48 GB card", "value": 262144, "unit": "tokens" }, { "label": "Default context, 16 GB card", "value": 4096, "unit": "tokens" }, { "label": "Est. 35B Q4_K_M + projector at 256K", "value": 26.6, "unit": "GiB" } ], "sources": [ "https://github.com/ollama/ollama/blob/v0.40.13/server/routes.go", "https://github.com/ollama/ollama/blob/v0.40.13/server/sched.go", "https://github.com/ollama/ollama/blob/main/docs/context-length.mdx", "https://github.com/ollama/ollama/blob/main/docs/faq.mdx", "https://ollama.com/library/qwen3.6/tags" ], "next": "Correct the Ollama setup page and add a per-card default-context table." }, { "id": "2026-10-30-35b-ubatch-dead-band", "date": "2026-10-30T18:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.6-35B on one R9700: pick -ub 512 or 2048, never 65-256", "summary": "Published RDNA4 Vulkan results show two things about llama.cpp's ubatch (-ub) for Qwen3.5/3.6 hybrid models. Values from 65 to 256 can make them 10-40x slower (Qwen3.6-27B: prompt 774 to ~17 tok/s, decode 25.4 to 2.1), and a plain llama-bench tg128 lands in that band by itself. On one R9700, -ub 2048 raised Qwen3.5-35B-A3B prefill by 29% at 2K tokens and 32% at 8K, with decode unchanged.", "details": "The dead band was measured on Qwen3.6-27B (llama.cpp b10107, Mesa 26.0.3). It is not yet confirmed on the 35B-A3B MoE. A similar decode collapse reported on the 35B with an R9700 in one build was not reproduced on a newer one. The -ub 2048 gain is hardware-dependent: on a Strix Halo APU, Qwen3.6-35B was 12% slower at -ub 2048 than at 1024 with no prior context. Memory: llama.cpp's compute buffer grows linearly with -ub. Scaling a public Vulkan log (690.66 MiB at -ub 1088, 262K context) puts -ub 2048 at ~1.3 GiB (estimate). With the 35B UD-Q5_K_XL plus f16 KV at 262K (29.25 GiB measured), that leaves roughly 1 GiB free on a 32 GB card. -ub 4096 only fits with q8_0 KV. Measuring this on our own 35B lane is next.", "setup": { "model": "Qwen3.6-35B-A3B / Qwen3.5-35B-A3B / Qwen3.6-27B", "quant": "UD-Q4_K_XL, Q4_K_M (published); UD-Q5_K_XL (fit estimate)", "engine": "llama.cpp Vulkan (RADV), published", "gpus": "1x Radeon AI PRO R9700", "context": "262,144 for the fit estimate", "flags": "-ngl 99 -fa 1, -b 16384 -ub 2048 vs default -b 2048 -ub 512" }, "metrics": [ { "label": "35B-A3B prefill pp2048, -ub 512 vs 2048 (R9700, published)", "value": "2,381 -> 3,074", "unit": "tok/s" }, { "label": "35B-A3B prefill pp8192, -ub 512 vs 2048 (R9700, published)", "value": "2,262 -> 2,983", "unit": "tok/s" }, { "label": "27B hybrid prefill pp512 at -ub 65 vs 512 (RDNA4, published)", "value": "16.2 vs 774.4", "unit": "tok/s" }, { "label": "27B hybrid decode at -ub 128 vs 512 (RDNA4, published)", "value": "2.09 vs 25.3", "unit": "tok/s" }, { "label": "Compute buffer at -ub 2048, 262K (estimate)", "value": 1.27, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/21043", "https://github.com/ggml-org/llama.cpp/issues/26795", "https://github.com/ggml-org/llama.cpp/issues/25121", "https://github.com/dzltron/strix-halo-llamacpp" ], "next": "Sweep -ub 64-2048 (plus 64-300-token prompts) on our 35B lane and measure free VRAM at 262K with -ub 1024 and 2048 before changing anything." }, { "id": "2026-10-30-radeon-llm-price-per-gb", "date": "2026-10-30T17:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Radeon for local LLMs, Oct 30 prices: the R9700 is the cheapest new card that holds Qwen3.6-35B at 256K; the 9070 XT costs the same per GB for half the VRAM", "summary": "Cheapest new, store-sold US prices today: R9700 32 GB $1,750 ($54.7/GB), RX 9070 XT 16 GB $830 ($51.9/GB), RX 9070 $700, RX 9060 XT 16 GB $500 ($31.2/GB), PRO W7800 32 GB $2,100, PRO W7900 48 GB $3,999. Only 32 GB+ cards run the 35B fully in VRAM at 256K; 16 GB cards need expert offload to system RAM.", "details": "The only 32 GB Radeon Newegg itself sells in stock today is the PRO W7800 (Gigabyte, $2,100): $350 (20%) more than the cheapest R9700, with 576 vs 640 GB/s bandwidth on the older RDNA 3 chip. Newegg's R9700 listings are marketplace sellers. No new store-sold RX 7900 XTX, 7900 XT, 7900 GRE or 7600 XT offers were listed. Prices are US, new, sold by the store itself, before tax; AMD's R9700 launch MSRP was $1,299. Fit verdicts use the site's 256K calculator totals, plus our measured 28.7 GB peak for UD-Q5_K_XL at 262K on one R9700.", "metrics": [ { "label": "R9700 32 GB, cheapest new", "value": 1750, "unit": "USD" }, { "label": "R9700 price per GB VRAM", "value": 54.7, "unit": "USD/GB" }, { "label": "RX 9070 XT price per GB VRAM", "value": 51.9, "unit": "USD/GB" }, { "label": "RX 9060 XT 16 GB price per GB VRAM", "value": 31.2, "unit": "USD/GB" }, { "label": "PRO W7800 32 GB at Newegg (store-sold)", "value": 2100, "unit": "USD" } ], "sources": [ "https://vga.live/gpus/radeon-ai-pro-r9700", "https://vga.live/gpus/radeon-rx-9070-xt", "https://vga.live/gpus/radeon-rx-9070", "https://vga.live/gpus/radeon-rx-9060-xt-16gb", "https://vga.live/gpus/radeon-pro-w7800", "https://vga.live/gpus/radeon-pro-w7900", "https://www.amd.com/en/products/graphics/workstations/radeon-pro/w7800.html" ], "next": "Prices checked 2026-10-30 5:03 PM CT; re-check before buying." }, { "id": "2026-10-30-strata-q36-262k-fit", "date": "2026-10-30T16:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Strata's Qwen3.6-35B path is fast on one R9700, but its fp32 cache won't hold 262K with every expert on the card", "summary": "Strata (0.1.45+) publishes 117.8 tok/s decode and 183.7 with --mtp 2 for Qwen3.6-35B UD-Q4_K_XL on one R9700, but those are short-context numbers. Its Qwen3.6 engine keeps 45 KB of fp32 K/V per token (llama.cpp f16: 20 KB), so at 262,144 tokens the cache alone is ~11 GiB and UD-Q4_K_XL needs ~33 GiB (estimate), more than a 32 GB card: some experts would spill to DDR5.", "details": "Estimate from the docs' 45 KB/token, a 0.77 GiB reserve and the 21.28 GiB UD-Q4_K_XL MTP file: every expert stays on one R9700 up to about 224K tokens. UD-Q6_K_XL (30.37 GiB) only fits all-in-VRAM at very short context. llama.cpp's f16 KV at 262K is 5 GiB, which is why UD-Q5_K_XL runs at full 262K there. Strata's plain-decode lead was measured against a llama.cpp baseline of ~82 tok/s; this site's run cards show llama.cpp at 117.7 (Q4_K_M) and 102.4 (UD-Q5_K_XL) at 4K on one R9700, so the real gains are MTP (the 35B's MTP block is Q5_K/Q6_K/Q8_0, not Q2_0) and serving several requests (325 vs 208 tok/s at 8 requests). Strata has no UD-Q5_K_XL for this model. For Qwen 3.8 Flash, Strata's runtime draft head is still Q2_0-only (issue #3728 open), so drafter-less setups only get the non-MTP gains since 0.1.42 (HC_Q8 +2.5-4% decode on gfx12, K-quant prompt kernels x1.3-1.7, long-prompt top-k +8.7% at 131K, 0.2.7's R9700 split +14-31% on IQ3_S).", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "UD-Q4_K_XL / UD-Q6_K_XL", "engine": "Strata (published) vs llama.cpp", "engine_version": "Strata 0.1.45-0.2.8 docs", "gpus": "1x Radeon AI PRO R9700 32 GB", "context": "262144", "flags": "estimate from published cache size" }, "metrics": [ { "label": "Strata Qwen3.6 cache per token (fp32 K/V)", "value": 45, "unit": "KB" }, { "label": "llama.cpp f16 KV per token", "value": 20, "unit": "KB" }, { "label": "Strata UD-Q4_K_XL total at 262K (est.)", "value": 33.1, "unit": "GiB" }, { "label": "Max all-in-VRAM context, UD-Q4_K_XL (est.)", "value": 224000, "unit": "tokens" }, { "label": "Strata UD-Q4_K_XL --mtp 2, short prompts (published)", "value": 183.7, "unit": "tok/s" } ], "sources": [ "https://github.com/Niko1221/Strata/blob/main/docs/QWEN36.md", "https://github.com/Niko1221/Strata/blob/main/docs/MODELS.md", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF", "https://github.com/Niko1221/Strata/issues/3728", "https://github.com/Niko1221/Strata/releases/tag/v0.2.7" ], "next": "Measure Strata UD-Q4_K_XL with --mtp 2 at 32K, 128K and 262K on one R9700 against llama.cpp UD-Q5_K_XL at the same depths." }, { "id": "2026-10-30-flash-next-ud-iq4xs-mostly-3bit", "date": "2026-10-30T15:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.8 Flash-Next \"UD-IQ4_XS\" GGUF is mostly 3-bit inside: 57% of its routed experts are IQ3_S", "summary": "Unsloth's Qwen3.8-Flash-Next UD-IQ4_XS (87.2 GiB) looks like a cheaper 4-bit way to fit the model with 64-96 GB of RAM. Its GGUF tensor headers say otherwise: 31.6 of its 55.4 GiB of routed experts are IQ3_S, 18.9 GiB IQ4_NL, 4.2 GiB Q8_0 and only 0.8 GiB IQ4_XS. The 26.8 GiB n-gram table is IQ4_NL. For a 4-bit-or-better file, the smallest unsloth option is UD-Q4_K_XL at 103.7 GiB, which on two 32 GB R9700s needs about 74 GiB of system RAM at 262K context (estimate), so a 96 GB kit.", "details": "Header-only range reads of the 3 UD-IQ4_XS shards (1,224 tensors, arch qwen4exp, 48 expert layers at 1.11-1.66 GiB each); no weights downloaded. File sizes from the Hugging Face tree API (repo sha 766911a, 2026-10-30): UD-IQ4_XS 87.24 GiB, UD-Q4_K_XL 103.69, UD-Q5_K_XL 147.42, UD-Q6_K_XL 157.55. The 74 GiB figure uses the site calculator formula: (file - 64 GiB VRAM) + 7.61 GiB f16 KV at 262,144 + runtime + 8 GiB OS + 16 GiB headroom; a leaner estimate without headroom is ~67 GiB. This is a file-composition fact, not a quality test: IQ3_S on most experts means the file sits below our Q4-and-up line, whatever the name says.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-IQ4_XS vs UD-Q4_K_XL (unsloth GGUF)", "engine": "n/a (GGUF header read)", "engine_version": "n/a", "gpus": "sizing for 2x Radeon AI PRO R9700 (2x 32 GB)", "context": "262,144", "flags": "f16 KV" }, "metrics": [ { "label": "UD-IQ4_XS routed experts stored as IQ3_S", "value": 31.56, "unit": "GiB" }, { "label": "UD-IQ4_XS routed experts total", "value": 55.43, "unit": "GiB" }, { "label": "UD-IQ4_XS file size", "value": 87.24, "unit": "GiB" }, { "label": "UD-Q4_K_XL file size", "value": 103.69, "unit": "GiB" }, { "label": "Est. system RAM, UD-Q4_K_XL on 2x R9700 at 262K", "value": 74.1, "unit": "GiB" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-IQ4_XS", "https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main?recursive=true", "https://amdai.net/flash-ram-offload/" ], "next": "Fold a Qwen 3.8 Flash requirements page (96 GB DDR5 for two R9700s at Q4) into the site; measure real host-RAM use for two-card UD-Q4_K_XL at 262K." }, { "id": "2026-10-30-flash-next-sparse-fa-vulkan-not-hip", "date": "2026-10-30T12:30:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Flash-Next long-context decode on R9700: llama.cpp's sparse attention runs on Vulkan, not on ROCm/HIP", "summary": "Qwen3.8 Flash-Next attends to only ~2,048 picked cells per sparse-attention layer, and llama.cpp can skip the rest. On current master (a34de86, Oct 30) the ROCm/HIP backend enables that sparse path only for NVIDIA and CDNA data-center GPUs, so an R9700 on ROCm reads the whole cache every token. The Vulkan backend does use it on RDNA4: with f16 KV from ~4K depth, with q8_0 KV only from ~32K.", "details": "Source gates: ggml-cuda/fattn.cu shall_use_sparse is true only for NVIDIA Turing+ or AMD MFMA (CDNA, and there only the 576-dim MLA variant); RDNA4 uses WMMA. PR #31259 (merged Oct 29) opened CDNA and says other architectures will follow. ggml-vulkan.cpp uses sparse when cache length >= max(4096, ratio x kept cells), ratio 2 for f16 and 16 for quantized K/V; GGML_VK_FA_SPARSE_DISABLE=1 forces dense for A/B tests. Published Vulkan numbers (PR #29639, RDNA3/RDNA4): Flash-Next q8_0 decode 27.9 -> 32.3 tok/s at 64K (+15.6%), 20.6 -> 21.4 at 128K; on an R9700 the attention kernel at 32K cells took 61 us sparse vs 86 us dense. Estimate (not measured): dense f16 attention over 12 layers reads ~3.0 GiB per token at 128K and ~6.0 GiB at 256K vs ~0.05 GiB sparse, about 10 ms per token at 256K on R9700 bandwidth. Practical upshot until RDNA4 lands on HIP: for long agent contexts with llama.cpp, test the Vulkan build, and keep f16 KV so sparse starts at 4K instead of 32K.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "any (Q4-Q6)", "engine": "llama.cpp", "engine_version": "master a34de86 (2026-10-30)", "gpus": "Radeon AI PRO R9700 (RDNA4, gfx1201)", "context": "4K-256K", "flags": "-fa on; f16 vs q8_0 KV" }, "metrics": [ { "label": "Vulkan q8_0 decode at 64K, sparse vs dense (published)", "value": 15.6, "unit": "%" }, { "label": "R9700 Vulkan attention kernel at 32K cells, sparse", "value": 61, "unit": "us" }, { "label": "R9700 Vulkan attention kernel at 32K cells, dense", "value": 86, "unit": "us" }, { "label": "Est. dense f16 attention read per token at 256K (HIP)", "value": 6.0, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/31259", "https://github.com/ggml-org/llama.cpp/pull/29639", "https://github.com/ggml-org/llama.cpp/pull/28105", "https://github.com/ggml-org/llama.cpp/pull/28770", "https://github.com/ggml-org/llama.cpp/blob/a34de8618b25dfc32ee43749db47d767a9f0a25d/ggml/src/ggml-cuda/fattn.cu", "https://github.com/ggml-org/llama.cpp/blob/a34de8618b25dfc32ee43749db47d767a9f0a25d/ggml/src/ggml-vulkan/ggml-vulkan.cpp" ], "next": "Measure HIP vs Vulkan (and Vulkan with sparse forced off) on two R9700s at 0-200K depth, f16 and q8_0 KV." }, { "id": "2026-10-30-fused-moe-glu-nvidia-only", "date": "2026-10-30T10:15:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp's new fused MoE prefill kernel (+6-9% on Qwen3.6-35B-A3B) is NVIDIA-only; R9700 builds don't get it", "summary": "llama.cpp now fuses the MoE expert gate and up matmuls and the GLU into one kernel for prompt processing (PR #31188, merged 2026-10-28). The PR reports +9.1% (DGX Spark) and +6.1% (RTX 5090) pp16384 on Qwen3.6-35B-A3B UD-Q4_K_M, with identical perplexity. On current master the path is gated to NVIDIA MMA hardware, so HIP builds on the R9700 skip it, and the Vulkan backend has no gate/up/GLU fusion at all.", "details": "Fused types: Q4_K, Q5_K, Q6_K, Q8_0, IQ4_XS. The code comment says the fused write-back relies on NVIDIA's MMA accumulator layout. Decode is different: the older gate/up fusion for single-token matrix-vector kernels (PR #16630) has no AMD exclusion, so HIP decode already fuses. Numbers above are the PR author's NVIDIA results, not R9700 measurements.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "UD-Q4_K_M (PR benchmark)", "engine": "llama.cpp", "engine_version": "master aa588a3463 (2026-10-30)", "gpus": "Radeon AI PRO R9700 (gfx1201) vs NVIDIA", "context": "pp16384", "flags": "-fa on -b 2048 -ub 2048, f16 KV" }, "metrics": [ { "label": "PR #31188 pp16384 gain, DGX Spark, 35B-A3B UD-Q4_K_M", "value": 9.1, "unit": "%" }, { "label": "PR #31188 pp16384 gain, RTX 5090, 35B-A3B UD-Q4_K_M", "value": 6.1, "unit": "%" }, { "label": "Gain on R9700 (HIP or Vulkan) today", "value": 0, "unit": "% (path not taken)" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/31188", "https://github.com/ggml-org/llama.cpp/pull/29948", "https://github.com/ggml-org/llama.cpp/pull/16630", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/ggml-cuda.cu" ], "next": "Measure HIP fused vs unfused and Vulkan prefill for the 35B on one R9700, then scope an RDNA4 port of the fused write-back." }, { "id": "2026-10-30-96gb-kit-board-qvl", "date": "2026-10-30T04:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "96 GB DDR5 for an AM5 dual-R9700 build: the cheapest 2x48 kit is Intel-only; the Flare X5 5600 is $70 less and tested on 3 x8/x8 boards", "summary": "We read G.SKILL's compatibility lists (QVL) for seven 96 GB (2x48 GB) kits and checked Newegg. The cheapest 96 GB listing (Trident Z5 RGB 5600 CL40, $1,549.99) is an Intel XMP kit whose list has no AMD boards. The Flare X5 96 GB DDR5-5600 CL40 (AMD EXPO and XMP) is $1,479.99 sold by Newegg and is on the tested list for the ASRock X870E Taichi, Gigabyte B850 AI TOP and GIGABYTE X870E AORUS MASTER X3D.", "details": "The fastest tested option, Trident Z5 Neo 96 GB DDR5-6000 CL28 (EXPO, 1.35V), is also listed for the MSI MEG X870E GODLIKE, but costs $2,399.99 at Newegg (white version), about $920 more for at most ~7% more theoretical dual-channel bandwidth (89.6 vs 96.0 GB/s). No 6000 CL30 96 GB kit was sold by Newegg itself. The ASRock X870 Taichi Creator, ASUS ProArt X870E-Creator and ProArt B850-Creator Neo were on none of the seven lists; that means untested by the memory maker, not incompatible. Prices are Newegg list prices on Oct 30, 2026 and change often.", "metrics": [ { "label": "Flare X5 96 GB DDR5-5600 at Newegg", "value": 1479.99, "unit": "USD (2026-10-30)" }, { "label": "Intel-only 96 GB DDR5-5600 at Newegg", "value": 1549.99, "unit": "USD (2026-10-30)" }, { "label": "96 GB DDR5-6000 CL28 at Newegg", "value": 2399.99, "unit": "USD (2026-10-30)" }, { "label": "Site x8/x8 boards on the Flare X5 list", "value": 3, "unit": "boards" } ], "sources": [ "https://www.gskill.com/qvl/165/396/1683011081/F5-5600J4040D48GX2-FX5-QVL", "https://www.gskill.com/qvl/165/374/1682577390/F5-5600J4040D48GX2-TZ5RK-QVL", "https://www.gskill.com/qvl/165/390/1736133551/F5-6000J2836F48GX2-TZ5NR-QVL", "https://www.gskill.com/specification/165/396/1683011081/F5-5600J4040D48GX2-FX5-Specification", "https://www.newegg.com/p/pl?d=F5-5600J4040D48GX2&N=8000" ], "next": "Read the board makers' own memory lists for the three boards missing from every 96 GB kit list." }, { "id": "2026-10-30-ram-tier-speed", "date": "2026-10-30T00:15:00-05:00", "project": "dual-r9700", "kind": "result", "title": "More DDR5 buys a bigger model, not a faster one: estimated speed by RAM tier on two R9700s at 256K", "summary": "Estimate from GGUF sizes and memory bandwidth: with two R9700s and experts in DDR5, 96 GB (2 sticks) runs Qwen3.8 Flash UD-Q4 at a ~49-52 tok/s ceiling (~27-35 tok/s llama.cpp-like). 128 GB runs Q5 at ~39 (21-25) or Q6 at ~34 (19-23). 192 GB of 4 sticks runs GLM-5.3 Flash Q4 at only ~8-10 (4-7), and 256 GB of 4 sticks runs the same Flash Q6 about 13% slower than 128 GB of 2 sticks, because Ryzen 9000 drops from DDR5-5600 to DDR5-3600 officially with four DIMMs.", "details": "Bandwidth-only ceiling: per token, VRAM bytes at 75% of 640 GB/s plus DDR5 expert bytes at the read bandwidth for that memory layout; llama.cpp-like band = 55-66% of the ceiling, which is where a measured two-card llama.cpp Flash Q5_K_M run landed. Read bandwidth: 2 DIMMs DDR5-6000 79.8 GB/s and DDR5-5600 68.6 GB/s (published AIDA64, Ryzen 9 9950X); 4 DIMMs at 5200 measured 53.8-56.2 GB/s on our rig; 4 DIMMs at 3600 scaled from that. GLM-5.3 Flash moves ~3.8 GiB of experts from DDR5 per token vs ~0.5 GiB for Flash Q4 (top-8 of 288 wide experts vs top-10 of 512 narrow ones), which is why it is several times slower. No MTP or speculative decoding; engines with MTP can beat these ceilings. For comparison, Qwen3.6-35B UD-Q5_K_XL fully in one R9700's VRAM measured 100.8 tok/s.", "setup": { "model": "Qwen3.8-Flash-Next, GLM-5.3-Flash", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (modeled)", "engine_version": "n/a (estimate)", "gpus": "2x Radeon AI PRO R9700", "context": "262144", "flags": "layer split, experts beyond VRAM in DDR5, f16 KV, 8K fill, no MTP" }, "metrics": [ { "label": "Flash UD-Q4, 96 GB 2x DDR5-5600, ceiling (est)", "value": 49.4, "unit": "tok/s" }, { "label": "Flash UD-Q6, 128 GB 2 sticks, ceiling (est)", "value": 34.1, "unit": "tok/s" }, { "label": "Flash UD-Q6, 256 GB 4 sticks at 5200, ceiling (est)", "value": 29.8, "unit": "tok/s" }, { "label": "GLM-5.3 Flash UD-Q4, 192 GB 4 sticks, ceiling (est)", "value": 10.4, "unit": "tok/s" }, { "label": "DDR5 read, 2 DIMMs DDR5-6000 (published)", "value": 79.8, "unit": "GB/s" }, { "label": "DDR5 read, 4x64 GB at 5200 (measured)", "value": 55, "unit": "GB/s" } ], "sources": [ "https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9950x.html", "https://www.pugetsystems.com/labs/articles/does-2-dimm-per-channel-ram-impact-content-creation-performance/", "https://www.hardwareluxx.de/index.php/artikel/hardware/prozessoren/64190-amd-ryzen-9-9950x-im-test-nachzuegler-fuer-die-spitzenposition.html?start=9", "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "https://huggingface.co/zai-org/GLM-5.3-Flash", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" ], "next": "Measure two-card Flash UD-Q4 at 256K: host RAM used and decode tok/s, and if possible 2 sticks vs 4 sticks on the same model." }, { "id": "2026-10-29-ddr5-96gb-and-outlook", "date": "2026-10-29T23:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "A 96 GB DDR5 kit covers two-card Qwen3.8 Flash at Q4 for $1,549.99, and DRAM prices are still forecast to rise", "summary": "Two R9700s running Qwen3.8 Flash UD-Q4_K_XL at 256K context need about 67 GB of system RAM (estimate), so a two-stick 96 GB (2x48) kit covers it. On Oct 29, 2026 Newegg listed a 96 GB DDR5-5600 kit at $1,549.99 ($16.15/GB), versus $3,799.99 for 128 GB ($29.69/GB). Kits built on 64 GB sticks cost about 1.7x more per GB. TrendForce still expects DRAM contract prices to rise in Q4 2026, so waiting for a drop is not supported by current forecasts.", "details": "RAM need = routed experts and per-layer embeddings that don't fit in 2x 32 GB VRAM after KV cache and compute buffers, plus host buffers and ~10 GB OS; all estimates, not measured. UD-Q5/Q6 need roughly 111-121 GB (128 GB kit); GLM-5.3-Flash Q4/Q5 about 147-184 GB (192 GB kit, marketplace-only listings on Newegg today). Prices are Newegg search-page list prices, not cart checks. TrendForce (Sep 30, 2026) forecasts conventional DRAM contract prices +10-15% QoQ in Q4 2026 and says PC DRAM supply could decline in 2027; a Commercial Times report (Oct 20) puts Q4 at about +20%, slowing from ~80% in Q2 and ~40% in Q3. Two separate 2x48 kits are not a tested 4-stick set.", "setup": { "model": "Qwen3.8 Flash, GLM-5.3-Flash", "quant": "UD-Q4_K_XL to UD-Q6_K_XL", "engine": "llama.cpp estimate (layer split + expert offload)", "gpus": "2x Radeon AI PRO R9700", "context": "256K" }, "metrics": [ { "label": "96 GB DDR5-5600 2x48 kit, Newegg low", "value": 1549.99, "unit": "USD" }, { "label": "96 GB kit per GB", "value": 16.15, "unit": "USD/GB" }, { "label": "128 GB 2x64 kit per GB", "value": 29.69, "unit": "USD/GB" }, { "label": "RAM for Flash UD-Q4 on 2x R9700 at 256K (est.)", "value": 67, "unit": "GB" }, { "label": "TrendForce Q4 2026 DRAM contract forecast", "value": "10-15", "unit": "% QoQ" } ], "sources": [ "https://www.newegg.com/p/pl?d=DDR5+96GB+%282+x+48GB%29+desktop+memory", "https://www.trendforce.com/presscenter/news/20260930-13258.html", "https://www.trendforce.com/news/2026/10/20/news-dram-prices-seen-rising-20-qoq-in-4q26-despite-slower-gains-ai-demand-tight-supply-support-outlook/", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" ], "next": "Measure host RAM use for two-card Flash UD-Q4 at 256K to confirm the 96 GB estimate." }, { "id": "2026-10-29-ddr5-kit-prices-256k", "date": "2026-10-29T22:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "How much DDR5 a local LLM needs at 256K context, and what those kits cost at Newegg today", "summary": "Models that fit in one R9700's 32 GB VRAM (Qwen3.6-35B-A3B UD-Q5_K_XL, 27B dense, gpt-oss-20b) need only a 32 GB DDR5 kit even at 256K context (calculator estimate). Qwen3.8 Flash on two R9700s needs about 96 GB at UD-Q4 and 128 GB at UD-Q5; UD-Q6 and GLM-5.3-Flash need 192 GB. Newegg list prices on Oct 29, 2026 run about $16/GB for 2x32 GB kits but about $30/GB for 64 GB modules, so a 256 GB kit lists at $7,599.99.", "details": "RAM estimates = weights that spill out of VRAM + KV cache in RAM + runtime + 8 GB OS/desktop + 16 GB headroom, rounded up to a kit size (GiB reported as GB). Prices are Newegg search-page list prices, not cart checks; some listings are marketplace sellers. 192 GB (4x48) kits showed only marketplace listings at $4,956.99-$4,996.99. For two-card Flash at Q4/Q5, a 128 GB two-stick kit covers the estimate for about half the cost of 256 GB.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8 Flash, GLM-5.3-Flash and others", "quant": "Q4_K_M to UD-Q6_K_XL", "engine": "calculator estimate", "gpus": "1-2x Radeon AI PRO R9700", "context": "256K" }, "metrics": [ { "label": "32 GB DDR5-6000 2x16 kit", "value": 659.99, "unit": "USD" }, { "label": "64 GB DDR5 2x32 kit, low", "value": 999.99, "unit": "USD" }, { "label": "128 GB DDR5 2x64 kit, low", "value": 3799.99, "unit": "USD" }, { "label": "256 GB DDR5-6000 4x64 kit (G.SKILL Flare X5)", "value": 7599.99, "unit": "USD" }, { "label": "RAM for Flash UD-Q5 on 2x R9700 at 256K (est.)", "value": 128, "unit": "GB" } ], "sources": [ "https://www.newegg.com/p/pl?d=DDR5+64GB+%282+x+32GB%29+desktop+memory", "https://www.newegg.com/p/pl?d=DDR5+128GB+%282+x+64GB%29+desktop+memory", "https://www.newegg.com/p/pl?d=DDR5+256GB+%284+x+64GB%29+desktop+memory", "https://amdai.net/flash-ram-offload/", "https://amdai.net/can-run/qwen36-35b-on-r9700/" ], "next": "Price 2x48 GB (96 GB) kits for two-card Flash at Q4 and recheck prices weekly." }, { "id": "2026-10-29-256gb-kit-x8x8-board-qvl", "date": "2026-10-29T20:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "256 GB DDR5 for a dual-R9700 build: the kit makers' tested lists skip most x8/x8 boards", "summary": "G.SKILL's two consumer 256 GB (4x64 GB) DDR5-6000 UDIMM kits list 23 and 15 tested boards. None are the common dual-GPU creator boards (ASRock X870 Taichi Creator, X870E Taichi, ASUS ProArt X870E-Creator, ProArt B850-Creator Neo, Gigabyte B850 AI TOP). Two x8/x8 boards are listed: Gigabyte X870E AORUS MASTER X3D (CL36 kit) and MSI MEG X870E GODLIKE (both kits).", "details": "Not being on a kit's list doesn't mean it fails; check the board maker's own memory support list and use a 2025-or-newer BIOS with 64 GB-module support. Watch out for the MSI MPG X870E CARBON WIFI: it's on both kit lists, but its two CPU slots run x8/x4, not x8/x8. ASUS's ProArt X870E-Creator BIOS 1003 notes report 4x64 GB enabled at 5200 MT/s. Four sticks run above AMD's official 4-DIMM speed when set to 6000.", "setup": { "gpus": "2x Radeon AI PRO R9700", "ram": "4x64 GB DDR5 UDIMM", "platform": "AM5, Ryzen 9000" }, "metrics": [ { "label": "Boards on CL36 256 GB kit list", "value": 23, "unit": "boards" }, { "label": "Boards on CL34 256 GB kit list", "value": 15, "unit": "boards" }, { "label": "Common x8/x8 creator boards on either list", "value": 0, "unit": "of 5" } ], "sources": [ "https://www.gskill.com/qvl/165/390/1750238051/F5-6000J3644D64GX4-TZ5NR-QVL", "https://www.gskill.com/qvl/165/390/1750730893/F5-6000J3444F64GX4-TZ5NR-QVL", "https://www.gigabyte.com/Motherboard/X870E-AORUS-MASTER-X3D/sp", "https://www.msi.com/Motherboard/MEG-X870E-GODLIKE/Specification", "https://www.msi.com/Motherboard/MPG-X870E-CARBON-WIFI/Specification", "https://flashmyboard.com/mb/Asus_ProArt_X870E-CREATOR_WIFI" ], "next": "Check the board makers' own memory lists for 4x64 GB on the creator boards." }, { "id": "2026-10-29-expert-offload-granularity", "date": "2026-10-29T19:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Offloading only up/gate experts to RAM is likely slower than whole layers for Flash-Next on 2x R9700", "summary": "The popular llama.cpp recipe that moves only ffn_up/ffn_gate experts to the CPU and keeps ffn_down on the GPU reads the same DDR5 bytes per token as whole-layer --n-cpu-moe, but spreads them over ~1.8x as many layers (UD-Q5: 41 vs 23), and each touched layer adds two CPU-GPU hand-offs per token. Our estimate: 3% slower at 20 us per hand-off, 7% at 60 us, 15% at 180 us. Filling leftover VRAM with one more expert tensor gains about 1%.", "details": "Estimates, not measurements. Model: GPU decode at 75% of 640 GB/s, 55 GB/s DDR5 read, 262K f16 KV, 10 of 512 experts active, -sm layer. The per-hand-off cost on ROCm/HIP is unknown; 180 us is a third-party CUDA claim. The 'keep ffn_down on GPU' advice we found had no A/B behind it; in Unsloth UD quants ffn_down simply holds more bytes per weight (0.83 of 1.909 GiB per layer at UD-Q5). Note that --n-cpu-moe N takes the first N layers, which all sit on the first card under layer split, so the tensor split needs rebalancing. On Vulkan, one report saw expert offload regress 9-11% on an RX 580.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (modeled)", "engine_version": "n/a", "gpus": "2x Radeon AI PRO R9700", "context": "262K", "flags": "-sm layer, f16 KV, --n-cpu-moe vs -ot ffn_(up|gate)_exps=CPU" }, "metrics": [ { "label": "UD-Q5 est. whole-layer decode (60 us/hand-off)", "value": 31.2, "unit": "tok/s" }, { "label": "UD-Q5 est. up/gate-only decode (60 us/hand-off)", "value": 29.2, "unit": "tok/s" }, { "label": "Layers touched at UD-Q5, whole-layer vs up/gate-only", "value": "23 vs 41", "unit": "layers" }, { "label": "Est. gain from one-tensor VRAM fill", "value": 1, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/15077", "https://github.com/ggml-org/llama.cpp/pull/29943", "https://github.com/ggml-org/llama.cpp/pull/28414", "https://github.com/ggml-org/llama.cpp/discussions/24846", "https://github.com/voidstackloop/extended-strata-moe", "https://medium.com/@david.sanftenberg/gpu-poor-how-to-configure-offloading-for-the-qwen-3-235b-a22b-moe-model-using-llama-cpp-13dc15287bed", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" ], "next": "Measure the hand-off cost on HIP with an ABBA llama-bench run of whole-layer vs up/gate-only offload at equal DDR5 bytes." }, { "id": "2026-10-29-newegg-r9700-marketplace-only", "date": "2026-10-29T13:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Newegg's only in-stock Radeon AI PRO R9700 is a marketplace listing at $2,799.99, 56-60% over the card's going price", "summary": "On Oct 29, 2026 (about 1:10 PM CDT) Newegg itself had no R9700 in stock: the Sapphire card was sold out (last $1,799.99) and the ASRock Creator out of stock (last $1,699.99). The only in-stock new R9700 was the GIGABYTE AI TOP at $2,799.99, sold and shipped by a third-party marketplace seller, with a second offer at $3,290.00 shipping from Hong Kong. That is 56% over Newegg's own last R9700 price and 60% over the cheapest in-stock new R9700 we found earlier the same day ($1,749.99, in-store).", "details": "The Newegg item page says to contact the seller for warranty information. GIGABYTE's US warranty policy asks for a dated receipt from an authorized dealer and lists receipts from non-authorized dealers as not valid proof of purchase; without valid proof, the 3-year warranty is counted from the manufacture date. We could not confirm whether this seller is authorized. Before buying a marketplace listing, check the seller name under the price and ask for the warranty terms in writing. Other cards the same day: no new RX 7900 XTX and no Radeon PRO W7900 sold by Newegg, while many new RX 9070 XT cards were sold by Newegg and in stock ($879.99-$1,169.99). Our buy buttons will mark Newegg as marketplace-only when that is the case instead of sending readers there as the main pick.", "setup": { "retailer": "Newegg.com (US)", "checked": "2026-10-29 ~13:10 CDT", "cards": "Radeon AI PRO R9700, RX 7900 XTX, Radeon PRO W7900, RX 9070 XT" }, "metrics": [ { "label": "Cheapest in-stock new R9700 on Newegg (marketplace seller)", "value": 2799.99, "unit": "USD" }, { "label": "Newegg's own last R9700 price (Sapphire, sold out)", "value": 1799.99, "unit": "USD" }, { "label": "Markup vs cheapest in-stock new R9700 found the same day", "value": 60, "unit": "%" }, { "label": "R9700 models sold by Newegg in stock", "value": 0, "unit": "count" } ], "sources": [ "https://www.newegg.com/gigabyte-gv-r9700ai-top-32gd-radeon-ai-pro-r9700-32gb-graphics-card-turbo-fan-cooling-system/p/N82E16814932822", "https://www.newegg.com/p/pl?d=Radeon+AI+PRO+R9700&N=100007709", "https://www.gigabyte.com/us/Support/Consumer/Warranty", "https://www.gigabyte.com/Support/Consumer/Warranty/Graphics-Card", "https://www.reddit.com/r/Newegg/comments/1bnr5s2/how_to_know_if_it_sold_and_ship_by_newegg/" ], "next": "Make Newegg buy buttons seller- and stock-aware: primary only when sold by Newegg and in stock, marked as marketplace-only otherwise, with dated prices." }, { "id": "2026-10-29-35b-q6-one-card-256k-q8-kv", "date": "2026-10-29T12:25:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.6-35B-A3B at Q6 and full 262K context on one 32 GB card: only with a q8_0 KV cache, and it is not a free quality win", "summary": "From the GGUF headers, the 35B keeps KV only in its 10 full-attention layers, so the full 262,144-token cache is 5.00 GiB at f16 and 2.66 GiB at q8_0. Unsloth's UD-Q6_K puts 26.78 GiB of weights on the card (token embedding in system RAM), so with an f16 cache it needs 31.78 GiB at 262K and does not fit one 32 GB Radeon AI PRO R9700; with q8_0 it needs 29.44 GiB, only 0.19 GiB more than the UD-Q5_K_XL + f16 setup we run daily at 262K. UD-Q6_K_XL (29.14 GiB on card) cannot reach 256K on one card either way.", "details": "Quality is closer than the bit count suggests. A Q5-vs-Q6 rebuild of this model with Unsloth's recipes measured mean KL divergence 0.0145 for UD-Q5_K_XL and 0.0125 for UD-Q6_K, while a q8_0 KV cache on the 35B measured a mean KLD of about 0.0015 against f16 (98.4% same top token), so Q6 weights plus a q8_0 cache may land close to Q5 with an f16 cache rather than clearly ahead; it needs a direct KLD test. The likelier gain is speed at depth: decode at 256K is dominated by reading the KV cache, and llama.cpp's Vulkan flash attention reads q8_0 K/V directly. Our labeled estimate for one R9700 at 256K depth: about 48 tok/s today (UD-Q5_K_XL, f16 KV) versus about 54-64 with a q8_0 cache on either Q5 or Q6; short-context speed about the same or a few percent lower. With an f16 cache, UD-Q6_K tops out around 126K context in the same footprint.", "setup": "Sizes are exact tensor bytes from Unsloth's GGUF headers; the 262K fit baseline is our measured one-card UD-Q5_K_XL + f16 KV run on llama.cpp Vulkan. Speeds at depth are estimates, not measurements.", "metrics": [ { "label": "Full 262K KV cache, f16", "value": 5.0, "unit": "GiB" }, { "label": "Full 262K KV cache, q8_0", "value": 2.66, "unit": "GiB" }, { "label": "UD-Q6_K + q8_0 KV on the card at 262K", "value": 29.44, "unit": "GiB" }, { "label": "UD-Q6_K + f16 KV on the card at 262K", "value": 31.78, "unit": "GiB" }, { "label": "Estimated decode at 256K depth with q8_0 KV (one R9700)", "value": "54-64", "unit": "tok/s (estimate)" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-vulkan/ggml-vulkan.cpp", "https://github.com/TheTom/turboquant_plus/blob/main/docs/kl-divergence-results.md", "https://github.com/ggml-org/llama.cpp/issues/20035", "https://huggingface.co/lambsea/Qwen3.6-35B-A3B-Abliterated-Heretic-UD-GGUF", "https://openclawdc.com/blog/best-local-llm-32gb-vram/" ], "next": "Measure load headroom, decode at 0/32K/128K/240K depth and KL divergence for UD-Q5_K_XL f16, UD-Q5_K_XL q8_0 and UD-Q6_K q8_0 on one R9700." }, { "id": "2026-10-29-newegg-landing-audit", "date": "2026-10-29T11:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Where a Newegg search link lands: an R9700 search shows 1 card and about 30 prebuilt PCs", "summary": "A plain Newegg search for 'AMD Radeon AI PRO R9700' (checked Oct 29, 2026) listed one R9700 card and then about 30 prebuilt workstations priced $4,999.99 to $18,599.00. Adding Newegg's graphics-card category filter (N=100007709) returned cards only. Searching an exact part number lands on that part.", "details": "Same check on other generic searches: RX 7900 XTX mixed in GPU water blocks and sponsored off-target cards; a 'DDR5 64GB desktop kit' search put sponsored DDR4 and 16 GB kits on top, then marketplace kits up to $4,095.99; an RTX 3090 search showed gaming PCs with other cards first; a 1000 W PSU search was mostly on target but included SFX and 850 W units. With the category filter the R9700 list showed the Gigabyte AI TOP at $2,799.99 in stock, the Sapphire sold out (last sold $1,799.99) and the ASRock Creator out of stock (last sold $1,699.99). Lesson for any buying guide: link the exact part number or a category-filtered search, never a bare product-name search. Prices are Newegg's listed prices at fetch time and change often.", "setup": { "retailer": "Newegg.com (US)", "checked": "2026-10-29 ~11:15 CDT", "method": "7 public search pages fetched and read" }, "metrics": [ { "label": "R9700 cards in plain R9700 search results", "value": 1, "unit": "listings" }, { "label": "Prebuilt PCs in the same results", "value": "~30", "unit": "listings" }, { "label": "Cheapest prebuilt in results", "value": 4999.99, "unit": "USD" }, { "label": "Most expensive prebuilt in results", "value": 18599.0, "unit": "USD" }, { "label": "R9700 Gigabyte AI TOP listed price (in stock)", "value": 2799.99, "unit": "USD" } ], "sources": [ "https://www.newegg.com/p/pl?d=AMD+Radeon+AI+PRO+R9700", "https://www.newegg.com/p/pl?d=Radeon+AI+PRO+R9700&N=100007709", "https://www.newegg.com/p/pl?d=AMD+Radeon+RX+7900+XTX", "https://www.newegg.com/p/pl?d=DDR5+64GB+desktop+kit", "https://www.newegg.com/p/pl?d=NVIDIA+GeForce+RTX+3090", "https://www.newegg.com/p/pl?d=F5-6000J3644D64GX4-TZ5NR" ], "next": "Switch the site's Newegg buttons to category-filtered or part-number links, then compare click-through over the next week." }, { "id": "2026-10-29-buy-click-bot-filter", "date": "2026-10-29T10:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "First full day of server-side buy-click counting: 97 raw clicks, 0 from people", "summary": "We count buy-button clicks from our web server's access log, with no page scripts. A simple User-Agent bot filter reported 97 buy clicks in the first ~22 hours. Checking each visitor by hand showed none came from a person: browser-looking crawlers, a rotating-proxy burst, and a single visitor opening 16 shop links in 4 seconds. A stricter counter that requires the same visitor to have loaded the referring page and its styles finds 0.", "details": "Breakdown of the 97: 44 from a crawler with a fake old-iPhone User-Agent on cloud IP ranges, one link every ~10 minutes, no referring page; 30 from a 6-minute burst of 30 rotating residential IPs where the redirect and its follow-up came from different addresses; 8 from one address opening 16 links in 4 seconds; 15 others with no page-plus-assets visit. Lesson for anyone measuring affiliate clicks from logs: a User-Agent filter is not enough. Require a same-visitor page view with asset loads before the click, drop sessions that open many shops at once, and disallow the redirect path in robots.txt.", "setup": "Static site behind Caddy with JSON access logs; buy buttons are /go// redirect stubs (noindex, nofollow).", "metrics": [ { "label": "Raw buy clicks (User-Agent filter only)", "value": "97", "unit": "clicks" }, { "label": "Clicks that passed the page-visit check", "value": "0", "unit": "clicks" }, { "label": "Visitors that loaded any CSS/JS", "value": "26", "unit": "IPs" }, { "label": "Window", "value": "22", "unit": "hours" } ], "sources": [], "next": "Use the stricter counter for every buy-box test and re-measure after 7 days." }, { "id": "2026-10-29-70b-on-24gb-check", "date": "2026-10-29T08:25:00-05:00", "project": "idea-lab", "kind": "result", "title": "Llama 3.3 70B at Q4 does not fit on a 24 GB Radeon; at 256K only 4 of 20 Radeons hold Qwen3.6-35B fully in VRAM", "summary": "Some 2026 AMD buyer guides say a 24 GB RX 7900 XTX runs Llama 3.3 70B at Q4 with room for the KV cache. The Q4_K_M file alone is 42.52 GB (publisher table), about 43 GB with 8K context by our calculator, so it needs a 48 GB card, two 24 GB cards, or CPU offload. Judged at the full 262,144-token context, only the W7900 (Q6) and the 32 GB R9700, W7800 and W6800 (Q5) hold Qwen3.6-35B entirely in VRAM; the 7900 XTX tops out at Q4 with 128K.", "details": "Estimates from the amdai.net calculator data (GiB, called GB; f16 KV; 0.5 GB kept free). Qwen3.6-35B-A3B at 262K: Q4_K_M ~25.5, Q5_K_M ~28.9, Q6_K ~32.4. Llama 3.3 70B Q4_K_M: weights ~39.9 + 8K KV ~2.5 + runtime ~0.7 = ~43.1. 15 of the 20 sized Radeons cannot hold the 35B fully in VRAM at any context of 32K or more; on 16 GB cards it runs with experts in system RAM, more slowly.", "setup": { "model": "Qwen3.6-35B-A3B; Llama 3.3 70B", "quant": "Q4_K_M / Q5_K_M / Q6_K", "engine": "llama.cpp (calculator estimate)", "context": "262144 (35B); 8192 (70B)" }, "metrics": [ { "label": "Llama 3.3 70B Q4_K_M total at 8K, est.", "value": 43.1, "unit": "GB" }, { "label": "Qwen3.6-35B Q5_K_M at 262K, est.", "value": 28.9, "unit": "GB" }, { "label": "Radeons holding the 35B at 262K", "value": 4, "unit": "of 20" } ], "sources": [ "https://huggingface.co/bartowski/Llama-3.3-70B-Instruct-GGUF", "https://amdai.net/api/v1/models.json", "https://amdai.net/api/v1/gpus.json", "https://www.compute-market.com/blog/best-amd-gpu-local-llm-inference-2026" ], "next": "A 'best AMD GPU for local LLMs' page that picks cards at full context." }, { "id": "2026-10-29-mixed-radeon-r9700", "date": "2026-10-29T07:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Adding an R9700 to a 7900 XTX or 9070 XT: run the 35B on the R9700 alone, don't split it", "summary": "Owners of a 24 GB or smaller Radeon can't hold Qwen3.6-35B-A3B at full 256K context (est. Q4 25.5 GB; a 7900 XTX tops out near Q4 at 128K). Adding a 32 GB R9700 fixes that, but the fast setup is one model per card: the 35B on the R9700 alone at Q5 (est. 28.9 GB), the old card on a second model. Splitting one model across mixed cards is slower than one card in public reports, and mixed RDNA3 + RDNA4 splits still crash or corrupt on some stacks.", "details": "Public reports: a 7900 XT + 2x R9700 owner found the 7900 XT 'doesn't help much' and that one card is much faster than a split; a 7900 XTX + R9700 owner runs two models at once on llama.cpp Vulkan. A Vulkan layer split across a 7900 XTX and an RTX 3090 was slightly slower than either card alone. Open bugs (checked 2026-10-29): Ollama's bundled ROCm 7.2.x crashes with 'no kernel image' when a MoE model is split across an R9700 and a 7900 XTX (each card alone works); Windows HIP splits across gfx1100 + gfx1201 give silent gibberish unless built with GGML_CUDA_NO_PEER_COPY. A similar llama.cpp mixed-arch crash was fixed by rebuilding rocBLAS with rocm-libraries PR #4781. On Linux, a mixed box needs ROCm's multi-arch packages (or llama.cpp Vulkan) and llama.cpp built for both gfx1100 and gfx1201. Our own 2x R9700 sweep of the 35B: one card was faster in 12 of 16 rows. Sizes are estimates in GiB from the site calculator's formula.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / Q5_K_M", "engine": "llama.cpp (ROCm multi-arch or Vulkan)", "context": "262144", "flags": "one model per GPU via HIP_VISIBLE_DEVICES" }, "metrics": [ { "label": "35B Q5_K_M at 256K on one R9700 (est.)", "value": 28.9, "unit": "GB" }, { "label": "Best 35B fit on a 24 GB RX 7900 XTX alone (Q4, 128K, est.)", "value": 23.0, "unit": "GB" }, { "label": "2x R9700 35B sweep rows where one card beat the split (measured)", "value": "12 of 16", "unit": "rows" }, { "label": "7900 XTX vs R9700 single-card token-gen (published, Q4 GGUFs)", "value": "+24 to +29", "unit": "%" } ], "sources": [ "https://www.reddit.com/r/ROCm/comments/1sogv3x/question_7900xtx_with_r9700_ai_pro/", "https://www.reddit.com/r/unsloth/comments/1thoonn/rx_7900_xtx_vs_radeon_ai_pro_r9700_llamacpp/", "https://github.com/ollama/ollama/issues/18162", "https://github.com/ggml-org/llama.cpp/issues/28676", "https://github.com/ggml-org/llama.cpp/issues/19893", "https://github.com/ROCm/TheRock/blob/main/RELEASES.md", "https://amdai.net/compare/one-vs-two-gpu/" ], "next": "Add an 'already own a Radeon?' verdict page and fix the 7900 XTX sheet's 'do not mix a gfx1201 install' line for mixed-card owners." }, { "id": "2026-10-29-35b-full-context-which-cards", "date": "2026-10-29T06:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Which Radeons hold Qwen3.6-35B-A3B fully in VRAM at 256K context (Q4+ only)", "summary": "At the full 262,144-token context with f16 KV, Qwen3.6-35B-A3B needs an estimated 25.5 GB at Q4_K_M, 28.9 GB at Q5_K_M and 32.4 GB at Q6_K. No 16, 20 or 24 GB Radeon holds it at full context: an RX 7900 XTX tops out around Q4 at 128K (est. 23.0 GB). 32 GB cards (R9700, W7800, W6800) hold Q5 at 256K, and a 48 GB W7900 holds Q6.", "details": "Estimate = weights (35B params x bits per weight) + KV cache + 0.71 GB runtime, keeping 0.5 GB free. The model keeps KV on only 10 of its 40 layers (2 KV heads x 256 head dim), so 256K of f16 KV is only about 5 GB; the weights decide the fit. The same formula reproduces the calculator's 4K and 32K totals within 0.1 GB. Our measured Unsloth UD-Q5_K_XL run at 256K on one R9700 fits, consistent with this. Sizes in GiB.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / Q5_K_M / Q6_K", "engine": "llama.cpp", "context": "262144", "flags": "f16 KV, all layers on GPU" }, "metrics": [ { "label": "Q4_K_M at 256K (est.)", "value": 25.5, "unit": "GB" }, { "label": "Q5_K_M at 256K (est.)", "value": 28.9, "unit": "GB" }, { "label": "Q6_K at 256K (est.)", "value": 32.4, "unit": "GB" }, { "label": "KV cache at 256K, f16 (est.)", "value": 5.0, "unit": "GB" } ], "sources": [ "https://amdai.net/models/qwen36/", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "https://amdai.net/api/v1/models.json" ], "next": "Add a 256K column and a plain per-pair pick to the GPU compare pages." }, { "id": "2026-10-29-flash-fewer-experts-topk", "date": "2026-10-29T05:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Fewer active experts per token: a one-flag speed lever for DDR5-offloaded MoE decode", "summary": "Qwen3.8 Flash Next routes each token to 10 of its 512 experts. When experts sit in system RAM, decode speed tracks how many expert bytes each token reads, and llama.cpp can lower the count at load time with --override-kv qwen4exp.expert_used_count=int:N, with no new download. A 2026 study of 12 fine-grained MoE checkpoints found keeping about two thirds of the chosen experts preserved 98.8% of benchmark performance on average with a 1.2-1.7x measured speedup.", "details": "Cited measurements: a llama.cpp contributor measured Qwen3-VL-30B-A3B CPU-only on DDR5 going from 25.1 to 29.4 tok/s (+17%) at 4 of 8 experts through --override-kv. A second controlled study found plain static k reduction is the baseline that dynamic expert-skipping methods rarely beat, and recommended keeping more experts for image tokens. Our labeled estimate for two R9700s with DDR5 expert offload (llama.cpp layer split, 262K f16 KV): UD-Q5_K_XL from about 34 to 42 tok/s at k=7, UD-Q6_K_XL from about 30 to 38. Real gains will be smaller because routing and kernel launches don't shrink. The open question is quality: whether Q6 at 7 experts lands closer to full Q6 than Q5 at 10 does, which needs a KL-divergence test, not a guess. Not useful for models that already fit in VRAM.", "setup": "Estimate only: 2x Radeon AI PRO R9700, DDR5 expert offload, llama.cpp -sm layer; no measurement yet on this setup.", "metrics": [ { "label": "Experts kept that preserved 98.8% of performance (12 checkpoints)", "value": 67, "unit": "%" }, { "label": "Cited llama.cpp CPU-only gain, 4 of 8 experts via --override-kv", "value": 17, "unit": "%" }, { "label": "Estimated UD-Q5_K_XL decode at 7 of 10 experts (2x R9700 + DDR5)", "value": 42, "unit": "tok/s (estimate)" } ], "sources": [ "https://arxiv.org/abs/2609.25809", "https://arxiv.org/html/2610.28369", "https://github.com/ggml-org/llama.cpp/pull/18029", "https://github.com/ggml-org/llama.cpp/issues/19528" ], "next": "Measure KL divergence and decode speed for Q5 and Q6 at 10, 8, 7 and 6 experts against a full Q6 reference." }, { "id": "2026-10-29-r9700-prebuilts-newegg", "date": "2026-10-29T04:25:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 prebuilts on Newegg: only one AM5 box names a dual x8/x8 board, and no in-stock two-card AM5 system", "summary": "Newegg's own assembly service is discontinued, so assembled R9700 systems there come from marketplace builders or ABS. Of 7 listings checked on Oct 29, 2026, only the AVGPC FrostFire ($5,797) names a verified dual x8/x8 board (ASRock X870 Taichi Creator). Adamant's configs use an ASUS PRIME Z890-P with one CPU-wired x16 slot, fine for the 35B on one card but a dead end for a second R9700. The only two-card AM5 listing (Andromeda) is out of stock and ships a 1000 W PSU.", "details": "Checked against the builder's rules: the 35B needs one card and 32 GB RAM; a second card later needs a CPU-wired x8/x8 (or WRX90) board and a 1200 W+ PSU with two native 12V-2x6; Flash-class offload needs two cards and 192-256 GB RAM. Result: 4 listings fit the 35B, 2 fit 35B plus a second card later (AVGPC, and the 3x R9700 Threadripper PRO ABS at $15,999-$18,599), 1 fits Flash-class (ABS). Andromeda lists its board only as 'X870 ATX', so x8/x8 support can't be confirmed. Prices are from Newegg listing text on Oct 29, 2026 and change often.", "metrics": [ { "label": "R9700 prebuilt listings checked", "value": 7, "unit": "" }, { "label": "Listings naming a verified dual x8/x8 AM5 board", "value": 1, "unit": "" }, { "label": "In-stock two-card AM5 prebuilts", "value": 0, "unit": "" }, { "label": "AVGPC FrostFire price (Oct 29, 2026)", "value": 5797, "unit": "USD" } ], "sources": [ "https://kb.newegg.com/article-tags/pc-builder/", "https://www.newegg.com/p/2NS-00JR-00003", "https://www.newegg.com/p/3D5-002T-01WA3", "https://www.asus.com/motherboards-components/motherboards/prime/prime-z890-p/techspec/", "https://www.newegg.com/andromeda-insights-gaming-desktop-pcs-amd-radeon-ai-pro-r9700-ryzen-9-9950x-128gb-4tb-nvme-ssd-black/p/3D5-006J-000E7", "https://www.newegg.com/p/N82E16859991069" ], "next": "A prebuilt checker page that applies the same rules and links only boxes that fit the reader's plan." }, { "id": "2026-10-29-cpu-wake-knobs-hybrid-moe", "date": "2026-10-29T03:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Hybrid GPU+RAM MoE decode: llama.cpp's CPU threads spin at --poll 50 by default, and one public test found --poll 100 worth 8-14%", "summary": "When a big MoE model keeps its experts in system RAM, each token hands work from the GPU to the CPU and back once per offloaded layer. llama.cpp's CPU worker threads spin for 128K x poll rounds waiting for that work before they sleep, and the default poll level is still 50 on current master. A public MoE-offload project (RTX 5090, Ryzen 9 9950X3D, 64 GB DDR5-6000, an 80B-A3B model with 24 layers of experts in RAM, Windows) measured the default costing 8-14% decode against --poll 100, with more run-to-run drift. On Linux, Ryzen 9000 systems also default to the amd-pstate-epp driver with the power-saving governor and the balance_performance hint, which sets how fast an idle core ramps back up. We have not measured either knob on our dual R9700s yet; treat the 8-14% as someone else's result on different hardware.", "details": "How to check your own box: llama-bench accepts --poll, --prio and --cpu-strict, so an A/B is one flag (e.g. -p 0 -n 128 -r 5, with and without --poll 100, stock runs before and after). --poll 0 makes threads sleep at once and shows how much spinning matters at all. OS side: cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor and energy_performance_preference; cpupower frequency-set -g performance sets the governor (amd-pstate-epp then uses EPP performance); cpupower idle-set -D 100 disables idle states with an exit latency of 100 us or more, and idle-set -E re-enables them. Spinning many threads also uses CPU power budget, so --poll 100 is not automatically a win: measure it, and check that a model fully in VRAM does not change (a control).", "setup": { "engine": "llama.cpp (master, CPU threadpool)", "model": "MoE with experts in system RAM", "context": "n/a", "source": "code reading + one cited third-party measurement; not measured on our desktop" }, "metrics": [ { "label": "llama.cpp default --poll level", "value": 50, "unit": "" }, { "label": "Spin rounds per poll level", "value": 131072, "unit": "rounds" }, { "label": "Cited decode cost of poll 50 vs 100 (other rig)", "value": "8-14", "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/common/common.h", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cpu/ggml-cpu.c", "https://github.com/JigSawPT/moe-autopilot", "https://www.phoronix.com/review/ryzen-9950x-amd-pstate", "https://docs.kernel.org/admin-guide/pm/amd-pstate.html" ], "next": "Run the --poll, governor and idle-state A/B on dual R9700s with Flash-class experts in DDR5 and post the measured numbers." }, { "id": "2026-10-29-r9700-black-friday-baseline", "date": "2026-10-29T02:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 before Black Friday: $1,749.99 cheapest today, and $1,699.99 would only match Oct 26", "summary": "Cheapest new in-stock Radeon AI PRO R9700 on Oct 29 is $1,749.99 (Gigabyte AI TOP, Micro Center in-store), 34.7% over the $1,299 MSRP; most other models are $1,999.99. None of the six US models showed an in-stock Newegg offer. The lowest price seen since Sep 29 is $1,599.99, so a Black Friday price of $1,699.99 would just match Oct 26, not beat it.", "details": "Prices from vga.live's public card pages (schema.org offers), before tax, read 2026-10-29 02:08 CT, compared with our snapshots from Oct 26 and Oct 27. PowerColor's Amazon listing fell from $2,680.81 (Oct 26) to $1,943.99. Newegg hasn't announced 2026 Black Friday dates (its landing page says deals are coming soon); in 2025 its main sale ran Nov 20-29 with daily doorbusters through Dec 2. Our tracking covers only a few days so far, so treat 'typical' as provisional. When a store shows a 'was' price, check that it held for a real stretch; the FTC's pricing guides say a former price must be one the item was actually offered at for a reasonably substantial period.", "metrics": [ { "label": "Cheapest new in-stock R9700 (Oct 29)", "value": 1749.99, "unit": "USD" }, { "label": "Over MSRP", "value": 34.7, "unit": "%" }, { "label": "Lowest seen since Sep 29", "value": 1599.99, "unit": "USD" }, { "label": "Two cards at today's cheapest", "value": 3499.98, "unit": "USD" } ], "sources": [ "https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation", "https://dealsinsiders.com/newegg-black-friday-deals/", "https://www.newegg.com/BLACK-FRIDAY-DEALS-Be-the-First-to-Save-Big/EventSaleStore/ID-1132", "https://www.ecfr.gov/current/title-16/chapter-I/subchapter-B/part-233" ], "next": "Keep checking prices daily through Cyber Monday so the 'typical' and 'lowest' lines rest on weeks of data." }, { "id": "2026-10-29-01-35b-np-vs-replicas", "date": "2026-10-29T01:10:00-05:00", "project": "idea-lab", "kind": "progress", "title": "One GPU with parallel slots vs two 35B copies: estimates vs measured soak", "summary": "For Qwen3.6-35B UD-Q5_K_XL at short context, one Radeon AI PRO R9700 is measured at 102.39 tok/s. Layer-split across two cards is measured slower at 82.78 tok/s. The dual-copy page reports about 100 tok/s per copy and 172 tok/s aggregate with two concurrent chats. Applying a cited Strix Halo parallel-slot scaling shape (not an R9700 measurement) gives an estimate of about 167 tok/s aggregate on one card with -np 2 (~84 per stream). Two full-speed copies stay the honest reason to buy a second card; splitting one 35B is not.", "details": "Strix Halo cited shape for Qwen3.6-35B-A3B UD-Q4_K_M Vulkan: 58.8 / 96.1 / 138.8 aggregate tok/s at -np 1/2/4. That efficiency curve was applied to the measured single-card R9700 UD-Q5_K_XL short row to label the -np 2 estimate. Dual-copy soak figures are from the published /qwen36-dual page. Q3 and below were not considered.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "UD-Q5_K_XL (house); Q4_K_M / Q5_K_M / Q6_K in model table", "engine": "llama.cpp / llama-server", "gpus": "1\u20132\u00d7 Radeon AI PRO R9700 (32 GB)", "context": "short (plus 32K / 128K / 256K rows in artifact)", "flags": "one-card -np N (estimate); two pinned servers + least-conn (page soak); layer-split (measured)" }, "metrics": [ { "label": "1\u00d7 R9700 short UD-Q5_K_XL", "value": 102.39, "unit": "tok/s" }, { "label": "2\u00d7 layer-split short UD-Q5_K_XL", "value": 82.78, "unit": "tok/s" }, { "label": "2\u00d7 replicas aggregate at 2 concurrent (page soak)", "value": 172, "unit": "tok/s" }, { "label": "1\u00d7 R9700 -np 2 aggregate (ESTIMATE from Strix Halo shape)", "value": 167.3, "unit": "tok/s" } ], "sources": [ "https://amdai.net/compare/one-vs-two-gpu/", "https://amdai.net/qwen36-dual/", "https://amdai.net/concurrency-ab/" ], "next": "Measure -np 2 and -np 4 on one R9700 with the same short UD-Q5 pack; publish measured per-stream and aggregate; add a buy box to /qwen36-dual that sells a second copy, not a split." }, { "id": "2026-10-29-second-r9700-worth-it", "date": "2026-10-29T00:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Is a second R9700 worth it? Not for the 35B: one card decodes faster in 12 of 16 measured rows; buy it for Flash-class models or two models at once", "summary": "On our 16-row sweep of Qwen3.6-35B-A3B (llama.cpp Vulkan, f16 KV, Q4_K_M to Q6_K, short to 256K context), one R9700 decoded faster than a split across two cards in 12 rows, by 2.4 to 24.4 tok/s. The split only wins decode on Q6_K past short context (+3.5 to +4.4 tok/s, because Q6_K spills into DDR5 on one card), and UD-Q5_K_XL on one card beats that Q6_K split at every context. The split does read long prompts faster: +14% at 128K and +29% at 256K on UD-Q5_K_XL.", "details": "Where a second card does pay: Flash-class MoE models that don't fit one 32 GB card at 4-bit or more (two R9700s plus DDR5 experts on Strata measured 52 tok/s story / 65 code at UD-Q5_K_XL; the Strata maintainer's tables show the second card moving UD-Q5 from 57 to 60 story and 83 to 94 code with MTP on, so most of the win is the RAM, the card adds about 5-13%), running two models side by side with one per card, and going past 262K context on the 35B at f16 KV (our KV math). The 35B sweep rows date from 2026-10-12 and are pending a re-verify on stable DDR5.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / Q5_K_M / UD-Q5_K_XL / Q6_K", "engine": "llama.cpp Vulkan", "gpus": "1x vs 2x Radeon AI PRO R9700 (layer split)", "context": "short / 32K / 128K / 256K", "flags": "f16 KV" }, "metrics": [ { "label": "Rows where one card decodes faster", "value": "12 of 16", "unit": "" }, { "label": "UD-Q5_K_XL short decode, 1 card vs split", "value": "102.4 / 82.8", "unit": "tok/s" }, { "label": "Q6_K 128K decode, 1 card vs split", "value": "54.5 / 58.0", "unit": "tok/s" }, { "label": "UD-Q5_K_XL 128K decode, 1 card", "value": 66.4, "unit": "tok/s" }, { "label": "UD-Q5_K_XL 256K prefill, 1 card vs split", "value": "519 / 672", "unit": "tok/s" }, { "label": "Flash UD-Q5 code decode, 1 vs 2 cards (Strata, cited, MTP)", "value": "83 / 94", "unit": "tok/s" } ], "sources": [ "https://amdai.net/compare/one-vs-two-gpu/", "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md" ], "next": "Rerun the 35B sweep on stable DDR5 to drop the re-verify flag, and turn the one-vs-two page's buy box into a verdict that only suggests a second card when it helps." }, { "id": "2026-10-28-r9700-compute-profile-index", "date": "2026-10-28T23:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "R9700 tuning: the COMPUTE power profile is index 5 on RDNA4, and it turns off GPU deep sleep", "summary": "On RDNA4 cards such as the Radeon AI PRO R9700 (SMU 14.0.2), the amdgpu pp_power_profile_mode list is 0 BOOTUP_DEFAULT, 1 3D_FULL_SCREEN, 2 POWER_SAVING, 3 VIDEO, 4 VR, 5 COMPUTE, 6 CUSTOM, 7 WINDOW_3D. Guides that say to write 4 for compute select VR on these cards. Selecting COMPUTE also disables GFX deep sleep in the SMU 14 driver code, which is the setting that matters when the GPU sits idle between steps, as in DDR5 expert-offload decode.", "details": "A profile only applies after power_dpm_force_performance_level is set to manual; with no clock mask written, all DPM levels stay enabled, so this is not a clock lock. Safest method: grep the row named COMPUTE in pp_power_profile_mode and write that index rather than a number copied from another GPU family. One published dual-R9700 report saw +6% decode (102 to 110 tok/s) from a 330 W limit plus COMPUTE with the model fully in VRAM; that is a single self-report and was not measured with expert offload. Revert by writing 0 to the profile and auto to the performance level.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201, RDNA4)", "engine": "amdgpu kernel driver (SMU 14.0.2)" }, "metrics": [ { "label": "COMPUTE profile index on RDNA4", "value": 5, "unit": "" }, { "label": "Reported dual-R9700 decode gain, 330 W + COMPUTE (in VRAM, self-report)", "value": 6, "unit": "%" } ], "sources": [ "https://github.com/torvalds/linux/blob/2b414a95/drivers/gpu/drm/amd/pm/swsmu/smu14/smu_v14_0_2_ppt.c", "https://upd.dev/torvalds/linux/commit/f67644b219d458d4b314e78f2304b73985390311.patch", "https://github.com/torvalds/linux/blob/master/drivers/gpu/drm/amd/pm/amdgpu_pm.c", "https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md" ], "next": "Measure at 100 Hz whether the cards drop memory clock inside each token during Flash UD-Q5 offload decode, then A/B the COMPUTE profile against stock." }, { "id": "2026-10-28-cpu-cooler-for-dual-r9700", "date": "2026-10-28T22:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "CPU cooler for an R9700 build: Ryzen 9000X has none in the box, and a 360 AIO in the front or bottom of the case feeds warm air to the cards", "summary": "AMD ships the Ryzen 5 9600X, Ryzen 9 9900X and Ryzen 9 9950X without a cooler. AMD recommends a premium air cooler for the 65 W 9600X and a liquid cooler for the 120 W 9900X and 170 W 9950X for best boost. For a local-AI box with blower R9700s, where the radiator goes matters: the cards pull their air from the case's front and bottom intake, so a 360 radiator should go on top as exhaust, and a case that only takes one in front or bottom is better paired with a large dual-tower air cooler.", "details": "Case limits checked on maker pages: Fractal Torrent takes a 360 radiator only in front or bottom and air coolers up to 188 mm; Fractal Meshify 2 takes a 360 on top or front and air coolers up to 185 mm; Phanteks Enthoo Pro 2 Server Edition takes radiators on top, front or side and air coolers up to 195 mm. Thermalright Peerless Assassin 120 SE (155 mm) and Noctua NH-D15 G2 (168 mm) fit all three. ARCTIC's Liquid Freezer III Pro 360 needs 63 mm of radiator clearance, and ARCTIC warns that some boards' oversized M.2_1 heatsinks clash with it. With MoE expert offload, token generation is limited by memory and PCIe traffic rather than CPU compute, so AMD's liquid recommendation is about peak boost, not decode speed; we have not measured cooler choice on our own cards.", "setup": "Spec review: amd.com product pages for 9600X/9900X/9950X; Fractal, Phanteks, Thermalright, Noctua and ARCTIC product pages; published llama.cpp MoE offload guides", "metrics": [ { "label": "AMD default TDP, 9600X / 9900X / 9950X", "value": "65 / 120 / 170", "unit": "W" }, { "label": "Air cooler height limit, Torrent / Meshify 2 / Enthoo Pro 2 Server", "value": "188 / 185 / 195", "unit": "mm" }, { "label": "Cooler height, Peerless Assassin 120 SE / NH-D15 G2", "value": "155 / 168", "unit": "mm" } ], "sources": [ "https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-5-9600x.html", "https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9900x.html", "https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9950x.html", "https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/", "https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/", "https://phanteks.com/product/enthoo-pro-2-server-edition-tg/", "https://www.thermalright.com/product/peerless-assassin-120-se/", "https://noctua.at/en/nh-d15-g2/specification", "https://www.arctic.de/en/Liquid-Freezer-III-Pro-360/ACFRE00180A", "https://gist.github.com/DocShotgun/a02a4c0c0a57e43ff4f038b46ca66ae0" ], "next": "Add a CPU-cooler line to the amdai.net builder parts list." }, { "id": "2026-10-28-case-for-two-r9700s", "date": "2026-10-28T21:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Case for two R9700s: a standard 7-slot ATX case is enough on 4-slot-spacing x8/x8 boards, and case fans should follow the GPUs, not the CPU", "summary": "Every R9700 board-partner card is a 2-slot blower no longer than 271 mm. On the ASRock X870 Taichi Creator and Gigabyte B850 AI TOP, owners report 4-slot spacing between the two CPU x8/x8 slots, so the bottom card ends by expansion slot 7 and leaves a 2-slot gap above it. That means any 7-slot ATX case with at least 271 mm of card length fits two cards; no 8-slot case is needed unless you plan a third card or a riser. Blowers push their heat out the back, so the case's job is fresh intake air for the card area.", "details": "In a published 4x R9700 build, case fans on the stock CPU-following curve let the card area reach its 68 C warning and pause work; full-speed case fans cooled the card area from 67 C to 57 C but changed GPU junction temperature by only 1-4 C, because each card's own blower cools its GPU. Tuning the case fans to follow card power and temperature kept the same GPU temperatures at 26% lower fan speed. Owners stacking R9700 blowers with no gap report it works, but one owner found the ASUS Turbo's fan is not recessed and may choke without a slot gap, so use matching cards. Slot spacing for the ASUS ProArt X870E/B850 boards and the ASRock X870E Taichi was not verified here; check the manual layout or choose an 8+ slot case. Checked case specs: Fractal Torrent (7 slots, 2x 180 mm front + 3x 140 mm bottom fans included, PSU up to 230 mm), Fractal Meshify 2 (7 slots, E-ATX, PSU up to 250 mm), Phanteks Enthoo Pro 2 Server Edition (11 slots, PSU up to 240 mm). Temperatures are published by others, not measured on our cards.", "setup": "Spec review: R9700 board-partner spec sheets, ASRock/Gigabyte board pages and owner reports, Fractal and Phanteks case pages, one published 4x R9700 build guide", "metrics": [ { "label": "Longest R9700 board-partner card", "value": "271", "unit": "mm" }, { "label": "Slot spacing, X870 Taichi Creator / B850 AI TOP (owner reports)", "value": "4", "unit": "slots" }, { "label": "Card-area temp, CPU-following vs full-speed case fans (4x R9700, published)", "value": "67 vs 57", "unit": "C" }, { "label": "GPU junction change from full-speed case fans (published)", "value": "1-4", "unit": "C" } ], "sources": [ "https://forum.level1techs.com/t/asrock-x870-taichi-creator-dual-gpu/243897", "https://pg.asrock.com/general/news.us.asp?id=5643", "https://www.reddit.com/r/LocalLLaMA/comments/1nqi0di/are_there_any_consumer_mobos_that_can_fit_2x/", "https://nicholasreese.com/guides/quad-r9700/quad-r9700-build-guide.md", "https://www.reddit.com/r/LocalLLM/comments/1vyaesz/blower_r9700_cards_stacked_with_very_little_gap/", "https://forum.level1techs.com/t/radeon-pro-ai-r9700-fan-noise-which-model-to-buy/247552?page=2", "https://www.fractal-design.com/products/cases/torrent/torrent/black-solid/", "https://www.fractal-design.com/products/cases/meshify/meshify-2/black-tg-dark-tint/", "https://phanteks.com/product/enthoo-pro-2-server-edition-tg/" ], "next": "Add a case step to the amdai.net builder parts list, and verify slot spacing for the ASUS ProArt boards from their manuals." }, { "id": "2026-10-28-nvme-for-local-models", "date": "2026-10-28T20:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "How much NVMe a Radeon local-AI box needs: 2 TB holds every Q4-Q6 quant of three model families, and Gen5 barely speeds up loading", "summary": "Every Q4 to Q6 quant of Qwen3.6-35B-A3B (20.6 to 29.7 GiB), Qwen3.8-Flash-Next (103.7 to 157.6 GiB) and GLM-5.3-Flash (186.0 to 271.8 GiB) adds up to about 1,587 GiB once you keep one extra file of room for the next download and 150 GiB for the OS. That fits a 2 TB drive (1,863 GiB) with 10% left free; a fourth Flash-class family such as MiniMax-M3.1-Flash (UD-Q4 232 GiB) needs 4 TB. On llama.cpp's default memory-mapped loader, loading is usually limited by page faults and one CPU core rather than the drive, so a Gen5 SSD helps less than its doubled rating suggests.", "details": "A llama.cpp user saw about 2.4 GB/s buffered loads from NVMe rated over 7 GB/s, with one core at 100%. llama.cpp's direct I/O loader reads at disk speed: its PR reports GPT-OSS-120B loading in about 10.5 s versus about 110 s on the first mmap load. One third-party test found a 40.5 GB Q4_K_M file loading in 15.2 s on a Gen4 990 PRO versus 13.8 s on a Gen5 T705 (about 9%). So capacity and a Gen4 TLC drive with DRAM is the better buy for most builds. Sizes are summed GGUF shard bytes from Hugging Face; load figures are published, not measured on our cards.", "setup": "Spec and file-size review: Samsung 990 PRO 2/4 TB, WD_BLACK SN850X 4/8 TB, Samsung 9100 PRO 4/8 TB; unsloth GGUF repos", "metrics": [ { "label": "All Q4-Q6 quants of 3 families + headroom", "value": "1587", "unit": "GiB" }, { "label": "Usable space on a 2 TB drive", "value": "1863", "unit": "GiB" }, { "label": "Gen4 vs Gen5 cold load, 40.5 GB Q4_K_M (third-party)", "value": "15.2 vs 13.8", "unit": "s" }, { "label": "GPT-OSS-120B load, mmap vs direct I/O (llama.cpp PR)", "value": "110 vs 10.5", "unit": "s" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "https://github.com/ggml-org/llama.cpp/pull/18012", "https://github.com/ggml-org/llama.cpp/discussions/12223", "https://craftrigs.com/guides/nvme-speed-llm-load-times-benchmarks/", "https://download.semiconductor.samsung.com/resources/data-sheet/samsung_nvme_ssd_990_pro_datasheet_rev.2.0.pdf", "https://download.semiconductor.samsung.com/resources/data-sheet/Samsung_NVMe_SSD_9100_PRO_Datasheet_Rev.2.0.pdf", "https://documents.sandisk.com/content/dam/asset-library/en_us/assets/public/sandisk/product/internal-drives/wd-black-ssd/data-sheet-wd-black-sn850x-nvme-ssd.pdf" ], "next": "Time a cold Flash-Next load on R9700s with the mmap loader versus direct I/O to get our own number." }, { "id": "2026-10-28-cpu-pick-dual-r9700", "date": "2026-10-28T19:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Which Ryzen for one or two R9700s: CPU barely matters in VRAM, two CCDs matter for DDR5 offload, and 8000G APUs can't do dual GPU", "summary": "Every Ryzen 9000 desktop chip has 24 usable PCIe 5.0 lanes, so a 9600X can drive two R9700s at x8/x8 on the right board when the model fits in GPU memory. For DDR5 expert offload, one Zen 5 CCD reads from memory at about 64 GB/s at the default 2000 MHz fabric clock, below dual-channel DDR5-6000's 96 GB/s theoretical, so a two-CCD 9900X or 9950X is the safer pick. Ryzen 8700G/8600G have only 8 graphics lanes (second slot disabled) and 8500G only 4.", "details": "With four DIMMs at AMD's official 3600 MT/s (57.6 GB/s theoretical), DRAM sits under the single-CCD cap, so the second CCD only pays off if the memory runs faster. Figures are theoretical caps from published fabric widths, not measurements on our cards.", "setup": "Spec review: Ryzen 9600X, 9700X, 9900X, 9950X, 8700G, 8600G, 8500G", "metrics": [ { "label": "Usable CPU PCIe 5.0 lanes, Ryzen 9000", "value": "24", "unit": "lanes" }, { "label": "Fabric read cap per CCD at FCLK 2000", "value": "64", "unit": "GB/s" }, { "label": "Dual-channel DDR5-6000 theoretical", "value": "96", "unit": "GB/s" }, { "label": "Ryzen 8700G graphics lanes (PCIe 4.0)", "value": "8", "unit": "lanes" } ], "sources": [ "https://www.amd.com/en/products/processors/desktops/ryzen/9000-series/amd-ryzen-9-9900x.html", "https://www.amd.com/content/dam/amd/en/documents/partner-hub/ryzen/amd-ryzen-8000g-quick-reference-guide-competitive.pdf", "https://www.supermicro.com/en/support/faqs/faq.php?faq=42600", "https://old.chipsandcheese.com/2024/11/24/pushing-amds-infinity-fabric-to-its-limits/" ], "next": "Measure Flash offload decode with CPU threads on one CCD vs both to put a real number on the single-CCD loss." }, { "id": "2026-10-28-r9700-psu-sizing", "date": "2026-10-28T18:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "PSU for R9700 builds: 850 W with one 12V-2x6 for one card, 1200 W+ with two native 12V-2x6 for two", "summary": "The R9700 draws almost all its power through a single 12V-2x6 connector, so each card wants its own native 16-pin cable. Many 1000 W PSUs ship only one (Corsair's RM1000x 2024 is an example), and two cards plus a Ryzen 9 9950X can reach roughly 1,100 W if both cards hit the ~400 W measured under full compute. For two cards, or one card you plan to double later, buy a 1200 W+ ATX 3.1 unit with two native 12V-2x6 cables once.", "details": "Sizing (estimate): rated board power 300 W per card (AMD), ~390-400 W measured at the card under full compute (Igor's Lab), 9950X 200 W PPT, rest of system ~100 W. One card about 600-700 W, so an 850 W unit with one native 12V-2x6 fits (AMD lists 750 W minimum). Two cards about 900-1,100 W. Units checked on maker or listing pages 2026-10-28 with two native 12V-2x6 cables: Thermaltake Toughpower GT 1200W (Gold), be quiet! Straight Power 12 1200W (Platinum; its two 12V-2x6 share an 800 W combined limit), Thermaltake Toughpower TF3 1300W (Titanium), Corsair HX1500i 2025 (Platinum; rated 1500 W at 115-240 V but 1200 W at 100-115 V). Decode-heavy llama.cpp layer split draws far less (Puget measured 339-353 W for both cards together), so the peak case is batched or prefill-heavy work. No prices listed; check the store.", "setup": { "platform": "AM5, Ryzen 9 9950X", "gpus": "1-2x Radeon AI PRO R9700" }, "metrics": [ { "label": "Estimated peak draw, 1 card + 9950X", "value": 700, "unit": "W" }, { "label": "Estimated peak draw, 2 cards + 9950X", "value": 1100, "unit": "W" }, { "label": "Power drawn from the PCIe slot", "value": 1.5, "unit": "W" } ], "sources": [ "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html", "https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/", "https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/5/", "https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/", "https://www.corsair.com/us/en/explorer/diy-builder/power-supply-units/corsair-rmx-series/", "https://www.newegg.com/thermaltake-atx-fully-modular-1200-w-80-plus-gold-certified-power-supply-black-ps-tpt-1200fnfagu-3/p/N82E16817153476", "https://aphnetworks.com/reports/be-quiet-straight-power-12-1200w", "https://www.newegg.com/p/1HU-0014-006X8", "https://www.corsair.com/us/en/explorer/diy-builder/power-supply-units/corsair-hxi-series-2025/" ], "next": "Add a power-supply step to the builder that sizes by card count and the 'second GPU later?' answer." }, { "id": "2026-10-28-ddr5-buy-now-or-later", "date": "2026-10-28T17:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "DDR5 for an R9700 build: 32 GB if you stay on the 35B, 192-256 GB up front if you plan Flash-class offload", "summary": "On AM5, 192 GB and 256 GB only come as four-DIMM kits (4x48 or 4x64 GB), and memory makers say not to mix kits, so a 2x16 GB kit can't be topped up to offload size later; it gets replaced. Ryzen 9000's official speed is DDR5-5600 with two DIMMs but DDR5-3600 with four, so EXPO 6000 on four DIMMs is an overclock to stability-test.", "details": "Kits checked on maker pages 2026-10-28: 32 GB 2x16 F5-6000J3038F16GX2-FX5 (EXPO 6000 CL30-38-38-96 1.35V); 128 GB 2x64 KF556C36BBEK2-128 (EXPO 5600 CL36); 192 GB 4x48 F5-6000J2836G48GX4-FX5 (EXPO 6000 CL28-36-36-96 1.40V); 256 GB 4x64 F5-6000J3644D64GX4-TZ5NR (EXPO 6000 CL36-44-44-96 1.25V); 256 GB 4x64 F5-6000J3444F64GX4-TZ5NR (EXPO 6000 CL34-44-44-96 1.35V). The 256 GB G.SKILL kits need a 2025-or-newer BIOS with 64 GB-module support. A 2x64 kit gives 128 GB while staying at the two-DIMM official speed. RAM speed matters for offload: on our two-card Flash Q6 run, raising only the DIMM clock moved llama.cpp from 15.9 to 18.5 tok/s at 8K. No prices listed; check the store.", "setup": { "platform": "AM5, Ryzen 9000, DDR5 UDIMM", "gpus": "1-2x Radeon AI PRO R9700" }, "metrics": [ { "label": "Ryzen 9000 official speed, 2 DIMMs", "value": 5600, "unit": "MT/s" }, { "label": "Ryzen 9000 official speed, 4 DIMMs", "value": 3600, "unit": "MT/s" }, { "label": "Largest consumer UDIMM kit found", "value": 256, "unit": "GB" } ], "sources": [ "https://www.gskill.com/specification/165/396/1673491242/F5-6000J3038F16GX2-FX5-Specification", "https://www.kingston.com/en/memory/validations/amd-expo-self-certifications", "https://www.amazon.com/dp/B0FHN24M8C", "https://www.gskill.com/product/165/390/1750238051/F5-6000J3644D64GX4-TZ5NR", "https://www.gskill.com/qvl/165/390/1750730893/F5-6000J3444F64GX4-TZ5NR-QVL", "https://en.wikipedia.org/wiki/Template:AMD_Ryzen_9000_Series", "https://www.tomshardware.com/pc-components/ram/g-skill-debuts-worlds-first-256gb-64gbx4-ddr5-6000-cl32-udimm-memory-kit-with-expo-support", "https://www.gskill.com/qvl/165/396/1673491242/F5-6000J3038F16GX2-FX5-QVL", "https://amdai.net/flash-ram-offload/" ], "next": "Add a 'larger models later?' RAM step to the builder, and measure four-DIMM 256 GB at JEDEC vs EXPO on Flash offload." }, { "id": "2026-10-28-x8x8-boards", "date": "2026-10-28T15:30:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Which AM5 motherboards really run two R9700s at x8/x8 (and fit 256 GB of DDR5)", "summary": "From manufacturer spec pages: the ASRock X870 Taichi Creator, ASRock X870E Taichi, Gigabyte B850 AI TOP and ASUS ProArt B850-Creator WiFi Neo all wire two x16 slots to the CPU for PCIe 5.0 x8/x8 and list 256 GB max over 4 DIMMs. So a dual-GPU board no longer needs an X870E chipset. Some pricey boards are not dual-GPU boards: the ROG Strix X870E-E has one CPU slot plus a chipset x4.", "details": "Traps: on the ProArt X870E-Creator, using M.2_2 drops GPU slot 2 to x4. The Crosshair X870E Apex has 2 DIMMs (128 GB max), and M.2_2 plus M.2_3 disables slot 2. The ProArt B650-Creator does x8/x8 but only at PCIe 4.0 and lists 192 GB max, so no 4x64 GB kit. On the X870 Taichi Creator, M.2 lane sharing hits USB4 and the chipset slot, never the GPU slots. Budget boards like the ASRock B850 Pro RS have one CPU x16 slot; the second slot is chipset PCIe 4.0 x4 and turns off when M2_3 is used, so a second GPU later means a slow link or a new board. Ryzen 9000 has 28 PCIe 5.0 lanes, so x8/x8 is the AM5 maximum for two GPUs.", "setup": { "gpus": "2x Radeon AI PRO R9700", "platform": "AM5, Ryzen 9000" }, "metrics": [ { "label": "Boards checked", "value": 9, "unit": "" }, { "label": "Verified x8/x8 Gen5 with 256 GB max", "value": 4, "unit": "" } ], "sources": [ "https://asrock.com/MB/AMD/X870%20Taichi%20Creator/index.asp", "https://www.asrock.com/MB/AMD/X870E%20Taichi/", "https://www.gigabyte.com/Motherboard/B850-AI-TOP/sp", "https://www.asus.com/ie/motherboards-components/motherboards/proart/proart-b850-creator-wifi-neo/techspec/", "https://www.asus.com/us/motherboards-components/motherboards/proart/proart-b650-creator/techspec/", "https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x870e-creator-wifi/techspec/", "https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM5/ROG_CROSSHAIR_X870E_APEX/E25780_ROG_CROSSHAIR_X870E_APEX_EM_WEB.pdf", "https://rog.asus.com/us/motherboards/rog-strix/rog-strix-x870e-e-gaming-wifi7-neo/wtb/", "https://asrock.com/MB/AMD/B850%20Pro%20RS/index.us.asp" ], "next": "Add a dual x8/x8 board pick to the budget builds and the builder, with a cheaper single-slot option clearly labeled as limiting a second GPU." }, { "id": "2026-10-28-third-r9700-link", "date": "2026-10-28T11:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Adding a 3rd R9700 over x4, OCuLink or USB4: decode barely cares, but keep a full-speed card as device 0", "summary": "On a typical X870 board the third card lands on a chipset Gen3/Gen4 x4 slot, an M.2-to-OCuLink adapter, or USB4. Estimate: with llama.cpp layer split the per-token hop is about 40 KiB, so decode loses about 1% or less on any of them, and load takes seconds longer. The catch: when Q5/Q6 Flash-Next still keeps expert layers in DDR5, llama.cpp copies those experts over the first GPU's link on every prompt batch, so a slow-link card as device 0 cuts the prefill ceiling about 6x.", "details": "Example board (ASRock X870 Taichi Creator): PCIE3 is chipset PCIe 3.0 x4 (x2 if M2_4 is used); M2_2 is CPU Gen5 x4 but populating it drops the USB4 ports to x2; every chipset device shares one PCIe 4.0 x4 uplink. Published link speeds: OCuLink Gen4 x4 5.7-6.7 GB/s, ASM2464 USB4 enclosures about 3.8 GB/s (1.2-1.8 on some hosts). ROCm needs PCIe atomics through every bridge; one R9700 USB4 eGPU runs ROCm 7.1, but tunnels are reported to drop atomics on other hosts, so check AtomicOpsCtl with lspci or use Vulkan. Modeled for Qwen3.8-Flash-Next on 3 cards: UD-Q4_K_XL fits fully in VRAM (no expert copies); UD-Q5_K_XL leaves 8 expert layers (15.3 GiB) in DDR5, giving an estimated prefill ceiling at ubatch 2048 of about 2,870 tok/s via a CPU Gen5 x8 device 0 vs about 475 via USB4 or 400 via chipset Gen3 x4. All numbers except the link speeds are estimates, not measurements.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (modeled)", "gpus": "3x Radeon AI PRO R9700 (planned)", "flags": "-sm layer, experts in DDR5 for Q5/Q6" }, "metrics": [ { "label": "Estimate: decode tax, layer split, <=60 us per hop", "value": "<=1.1", "unit": "%" }, { "label": "Estimate: 3rd card load floor (27.6 GiB), OCuLink vs USB4", "value": "4.8 / 7.8", "unit": "s" }, { "label": "Estimate: Q5 prefill ceiling ub2048, CPU-x8 device 0", "value": 2870, "unit": "tok/s" }, { "label": "Estimate: Q5 prefill ceiling ub2048, USB4 device 0", "value": 475, "unit": "tok/s" }, { "label": "Published: OCuLink Gen4 x4 host-to-device", "value": "5.7-6.7", "unit": "GB/s" }, { "label": "Published: ASM2464 USB4 host-to-device", "value": 3.8, "unit": "GB/s" } ], "sources": [ "https://www.asrock.com/MB/AMD/X870%20Taichi%20Creator/index.asp", "https://github.com/lemonade-sdk/lemonade/issues/2609", "https://rocm.docs.amd.com/projects/install-on-linux/en/docs-7.2.4/reference/system-requirements.html", "https://www.techpowerup.com/341632/external-gpu-connection-battle-oculink-beats-thunderbolt-5-by-about-16", "https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md", "https://ai-hardware-zukan.com/llm-gpu-vram-oculink-local/" ], "next": "Map slots, atomics and device order read-only on the test rig, then measure 3-card device order A/B once a third card is installed." }, { "id": "2026-10-28-35b-decode-gap-graphics-queue", "date": "2026-10-28T10:00:00-05:00", "project": "idea-lab", "kind": "result", "title": "Our single-R9700 35B decode (116.5 tok/s) trails public R9700 results (~165) by 42%; llama.cpp's Vulkan compute-queue default is the likely free fix", "summary": "Measured: our Qwen3.6-35B-A3B Q4_K_M lane does 116.5 tok/s on one R9700 (llama.cpp Vulkan, Aug build). Published: 165.2 tg128 from the llama.cpp Vulkan maintainer on an R9700 and 164.6 in a community R9700 guide on the same Mesa 25.2.8 / Ubuntu 24.04. Sourced: llama.cpp's Vulkan backend avoids the graphics queue by default; using it measured +10% (RX 9070 XT, 30B-A3B), about +15% (R9700 35B, community guide, via RADV_DEBUG=nocompute) and +6.9% (RX 7900 XTX headless, identical output). Estimate, not measured: the queue alone would bring our Q4_K_M to about 122-134 tok/s.", "details": "The graphics queue was the default on AMD for a few days in March (PR #20551) and was reverted to opt-in (PR #20599) because it lagged desktops on iGPUs and small GPUs; the maintainer called it only really safe headless, so use it on a card that drives no monitor. A request to auto-enable it on RADV (#22457) was closed. Upgrade trap: llama.cpp b11594 to b11715 cut decode on multi-GPU RADV hosts, dual R9700 layer split from 42 to 16 tok/s (#30552), fixed in b11716 (#30468); skip that range on any 2-card box. Of 98 Vulkan backend commits since our build, the measured RDNA4 gains are prefill-side (int8 coopmat #27952, concat #30591 +15.8% pp2048 on R9700 with tg about flat). Part of the remaining gap is method: public numbers are llama-bench tg128 at depth 0, ours is a server run at 262K allocated context. A three-arm interleaved A/B (default, GGML_VK_ALLOW_GRAPHICS_QUEUE=1, RADV_DEBUG=nocompute) is queued for our cards.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / UD-Q5_K_XL", "engine": "llama.cpp Vulkan (RADV)", "engine_version": "b10603 (ours); public results per source", "gpus": "1x Radeon AI PRO R9700", "context": "262144 (ours); tg128 depth 0 (published)", "flags": "-fa on, f16 KV, -b 2048 -ub 512, no queue env (ours)" }, "metrics": [ { "label": "Measured: our Q4_K_M decode, 1x R9700", "value": 116.5, "unit": "tok/s" }, { "label": "Measured: our UD-Q5_K_XL decode, 1x R9700", "value": 100.8, "unit": "tok/s" }, { "label": "Published: Q4_K_M tg128, 1x R9700 (llama.cpp PR #25051)", "value": 165.2, "unit": "tok/s" }, { "label": "Published: Q4_K tg128, 1x R9700, nocompute (community guide)", "value": 164.6, "unit": "tok/s" }, { "label": "Published: graphics-queue decode gain, RX 9070 XT 30B-A3B", "value": 10.0, "unit": "%" }, { "label": "Published: graphics-queue decode gain, RX 7900 XTX headless", "value": 6.9, "unit": "%" }, { "label": "Estimate: our Q4_K_M with graphics queue, low", "value": 122, "unit": "tok/s" }, { "label": "Estimate: our Q4_K_M with graphics queue, high", "value": 134, "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/20551", "https://github.com/ggml-org/llama.cpp/pull/20599", "https://github.com/ggml-org/llama.cpp/issues/22457#issuecomment-6205365031", "https://github.com/ggml-org/llama.cpp/pull/25051#issuecomment-6382846940", "https://github.com/ggml-org/llama.cpp/issues/30552", "https://github.com/ggml-org/llama.cpp/pull/30468", "https://github.com/aguaishuo/AMD-Radeon-AI-PRO-R9700-llama-cpp-Vulkan-MTP" ], "next": "Run the three-arm queue A/B on our 35B card (llama-bench tg128/tg512 at depth 0 and 32K, plus the production server at 262K), then a build at b11716 or later; adopt the variable only if it is at least 3% faster with identical greedy output." }, { "id": "2026-10-28-r9700-vs-32gb-nvidia", "date": "2026-10-28T09:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 vs RTX 5090 vs RTX PRO 4500 on Qwen3.6-35B-A3B: NVIDIA is 1.6-2.3x faster, R9700 gives ~2x the speed per dollar at today's prices", "summary": "All three 32 GB cards hold Qwen3.6-35B-A3B fully in VRAM. Measured on our R9700: 116.5 tok/s (Q4_K_M) and 100.8 (UD-Q5_K_XL). Published: RTX 5090 ~264 tok/s at Q4 (256.9-270.6), RTX PRO 4500 Blackwell 162.2 at UD-Q5_K_XL. At Oct 28 US prices (R9700 $1,799.99, 5090 $7,999.99 Amazon 7-day low, PRO 4500 $4,699.99) that is 64.7 tok/s per $1,000 for the R9700 vs 33.0 and 34.5.", "details": "The 5090 is 2.27x faster for 4.44x the price; only at the $4,299 in-store price reported mid-October would it come close on speed per dollar (61 vs 65). The PRO 4500 is 1.61x faster for 2.61x the price but uses a 200 W board (0.81 tok/s per board watt vs 0.34 for the R9700). Two R9700s (64 GB) cost $3,599.98, less than one PRO 4500. NVIDIA numbers come from other hosts and llama.cpp builds; Blackwell gains another 6-9% with GGML_CUDA_PDL=ON. Short-context single-user decode only.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / UD-Q5_K_XL", "engine": "llama.cpp", "gpus": "1x R9700 (ours) vs published RTX 5090 and RTX PRO 4500 results", "context": "short (tg128 class)" }, "metrics": [ { "label": "Measured: R9700 Q4_K_M decode", "value": 116.5, "unit": "tok/s" }, { "label": "Measured: R9700 UD-Q5_K_XL decode", "value": 100.8, "unit": "tok/s" }, { "label": "Published: RTX 5090 Q4 decode (median of 2)", "value": 263.9, "unit": "tok/s" }, { "label": "Published: RTX PRO 4500 UD-Q5_K_XL decode", "value": 162.24, "unit": "tok/s" }, { "label": "R9700 speed per $1,000 (Q4_K_M)", "value": 64.7, "unit": "tok/s" }, { "label": "RTX 5090 speed per $1,000 (Q4)", "value": 33.0, "unit": "tok/s" }, { "label": "RTX PRO 4500 speed per $1,000 (UD-Q5_K_XL)", "value": 34.5, "unit": "tok/s" } ], "sources": [ "https://localllmadvisor.com/can-run/qwen3-6-35b-a3b-on-rtx-5090-32gb.html", "https://huggingface.co/datasets/witcheer/rtx-5090-benchmarks/blob/5946aab7a46e23fc4ec4205178eab0dcb94c2ea6/reports/qwen3-6-35b-a3b-ud-q4-k-m.md", "https://www.reddit.com/r/LocalLLaMA/comments/1tkw1su/blackwell_and_pdl_performance_increase/", "https://graphiccardprices.com/gpu/rtx-5090/", "https://tech-insider.org/rtx-5090-shortage-price-surge-2026/", "https://www.nowinstock.net/computers/videocards/nvidia/rtxpro/", "https://www.newegg.com/powercolor-ai-pro-r9700-32g-b-radeon-ai-pro-r9700-32gb-graphics-card/p/N82E16814131885" ], "next": "Turn this into a comparison page and refresh the prices weekly; check why the R9700's Q5 lane uses less of its bandwidth than the Q4 lane." }, { "id": "2026-10-28-rocm-free-vram-blind-spot", "date": "2026-10-28T08:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "ROCm can't see the VRAM your desktop uses, and AMD says that's by design: what it means for llama.cpp --fit on a card that drives a monitor", "summary": "On Linux, the ROCm free-memory query (hipMemGetInfo) only counts ROCm compute allocations and pinned buffers, not VRAM held by the desktop compositor, browsers or Vulkan/OpenGL apps. A reported 16 GB card showed 50 MiB used to ROCm vs 1,480 MiB in sysfs at idle, and 58 vs 4,935 MiB with a game open. Fixes in llama.cpp, ROCr and the kernel were all closed, the last on 2026-08-17 by AMD as expected KFD behavior. llama.cpp master (2026-10-28) still trusts this number on the ROCm backend, while its Vulkan backend uses the memory-budget extension and does see other clients.", "details": "Impact: llama.cpp's default-on --fit sizes placement from the free value, so on an R9700 that also drives a display it can plan for more VRAM than exists; one report in the fix thread says this often ends in a GPU reset or full system freeze. Typical desktop sizes reported: GNOME 600 MiB+ idle, a GNOME Wayland baseline of 1,649 MiB, 1-4 GB on Windows, about 110 MiB for a 4K Linux desktop with software rendering. Known growth: GNOME's portal Save dialog adds 80-200 MiB per use, and cosmic-comp grew up to ~450 MiB/hour under heavy browser load until re-login. Repeated llama-server start/stop on ROCm also raised the idle VRAM floor (+77.7 MiB in one run). Workarounds: read the real number from /sys/class/drm/cardN/device/mem_info_vram_used or per-process drm-resident-vram in /proc//fdinfo; raise the display card's fit margin with the per-device list form of --fit-target (MiB per GPU, e.g. 2560,1024); or move the display to the CPU's integrated graphics or run headless so both R9700s are compute-only.", "setup": { "gpus": "Radeon GPUs on Linux (ROCm/HIP vs Vulkan backends)", "engine": "llama.cpp master f5222f25c1 (source read, not a local run)" }, "metrics": [ { "label": "ROCm-reported used VRAM, idle desktop, 16 GB card (published)", "value": 50, "unit": "MiB" }, { "label": "Actual used VRAM per sysfs, same idle desktop (published)", "value": 1480, "unit": "MiB" }, { "label": "Actual used VRAM with a game open, ROCm still reported 58 MiB (published)", "value": 4935, "unit": "MiB" }, { "label": "GNOME Wayland desktop baseline, 7900 XTX (published)", "value": 1649.4, "unit": "MiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/issues/24906", "https://github.com/ROCm/rocm-systems/pull/7990", "https://github.com/ROCm/amdgpu/issues/223", "https://github.com/ggml-org/llama.cpp/issues/24330", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/ggml-cuda.cu", "https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp", "https://docs.kernel.org/gpu/amdgpu/driver-misc.html", "https://docs.kernel.org/gpu/drm-usage-stats.html", "https://www.reddit.com/r/LocalLLaMA/comments/1vw970u/how_to_have_all_of_your_gpu_vram_just_for_local/", "https://www.reddit.com/r/gnome/comments/18ku5lt/warning_xdgdesktopportalgnome_has_a_huge_vram/", "https://github.com/pop-os/cosmic-comp/issues/2344" ], "next": "Measure the desktop's VRAM share on our display R9700 and what freeing it buys the dual-card Flash Q5/Q6 split (extra expert layers or context)." }, { "id": "2026-10-28-r9700-board-partner-models", "date": "2026-10-28T07:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Which R9700 to stack: six board-partner models compared, ASRock ran 9 C cooler than ASUS in the only owner A/B", "summary": "All six R9700 versions (Sapphire, ASRock Creator, ASUS Turbo, Gigabyte AI TOP, PowerColor, XFX) are 2-slot 300 W blowers with one 12V-2x6 plug and the same 2,920 MHz boost (ASUS 2,940). In the only head-to-head we found, an owner's 300 W stress test held the ASRock at 94 C junction and 3,725 RPM while the ASUS reached 103 C at 3,967 RPM. Cheapest new in stock on Oct 28 was the PowerColor at $1,799.99.", "details": "Differences that matter: Gigabyte is the only one with HDMI (3 DP + 1 HDMI 2.1b) and ships an 8-pin adapter; ASRock has 4 DP, an open-backed intake and is longest at 271 mm; XFX is listed by retailers as the smallest at 99 mm tall and 35 mm thick. The same owner felt the Gigabyte was louder than the ASUS and said the ASUS intake may choke without a slot gap. Under a two-card LLM split the owner saw one card near 95% load and the other near 50%, lighter than the stress test. Owners report capping board power to 210 W with amd-smi cuts fan noise sharply for about a 10% inference loss. Street prices were 39-54% over the $1,299 MSRP.", "setup": { "gpus": "Radeon AI PRO R9700 (six board-partner versions)", "engine": "n/a (vendor specs, owner reports, retailer prices)" }, "metrics": [ { "label": "ASRock Creator junction temp, 300 W stress (owner test)", "value": 94, "unit": "C" }, { "label": "ASUS Turbo junction temp, 300 W stress (owner test)", "value": 103, "unit": "C" }, { "label": "Cheapest new in stock (PowerColor, Oct 28)", "value": 1799.99, "unit": "USD" }, { "label": "Most other models, best new in stock (Oct 28)", "value": 1999.99, "unit": "USD" } ], "sources": [ "https://forum.level1techs.com/t/radeon-pro-ai-r9700-fan-noise-which-model-to-buy/247552", "https://www.reddit.com/r/ROCm/comments/1rcbfqf/psa_heat_and_fan_noise_tip_for_r9700_pro_owners/", "https://www.servethehome.com/asrock-creator-radeon-ai-pro-r9700-review-workstation-inference-amd-style/", "https://www.gigabyte.com/Graphics-Card/GV-R9700AI-TOP-32GD-rev-11/sp", "https://www.asus.com/motherboards-components/graphics-cards/turbo/turbo-ai-pro-r9700-32g/", "https://www.powercolor.com/product-detail258.htm", "https://cdn.sapphiretech.global/assets/documents/RADEON_AI_PRO_R9700_all_Specsheet_202607_M.pdf", "https://www.gpus.net/gpus/amd_ai_pro_r9700/xfx_radeon_ai_pro_r9700", "https://vga.live" ], "next": "Add a per-model buying section to the R9700 page and collect a second same-load temperature comparison." }, { "id": "2026-10-28-hidden-state-drafters-flash-next", "date": "2026-10-28T04:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "DFlash and EAGLE-3 drafts now run on Qwen3.8-Flash-Next in llama.cpp, but with experts in DDR5 they likely cost speed on 2x R9700", "summary": "llama.cpp has EAGLE-3, DFlash and DSpark draft models, and Flash-Next drafts stopped aborting on the first request with PR #30927 (merged Oct 22, 2026). ggml-org's Flash-Next DFlash drafts (Q4_0/Q8_0/BF16) and the published Qwen3.6-35B-A3B DFlash GGUF contain no tensor below Q4. Published MoE results are weak: 2.37 accepted tokens per step on Flash-Next (M5 Max, no baseline) and a loss on an RTX 3090 with the 35B (77.0 vs 138.9 tok/s). Our estimate for 2x R9700: 0.82-0.96x at UD-Q5 on today's -ncmoe split, turning into 1.04-1.19x (Q5) and 1.21-1.35x (Q4) only with the expert cache. Estimates, not measured here.", "details": "Why: each verified position routes to its own top-10 of 512 experts, so verifying n drafted tokens reads the union of their experts. Modeled at 3.2-3.9x the expert bytes of one token for n=4 and 5.5-7.5x for n=8 (Zipf-skewed vs independent routing; both assumptions). With experts in DDR5 that multiplies the dominant cost, so break-even at UD-Q5 needs about 2.5-2.9 accepted tokens per step at width 4. With -cmoe plus --moe-cache-mib most expert reads come from VRAM, which makes speculation pay. The Flash-Next DSpark draft reports 3.70 committed tokens per step on agent sessions vs 4.24 for the model's built-in MTP head at 7 steps (H100 TP2, SGLang only); llama.cpp loading of that checkpoint is untested. Strata's draft layer is 2-bit, so these drafts are a llama.cpp-only path for our rules. Not modeled: hybrid-state rollback cost and cache hit-rate loss from wider verify sets.", "setup": { "model": "Qwen3.8-Flash-Next (+ Qwen3.6-35B-A3B lane)", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL targets, Q8_0 DFlash draft (estimate)", "engine": "llama.cpp", "engine_version": "master at or after d9ee007 (PR #30927)", "gpus": "2x Radeon AI PRO R9700 (PCIe 5.0 x8 each)", "context": "262144", "flags": "--spec-type draft-dflash, --spec-draft-n-max 2-4; with and without -cmoe --moe-cache-mib" }, "metrics": [ { "label": "Published: Flash-Next DFlash mean accepted per step, 3 drafted (M5 Max)", "value": 2.37, "unit": "tokens" }, { "label": "Published: Qwen3.6-35B-A3B Q4 on RTX 3090, DFlash vs no spec", "value": "77.0 vs 138.9", "unit": "tok/s" }, { "label": "Published: Flash-Next DSpark vs MTP-7, agent sessions (H100)", "value": "3.70 vs 4.24", "unit": "tokens/step" }, { "label": "Estimate: expert bytes read verifying 4 drafted tokens", "value": "3.2-3.9", "unit": "x" }, { "label": "Estimate: 2x R9700 UD-Q5 DFlash speedup, today's -ncmoe split", "value": "0.82-0.96", "unit": "x" }, { "label": "Estimate: 2x R9700 UD-Q5 DFlash speedup with expert cache", "value": "1.04-1.19", "unit": "x" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/30927", "https://github.com/ggml-org/llama.cpp/pull/24593", "https://github.com/ggml-org/llama.cpp/pull/25173", "https://huggingface.co/ggml-org/Qwen3.8-Flash-Next-GGUF", "https://huggingface.co/shadowpa0327/Qwen3.8-Flash-Next-DSpark", "https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash", "https://hackmd.io/ODXuOQNzSiyUITz7g9mtBw", "https://arxiv.org/abs/2505.19645" ], "next": "Measure on 2x R9700 after the expert-cache test: no spec vs DFlash (Q8_0 draft, 2-3 drafted) vs MTP, with and without the cache, UD-Q5 first; adopt only at 10%+ with identical greedy output." }, { "id": "2026-10-28-llamacpp-moe-cache-flash-next", "date": "2026-10-28T03:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp's new MoE expert cache: up to ~1.85x decode for Qwen3.8-Flash-Next with experts in RAM, and what it should mean for 2x R9700", "summary": "Since Oct 7, 2026, llama.cpp can keep recently used CPU-resident experts in a GPU cache (--moe-cache-mib; multi-GPU since Oct 8, SLRU policy since Oct 19). Published runs show Flash-Next Q4_0 on 2x RTX 4090 going from 34.8 to 64.4 tok/s at a 95.8% hit rate, but GLM-5.3-Flash got slower when only ~10-20% of its experts fit. Our bandwidth estimate for 2x R9700 (PCIe 5.0 x8 each) with -cmoe and ~48 GiB of cache: about 1.45x at UD-Q4, 1.6x at UD-Q5 and 1.7x at UD-Q6 over today's -ncmoe split. Estimate only, not yet measured here.", "details": "How it works (llama.cpp master, read 2026-10-28): during decode, every expert of a cached layer runs on the GPU, and misses are first uploaded over PCIe; the CPU never computes a miss, so a low hit rate is worse than no cache. Only layers with all experts in host memory are cached, so use -cmoe or -ncmoe rather than a per-expert -ot mix. It refuses -sm tensor, turns off pipeline parallelism (published prefill went to 0.73-0.91x in one test and up 13-16% in another), and llama-bench has no flag for it, so measure with llama-server. Hit-rate stats print at exit with -lv 4. Estimate method: hit rates from a published 48-layer Flash-Next routing trace (62% at 7.8% of experts cached, 88% at 25%, 97% at 62.5%), cross-checked against measured 77% and 89% rates; PCIe 5.0 x8 at ~24 GB/s effective gives a break-even hit rate of about 56% against a 55 GB/s DDR5 CPU path. At UD-Q5, a real hit rate of 80% instead of the predicted 95% would make it break-even. Even at the estimated gain, llama.cpp at UD-Q5 (~31-37 tok/s est.) would stay behind Strata's measured 58 tok/s on the same two cards, so it is a better llama.cpp fallback, not a new default.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimate)", "engine": "llama.cpp", "engine_version": "master after 2026-10-19 (SLRU)", "gpus": "2x Radeon AI PRO R9700 (PCIe 5.0 x8 each)", "context": "262144", "flags": "-cmoe --moe-cache-mib ~48000, -sm layer, f16 KV" }, "metrics": [ { "label": "Published: Flash-Next Q4_0, 2x RTX 4090, no cache", "value": 34.78, "unit": "tok/s" }, { "label": "Published: same with -cmoe + cache (95.8% hit)", "value": 64.39, "unit": "tok/s" }, { "label": "Published: GLM-5.3-Flash, cache at 47% hit vs none", "value": "14.23 vs 17.75", "unit": "tok/s" }, { "label": "Estimate: 2x R9700 UD-Q5 speedup with -cmoe + cache", "value": 1.63, "unit": "x" }, { "label": "Estimate: break-even hit rate, PCIe 5.0 x8 vs DDR5", "value": 56, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/29887", "https://github.com/ggml-org/llama.cpp/pull/30112", "https://github.com/ggml-org/llama.cpp/pull/30637", "https://github.com/ggml-org/llama.cpp/discussions/24528" ], "next": "Measure on 2x R9700: HIP build, -cmoe with maximum cache vs the current -ncmoe split at UD-Q4, UD-Q5 and UD-Q6, with hit rates and prefill." }, { "id": "2026-10-28-r9700-vs-mac-studio", "date": "2026-10-28T02:40:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 vs Mac Studio (M5 Ultra / M5 Max / M3 Ultra): sourced Q4+ speeds", "summary": "On Qwen3.6-35B-A3B at 4-bit, one R9700 on llama.cpp Vulkan (165.2 tok/s tg128, public) out-decodes an M5 Ultra 80c on llama.cpp Metal (136.1) by about 1.2x, but the M5 Ultra prefills 1.56x faster (4,877 vs 3,121 pp512) and holds decode better at 64K (105.4 vs 82.4). On Flash-Next, our dual-R9700 Strata UD-Q5 (58 tok/s, no draft) beats M5 Ultra 4-bit plain decode (49.2) but trails Mac MTP drafting (81-84 tok/s on code).", "details": "Cross-host, cross-build comparison, not an A/B. Llama 2 7B Q4_0 scoreboards: M5 Ultra 179.1, R9700 Vulkan 152.7, M5 Max 119.9, M3 Ultra 92.1, M4 Max 83.1 tok/s. M5 Ultra keeps 59% of its 35B decode at 128K (84.6). M3 Ultra llama.cpp Metal on Flash-Next UD-Q4_K_XL is 26.4 tok/s plain. MLX 4-bit and GGUF Q4 are different formats; our Flash row is a higher quant (UD-Q5) with experts in DDR5. Cost estimate: dual R9700 cards-only about $62 per tok/s on Flash-Next vs roughly $113 (with MTP) to $193 (plain) for an M5 Ultra 64c 256 GB, whose price is derived from Apple's base price plus the memory upcharge. Macs still win capacity, prefill, long context and power. Speculative decoding (MTP) is the biggest Radeon lever left.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, Llama 2 7B", "quant": "Q4_0, Q4_K_M, UD-Q4_K_M, UD-Q4_K_XL, UD-Q5_K_XL, MLX 4-bit (Q4+ only)", "engine": "llama.cpp Vulkan / Metal, MLX/oMLX, Strata", "engine_version": "Mac llama.cpp rows b11896 (-fa on -ub 2048); others as cited", "gpus": "1-2x Radeon AI PRO R9700 vs Mac Studio M5 Ultra / M5 Max / M3 Ultra / M4 Max (third-party)" }, "metrics": [ { "label": "R9700 35B-A3B Q4_K_M decode (public, tg128)", "value": 165.2, "unit": "tok/s" }, { "label": "M5 Ultra 35B-A3B UD-Q4_K_M decode (llama.cpp)", "value": 136.1, "unit": "tok/s" }, { "label": "M5 Ultra 35B-A3B pp512 (llama.cpp)", "value": 4877, "unit": "tok/s" }, { "label": "R9700 35B-A3B UD-Q4_K_XL pp512 (public)", "value": 3121, "unit": "tok/s" }, { "label": "Dual R9700 Flash-Next UD-Q5 decode (Strata, measured)", "value": 58, "unit": "tok/s" }, { "label": "M5 Ultra Flash-Next 4-bit decode, plain", "value": 49.2, "unit": "tok/s" }, { "label": "M5 Ultra Flash-Next 4-bit decode, MTP (code)", "value": 84.3, "unit": "tok/s" } ], "sources": [ "https://huggingface.co/datasets/tristanbes/m5-ultra-llm-bench", "https://github.com/ggml-org/llama.cpp/discussions/4167", "https://github.com/ggml-org/llama.cpp/pull/25051", "https://github.com/ggml-org/llama.cpp/issues/26795", "https://omlx.ai", "https://www.apple.com/newsroom/" ], "next": "Run the M5 Ultra dataset's exact llama-bench method (flash attention on, ubatch 2048, depths 512 to 128K) on one R9700 for a like-for-like 35B row, and prioritize a draft/MTP path for Flash-Next on Radeon." }, { "id": "2026-10-28-tokenizer-efficiency-tok-s-across-families", "date": "2026-10-28T01:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Tokens per second isn't comparable across model families: Qwen3.x needs ~8% more tokens than GLM-5.3-Flash, MiniMax or gpt-oss for the same text (12-16% more on JSON)", "summary": "We encoded the same 256K-character mix of English prose, Python, C++, Markdown and JSON with each model's official Hugging Face tokenizer. Qwen3.6 and Qwen3.8 (35B-A3B, Flash-Next, 27B share one tokenizer) used 283 tokens per 1,000 characters; GLM-5.3-Flash used 262, MiniMax-M3.1-Flash 261, gpt-oss-20b 261 and DeepSeek-V4.1-Flash 274. So at the same tok/s, GLM, MiniMax and gpt-oss put out about 8% more text per second than Qwen, and about 12-16% more on JSON or tool-call-style output. Compare models across families by characters per second, not raw tok/s.", "details": "Chars per token: Qwen3.x 3.53, GLM-5.3-Flash 3.81, MiniMax-M3.1-Flash 3.83, gpt-oss-20b 3.83, DeepSeek-V4.1-Flash 3.65. JSON tokens per 1,000 chars: Qwen 372 vs 322-333 for the others. English prose: Qwen 1.40 tokens per word vs GLM 1.37 and MiniMax 1.33. Context: a 262,144-token window holds about 0.93M characters on Qwen3.x vs about 1.0M on GLM-5.3-Flash. Within one family (Qwen 35B vs Qwen Flash-Next) tok/s stays directly comparable. Limits: one corpus, no CJK or other languages, chat-template and thinking tokens excluded; this converts speed only and says nothing about answer quality. Corpus: Pride and Prejudice (Project Gutenberg #1342), CPython's json encoder and decoder, llama.cpp server-context.cpp and docs/build.md, and this lab log's public JSON feed.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, Qwen3.8-27B, GLM-5.3-Flash, DeepSeek-V4.1-Flash, MiniMax-M3.1-Flash, gpt-oss-20b", "quant": "n/a (tokenizer only)", "engine": "Hugging Face tokenizers (Python)", "engine_version": "official tokenizer.json from each model repo, 2026-10-28", "gpus": "none (CPU tokenization)", "context": "256,147-character corpus", "flags": "no special tokens, no chat template" }, "metrics": [ { "label": "Qwen3.x tokens per 1,000 chars", "value": 283.2, "unit": "tokens" }, { "label": "GLM-5.3-Flash tokens per 1,000 chars", "value": 262.2, "unit": "tokens" }, { "label": "MiniMax-M3.1-Flash tokens per 1,000 chars", "value": 261.1, "unit": "tokens" }, { "label": "gpt-oss-20b tokens per 1,000 chars", "value": 261.4, "unit": "tokens" }, { "label": "DeepSeek-V4.1-Flash tokens per 1,000 chars", "value": 274.1, "unit": "tokens" }, { "label": "Extra tokens Qwen3.x needs vs GLM-5.3-Flash (all text)", "value": 8.0, "unit": "%" }, { "label": "Qwen3.x JSON tokens per 1,000 chars", "value": 372.1, "unit": "tokens" } ], "sources": [ "https://huggingface.co/Qwen/Qwen3.6-35B-A3B", "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "https://huggingface.co/zai-org/GLM-5.3-Flash", "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash", "https://huggingface.co/openai/gpt-oss-20b", "https://www.gutenberg.org/ebooks/1342", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp" ], "next": "Add an estimated characters-per-second figure next to measured tok/s on cross-family comparisons, and confirm the factor on real generated outputs from the 35B and GLM lanes." }, { "id": "2026-10-28-mi50-vs-r9700-used-market", "date": "2026-10-28T00:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Used MI50 32GB vs new R9700: the MI50 now costs $619, not $150, and the R9700 still wins the 35B lane; 4x MI50 runs Flash-Next Q4 at ~49 tok/s", "summary": "A used Instinct MI50 32GB sells for a $619 eBay sold median (14 sales), so it is $19/GB vs $56/GB for the cheapest new R9700 ($1,799.99). On Qwen3.6-35B-A3B UD-Q4_K_XL one R9700 (Vulkan) does 117.3 tok/s decode and 3,121 tok/s prefill vs 89.9 and 968 on a tuned MI50 fork, despite 640 vs 1,024 GB/s. For Qwen3.8-Flash-Next, a published 4x MI50 rig with all of UD-Q4_K_XL in 128 GB of HBM decodes 49.2 tok/s (62 with MTP), about 2.3x our llama.cpp hybrid UD-Q4 on two R9700s, but our Strata UD-Q5 lane on two R9700s with experts in DDR5 is still faster at 58 tok/s, on half the cards and half the board power.", "details": "MI50 caveats: gfx906 is unsupported in ROCm 7.14 and absent from ROCm 10.0 (oldest Instinct listed is MI100); it runs via Vulkan or volunteer forks; MI50s can't do PCIe peer copies (build with GGML_CUDA_NO_PEER_COPY); passive cooler needs a shroud and blower; 300 W board power each. 4x MI50 = $2,476 used and 1,200 W board power for 128 GB; 2x R9700 = $3,600 new and 600 W for 64 GB, plus the system RAM the hybrid lane needs. Cross-host comparison, not an A/B: different hosts, builds and quants (our Strata row is UD-Q5, KLD 0.036 vs 0.060 for UD-Q4). The 4x MI50 Flash-Next numbers come from one maintainer's fork. Takeaway for a 4-card R9700 build: with UD-Q4 fully in VRAM, plain llama.cpp layer split reached ~49 tok/s on older cards, so moving experts out of system RAM should lift llama.cpp well above its ~21 tok/s hybrid rate (hypothesis, not yet tested). Llama 2 7B Q4_0 scoreboard rows (MI50 vs R9700): ROCm prefill 1,129 vs 4,773 and decode 106 vs 98; Vulkan prefill 1,128 vs 5,908 and decode 118 vs 153.", "setup": { "model": "Qwen3.6-35B-A3B; Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL (cited), UD-Q5_K_XL (measured Strata row)", "engine": "llama.cpp Vulkan / HIP gfx906 fork / Strata 0.2.3.1", "engine_version": "per source", "gpus": "1x and 4x Instinct MI50 32GB (cited); 1x and 2x Radeon AI PRO R9700", "context": "short to 36K", "flags": "fully in VRAM (MI50, single R9700); experts in DDR5 (2x R9700 Flash)" }, "metrics": [ { "label": "MI50 32GB used, eBay sold median (14 sales)", "value": 619, "unit": "USD" }, { "label": "Cheapest new R9700 in stock, 2026-10-28", "value": 1799.99, "unit": "USD" }, { "label": "35B-A3B UD-Q4_K_XL decode, 1x MI50 (gfx906 fork)", "value": 89.9, "unit": "tok/s" }, { "label": "35B-A3B UD-Q4_K_XL decode, 1x R9700 (Vulkan)", "value": 117.3, "unit": "tok/s" }, { "label": "35B-A3B UD-Q4_K_XL pp512, 1x MI50 vs 1x R9700", "value": "968 vs 3,121", "unit": "tok/s" }, { "label": "Flash-Next UD-Q4_K_XL decode, 4x MI50 all in VRAM (no MTP / MTP)", "value": "49.2 / 62", "unit": "tok/s" }, { "label": "Flash-Next UD-Q5_K_XL decode, 2x R9700 + DDR5, Strata (measured)", "value": 58, "unit": "tok/s" }, { "label": "Flash-Next UD-Q4_K_XL decode, 2x R9700 + DDR5, llama.cpp (measured)", "value": 21.2, "unit": "tok/s" } ], "sources": [ "https://github.com/sixvolts/llamacpp-gfx906-furnace", "https://github.com/ggml-org/llama.cpp/issues/26795", "https://gpusold.com/", "https://openclawdc.com/blog/amd-mi50-32gb-local-llm/", "https://vga.live/cards/" ], "next": "Add an 'MI50 vs R9700' buyer page and list the MI50 in the calculator as unsupported by current ROCm; once a 4-card R9700 setup exists, measure llama.cpp layer split with Flash-Next UD-Q4 fully in VRAM against this 4x MI50 row." }, { "id": "2026-10-27-context-overflow-262k", "date": "2026-10-27T23:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Hitting the 262K window: --context-shift is ignored on Qwen3.5/3.6 and Flash-Next, half-applied on GLM-5.3-Flash", "summary": "From llama.cpp source: an over-long prompt gets HTTP 400 exceed_context_size_error (with n_prompt_tokens and n_ctx), and a reply that reaches the window ends with finish_reason length. --context-shift is off by default and is auto-disabled with only a startup warning on any multi-section-RoPE model, which includes qwen35/qwen35moe (Qwen3.5/3.6) and qwen4exp (Flash-Next), so on those models overflow must be handled by the client.", "details": "llama_kv_cache::get_can_shift() returns false when n_pos_per_embd() > 1 (MROPE/IMROPE), and the same server check also forces --cache-reuse to 0; loading a vision projector disables the shift too. GLM-5.3-Flash (glm5-next) uses no RoPE and a hybrid memory whose can_shift comes from the attention cache only, so a shift is accepted: the DSA attention layers drop the discarded middle while the KDA recurrent layers keep its state. Output quality after such a half shift was not measured. Hermes Agent handles the llama.cpp error: it reads the per-slot n_ctx from /props, and its compression threshold (default 0.50) is floored to 0.75 for windows under 512K, so at 262,144 tokens it compacts at about 196,608 prompt tokens and keeps a tail of about 39K. Its compression summarizer defaults to provider auto (OpenRouter, then Nous Portal, then the main endpoint), so fully local setups should pin it to the local endpoint. Two amdai.net pages recommended --context-shift for overflowing context; that advice is a no-op for the house Qwen models and will be corrected.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, GLM-5.3-Flash", "quant": "any (Q4+ house files)", "engine": "llama.cpp llama-server (source read), Hermes Agent (source read)", "engine_version": "llama.cpp master 8ca4060b7e, hermes-agent main dc39239614", "gpus": "any", "context": "262144", "flags": "--context-shift on/off" }, "metrics": [ { "label": "Hermes compaction trigger at a 262,144 window (computed from source)", "value": 196608, "unit": "tokens" }, { "label": "Hermes recent tail kept after compaction (config formula)", "value": 39321, "unit": "tokens" }, { "label": "amdai.net pages recommending --context-shift", "value": 2, "unit": "pages" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-kv-cache.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-memory-hybrid-idx.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "https://github.com/NousResearch/hermes-agent/blob/main/agent/context_compressor.py", "https://github.com/NousResearch/hermes-agent/blob/main/cli-config.yaml.example" ], "next": "Check the always-on lanes' startup logs for the shift-disabled warning, pin the Hermes summarizer to the local endpoint, and fix the two site pages." }, { "id": "2026-10-27-gguf-sampler-defaults", "date": "2026-10-27T22:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama-server takes its default samplers from the GGUF: the Qwen 35B runs temp 1.0 / top_k 20, but min_p and presence penalty still differ from the card", "summary": "llama.cpp copies general.sampling.* keys from the GGUF into llama-server's defaults unless you pass a flag. The Unsloth and bartowski Qwen3.6-35B-A3B files and the Unsloth Qwen3.8-Flash-Next files carry temp 1.0, top_k 20 and top_p 0.95, so that is what runs with no flags, not llama.cpp's 0.80 / 40. None set min_p, so 0.05 stays (the cards say 0.0), and GGUF has no presence-penalty key, so the 35B's recommended 1.5 never applies. To match the 35B's thinking/general card row, add --min-p 0 --presence-penalty 1.5. Our own sampling page said the plain llama.cpp defaults apply; that is being corrected.", "details": "Checked in llama.cpp master (common_init_sampler_from_model; supported keys: sequence, top_k, top_p, min_p, xtc_probability, xtc_threshold, temp, penalty_last_n, penalty_repeat, mirostat). Header reads of the live files: Qwen3.6-35B-A3B UD-Q5_K_XL, UD-Q4_K_M and bartowski Q5_K_M, Qwen3.8-Flash-Next UD-Q6_K_XL and UD-Q5_K_XL all top_k 20 / top_p 0.95 / temp 1.0; GLM-5.3-Flash UD-Q6_K_XL temp 1.0 / top_p 0.95 with no top_k, so top_k 40 still applies. The defaults are not mode-aware: with thinking off the server still uses the thinking row (the instruct row is temp 0.7, top_p 0.8, presence 1.5). Passing --temp 0.80 --top-k 40 explicitly overrides the file's card-aligned values. Strata's server applies the card rows per mode by default when a client sends no sampling fields. Check what your server really uses with: curl -s localhost:8080/props (default_generation_settings.params). Clients that send their own fields override all of this per request. Quality effect of min_p 0.05 vs 0 and presence 0 vs 1.5 on these models is not measured.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-Flash-Next, GLM-5.3-Flash", "quant": "UD-Q4_K_M / UD-Q5_K_XL / UD-Q6_K_XL, bartowski Q5_K_M", "engine": "llama.cpp llama-server (source read), Strata serve layer (source read)", "engine_version": "llama.cpp master 8ca4060b7e, Strata main 5e8a4cca7d", "gpus": "any", "context": "n/a", "flags": "no sampler flags" }, "metrics": [ { "label": "35B GGUF temp (applied with no flags)", "value": 1.0, "unit": "" }, { "label": "35B GGUF top_k", "value": 20, "unit": "" }, { "label": "min_p that still applies (card 0.0)", "value": 0.05, "unit": "" }, { "label": "presence penalty that applies (35B card 1.5)", "value": 0.0, "unit": "" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/common/common.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF", "https://huggingface.co/bartowski/Qwen_Qwen3.6-35B-A3B-GGUF", "https://huggingface.co/Qwen/Qwen3.6-35B-A3B", "https://github.com/Niko1221/Strata/blob/main/serve/server.py" ], "next": "Read /props on the always-on 35B lane to confirm, then decide whether to pin --min-p 0 --presence-penalty 1.5; fix the site's sampling page." }, { "id": "2026-10-27-35b-on-16gb-radeon", "date": "2026-10-27T21:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.6-35B-A3B on one 16 GB Radeon: 32-66 tok/s at Q4+ with experts in system RAM, and our 'needs 2 GPUs' verdict was wrong", "summary": "Public reports show one 16 GB RDNA4 card runs the 35B-A3B MoE at 4-bit and above by keeping some expert layers in system RAM (llama.cpp --n-cpu-moe): RX 9070 XT Q4_K_M on ROCm 31.8 tok/s at --n-cpu-moe 40 rising to 43.9 at 16, with 256K context fitting at 24; RX 9070 XT IQ4_XS 57.9 tok/s fresh and 34.6 at 128K; RX 9060 XT UD-Q4_K_XL on Vulkan with MTP 66 tok/s at 40K (one report). Our /can-run pages told every 16, 20 and 24 GB owner this model 'needs 2 GPUs' while 8 and 12 GB cards got 'fits with RAM offload'. That verdict is being fixed.", "details": "Site scan of 160 /can-run pages on 2026-10-27: for Qwen3.6-35B, 4 pages on 8-12 GB cards said 'fits with RAM offload' and all 11 pages on 16-24 GB cards said 'needs 2 GPUs', so the verdict got worse as VRAM grew. The same inversion showed up for Llama 3.3 70B and Qwen3.6 27B, and none of the 160 pages mentioned expert offload. Tuning lessons from the sources: every expert layer moved back to the GPU helped decode monotonically; use physical cores (one tester saw SMT threads drop decode to about 9 tok/s); q8_0 KV beat q4_0 KV; MTP with 2 draft tokens helped most at long context (34.1 to 43.6 tok/s near 115K) while 6 draft tokens was a net loss. The 9070 XT Q4_K_M rows ran on dual-channel DDR4-3200, so a DDR5 host should do somewhat better on offloaded rows (not measured). For scale, one R9700 with the whole Q4_K_M file in its 32 GB runs it at 116.5 tok/s on our desktop, about 2.65x the best 9070 XT offload row (different hosts). Q3 and 2-bit reports were left out.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Q4_K_M / IQ4_XS / UD-Q4_K_XL (third-party)", "engine": "llama.cpp (ROCm and Vulkan)", "engine_version": "per source", "gpus": "1x RX 9070 XT 16GB or 1x RX 9060 XT 16GB", "context": "short to 128K", "flags": "--n-cpu-moe 16-40, q8_0 KV, -fa on" }, "metrics": [ { "label": "RX 9070 XT Q4_K_M decode, --n-cpu-moe 40", "value": 31.8, "unit": "tok/s" }, { "label": "RX 9070 XT Q4_K_M decode, --n-cpu-moe 16", "value": 43.9, "unit": "tok/s" }, { "label": "RX 9070 XT IQ4_XS decode at 128K", "value": 34.56, "unit": "tok/s" }, { "label": "RX 9060 XT UD-Q4_K_XL + MTP decode at 40K", "value": 66.0, "unit": "tok/s" }, { "label": "16-24 GB can-run pages wrongly saying 'needs 2 GPUs' (35B)", "value": 11, "unit": "pages" }, { "label": "1x R9700 Q4_K_M fully in VRAM (our measurement)", "value": 116.5, "unit": "tok/s" } ], "sources": [ "https://github.com/kr4ckhe4d/local-llm-benchmarks/blob/main/README.md", "https://wheeler-technology.co.nz/blog/local-ai-model-on-the-rx-9070-xt", "https://llamaperf.com/gpu/rx-9060-xt-16gb", "https://www.reddit.com/r/SelfHostedAI/comments/1uxco5t/local_llm_with_rx_9060_16gb/" ], "next": "Change the /can-run rule so a MoE model that fits one card plus system RAM says so, with suggested --n-cpu-moe values per context, and add a test that verdicts never get worse as VRAM grows." }, { "id": "2026-10-27-r9700-linux-distro-rows", "date": "2026-10-27T20:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Which Linux for an R9700: AMD lists 5 OS rows for Radeon, and Debian 13 / Ubuntu 24.04 archive ROCm predates RDNA4", "summary": "ROCm 10.1.0's compatibility matrix lists only Ubuntu 26.04.1, 24.04.5 (HWE 7.0), 22.04.5 (HWE 6.8), RHEL 10.2 and RHEL 9.8 for the R9700 (gfx1201). Debian, Oracle Linux, Rocky and SLES rows are Instinct-only. Distro archives on 2026-10-27: Debian 13 and Ubuntu 24.04 ship rocBLAS 5.5.1, older than the first RDNA4 release (ROCm 6.4.1), so apt install from those archives gives no R9700 libraries. Fedora 44 has ROCm 7.1, Arch 7.2.4, Ubuntu 26.10 10.0.", "details": "Our own /setup/fedora and /setup/arch pages said the 10.1 table names Debian, Oracle, Rocky and SLES; that is true for Instinct, not Radeon, and is being corrected. Archive versions checked via each distro's package API: Ubuntu 26.04 universe ROCm 7.1.0 (7.2.4 in -proposed); Ubuntu 26.10 (released Oct 15) ROCm 10.0, but 26.10 is not an AMD row; Fedora 43 6.4.4, Fedora 44 7.1.0, Rawhide 10.1.0, with gfx1201 in Fedora's default GPU target list on 43 and 44; EPEL 10 7.2.0; Arch extra 7.2.4 (10.0 is in Arch packaging git, not yet in extra); Debian sid 6.4.4, experimental 7.2.4. AMD driver versions listed for RDNA4: 31.60.0 and 31.50.0. Vulkan (Mesa RADV) with llama.cpp needs none of this and works on any current distro. No speed was measured on these distros.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201)", "engine": "ROCm 10.1.0 matrix + distro archives", "engine_version": "checked 2026-10-27" }, "metrics": [ { "label": "AMD Linux OS rows for Radeon/R9700", "value": 5, "unit": "rows" }, { "label": "Debian 13 archive rocBLAS", "value": "5.5.1", "unit": "" }, { "label": "Ubuntu 24.04 archive rocBLAS", "value": "5.5.1", "unit": "" }, { "label": "Fedora 44 rocm", "value": "7.1.0", "unit": "" }, { "label": "Arch extra rocBLAS", "value": "7.2.4", "unit": "" }, { "label": "Ubuntu 26.10 archive rocBLAS", "value": "10.0", "unit": "" } ], "sources": [ "https://rocm.docs.amd.com/en/docs-10.1.0/compatibility/compatibility-matrix.html", "https://rocm.docs.amd.com/en/docs-6.4.1/about/release-notes.html", "https://sources.debian.org/src/rocblas/", "https://launchpad.net/ubuntu/+source/rocblas", "https://packages.fedoraproject.org/pkgs/rocm/rocm/", "https://src.fedoraproject.org/rpms/rocm-rpm-macros", "https://archlinux.org/packages/extra/x86_64/rocblas/" ], "next": "Fix the Fedora and Arch setup pages, add a Debian page and a distro picker table on /setup." }, { "id": "2026-10-27-r9700-vm-passthrough-p2p", "date": "2026-10-27T18:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 in a Proxmox VM: one card just works, two cards lose P2P until two config fixes", "summary": "AMD has validated R9700 passthrough on Proxmox VE 9.1 (up to 4 cards, ROCm 7.2, Ubuntu 24.04 HWE guest), but inside a VFIO guest the two-card peer-to-peer path is off by default. One owner restored it by passing only the GPU function (not all functions) and setting cpu: host,guest-phys-bits=44, then measured about 13 GiB/s P2P and 31.4 tok/s on a 27B model split across 2x R9700.", "details": "AMD's guide needs the guest's HWE kernel (7.0+; the default 6.8 kernel has no usable amdgpu) and amdgpu-install --no-dkms, and calls the VM overhead negligible without publishing speeds or covering P2P. Passing the whole device made the guest log 'PCIE atomic ops is not supported'; high BAR placement blocked P2P until guest-phys-bits=44 (the owner's explanation, unverified). With a direct-P2P AllReduce patch, decode went 28.58 to 31.42 tok/s while prompt speed dropped 934.58 to 871.65 tok/s. Reset handling comes from RX 9070 XT reports (same Navi 48 chip): vendor-reset does not support RDNA4, pre-binding vfio-pci at boot can leave the card stuck in D3, and unbinding before the VM exits avoids an unstable bus reset. A VM does not protect against the gfx1201 hang where the GPU's security processor fails to resume; that still needs a reboot.", "setup": { "gpus": "2x Radeon AI PRO R9700 (cited owner, AM4, PCIe 4.0 x8/x8)", "model": "Qwen3.8-27B", "quant": "UD-Q8_K_XL", "engine": "llama.cpp ROCm, tensor split 1:1 (published)", "hypervisor": "Proxmox VE 9" }, "metrics": [ { "label": "P2P copy inside VM, each direction", "value": 13, "unit": "GiB/s" }, { "label": "Decode TG128, direct-P2P AllReduce", "value": 31.42, "unit": "tok/s" }, { "label": "Decode TG128, stock", "value": 28.58, "unit": "tok/s" }, { "label": "Prompt PP512, direct-P2P AllReduce", "value": 871.65, "unit": "tok/s" }, { "label": "Prompt PP512, stock", "value": 934.58, "unit": "tok/s" } ], "sources": [ "https://www.amd.com/en/developer/resources/technical-articles/2026/gpu-passthrough-on-proxmox-ve.html", "https://forum.level1techs.com/t/dual-r9700-llama-cpp-rocm-tensor-split-working-p2p-on-proxmox-qwen3-8-27b/254183", "https://forum.level1techs.com/t/r9700-vfio/248634", "https://gist.github.com/PhialsBasement/c34e908352bd2e8a49197edc70df2d94", "https://forum.proxmox.com/threads/amd-radeon-gpu-passthrough-infamous-reset-bug.186112/", "https://github.com/ROCm/legacy-rocm-build/issues/6630" ], "next": "Add a VM guide page and a read-only guest check script (device ID, BAR placement, bridge atomics, amdgpu pcie_p2p) to the site." }, { "id": "2026-10-27-yarn-past-262k-llama-server-cap", "date": "2026-10-27T17:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "Going past 262K on Qwen3.6-35B-A3B and Qwen3.8-Flash-Next: YaRN flags alone get capped by llama-server", "summary": "Both official model cards list 262,144 tokens as native and about 1M with YaRN (factor 4). The Unsloth GGUFs carry context_length 262144 and no rope-scaling keys, and current llama.cpp master still caps every server slot at the model's training context, so --rope-scaling yarn on its own silently stays at 262,144. Overriding the declared context (--override-kv qwen35moe.context_length=int:N, or qwen4exp for Flash-Next) is required. By our KV math, 524K on the 35B fits one R9700 at UD-Q5_K_XL only with q8_0 KV, and 1M needs two cards at f16 KV.", "details": "Checked llama.cpp master 1b4554eff1 (2026-10-27): n_ctx_slot() returns min(context, n_ctx_train). Issue #22140 was auto-closed as stale and fix PR #22145 was closed without merging. Both model graphs do pass YaRN parameters into the rotary embedding, including Flash-Next's sparse-attention indexer, so YaRN takes effect once the cap is lifted. KV cache per token at f16: 20 KiB for the 35B (10 full-attention layers, 2 KV heads, head dim 256; matches 5.0 GiB measured at 262K) and about 30 KiB for Flash-Next (from llama.cpp's allocator). Estimated KV at 1,010,000 tokens: 35B 19.3 GiB f16 / 10.3 q8_0 / 5.5 q4_0; Flash-Next 29.0 / 15.5 / 8.2. The model cards warn that static YaRN can hurt short prompts, so keep it in a separate long-context preset rather than the default. On the HIP backend, mixed K/V types such as q8_0 K with q4_0 V are converted to f16 on every attention call; use matched types. Fit numbers are estimates; needle and short-prompt quality tests on R9700 are pending.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-Flash-Next", "quant": "UD-Q5_K_XL (35B fit math)", "engine": "llama.cpp llama-server", "engine_version": "master 1b4554eff1", "gpus": "1-2x Radeon AI PRO R9700", "context": "262,144 native; 524,288-1,010,000 with YaRN", "flags": "--override-kv qwen35moe.context_length=int:524288 --rope-scaling yarn --rope-scale 2 --yarn-orig-ctx 262144 -c 524288" }, "metrics": [ { "label": "Native context (both models)", "value": 262144, "unit": "tokens" }, { "label": "35B KV per token, f16", "value": 20, "unit": "KiB" }, { "label": "Flash-Next KV per token, f16 (approx.)", "value": 30, "unit": "KiB" }, { "label": "35B KV at 1,010,000 tokens, f16 (est.)", "value": 19.3, "unit": "GiB" }, { "label": "Flash-Next KV at 1,000,000 tokens, q8_0 (est.)", "value": 15.5, "unit": "GiB" } ], "sources": [ "https://huggingface.co/Qwen/Qwen3.6-35B-A3B", "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF", "https://github.com/ggml-org/llama.cpp/issues/22140", "https://github.com/ggml-org/llama.cpp/pull/22145", "https://github.com/ggml-org/llama.cpp/blob/1b4554eff1/tools/server/server-context.cpp", "https://huggingface.co/PiehSoft/Qwen3.6-40B-Deckard-MTP/discussions/3" ], "next": "Run a cap check, a 128K-500K needle test and a short-prompt perplexity comparison with and without YaRN on R9700 before recommending a long-context preset." }, { "id": "2026-10-27-calculator-decode-estimate-calibration", "date": "2026-10-27T16:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "Bandwidth-based tok/s estimates: about right for dense models on Radeon, 1.6-2.3x too high for small-active MoE", "summary": "Checked the common decode estimate (published memory bandwidth x 75% / bytes per token) against public llama.cpp Vulkan results for 14 Radeon cards and against R9700 runs of Qwen3.6-35B-A3B. Dense models land at a median 76% of published bandwidth (58-89%), so 75% is a fair midpoint but not a floor. For the 3B-active 35B MoE the same formula predicts 225-264 tok/s on one R9700, while real runs give 100.8-116.5 (llama.cpp Vulkan, our runs) and 165.2 (public tg128).", "details": "Dense data: llama.cpp Vulkan scoreboard, Llama 2 7B Q4_0 tg128 (3.749 GB streamed per token). Highest delivered fraction: R9700 0.89 (152.7 tok/s on 640 GB/s); lowest: RX 7900 XT 0.58 (123.2 on 800 GB/s); RX 7900 XTX 0.75 (190.9 on 960). For small-active MoE, about 2-5.5 ms per token is fixed cost (routing and many small kernels across 40 layers) that a bandwidth-only formula cannot see, so a calibrated 33-47% of published bandwidth fits the measured R9700 numbers. While checking specs we also found AMD-published numbers worth using: RX 9060 XT 320 GB/s, RX 6700 XT 384 GB/s, Ryzen AI Max+ 395 256 GB/s, and 32 GB on the passive R9700S (300 W) and R9600D (150 W). Limits: scoreboard rows use different llama.cpp builds; MoE calibration is from one card type; no long-context data.", "setup": { "model": "Llama 2 7B Q4_0 (dense); Qwen3.6-35B-A3B (MoE)", "quant": "Q4_0; UD-Q5_K_XL / Q5_K_M / Q4_K_M", "engine": "llama.cpp Vulkan (published + ours)", "engine_version": "per source", "gpus": "14 Radeon cards (dense); 1x Radeon AI PRO R9700 (MoE)", "context": "tg128 / short", "flags": "defaults" }, "metrics": [ { "label": "Dense: median share of published bandwidth delivered", "value": 76, "unit": "%" }, { "label": "Dense: range across 14 Radeons", "value": "58-89", "unit": "%" }, { "label": "35B-A3B UD-Q5_K_XL on 1x R9700: formula vs measured", "value": "225 vs 100.8", "unit": "tok/s" }, { "label": "35B-A3B MoE: implied share of bandwidth", "value": "33-47", "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/10879", "https://www.amd.com/en/products/graphics/desktops/radeon/9000-series/amd-radeon-rx-9060xt.html", "https://www.amd.com/en/products/graphics/desktops/radeon/6000-series/amd-radeon-rx-6700-xt.html", "https://www.amd.com/en/products/processors/desktops/ryzen/ryzen-ai-halo/ryzen-ai-max-plus-395.html", "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700s.html", "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9600d.html", "https://rocm.docs.amd.com/en/latest/reference/gpu-specs.html" ], "next": "Show tok/s estimates as ranges with separate dense and MoE factors, fill the AMD-published specs, and add a second card to the MoE calibration when a public A3B result appears." }, { "id": "2026-10-27-sglang-tp2-vs-llamacpp-r9700", "date": "2026-10-27T15:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "SGLang TP2 on 2x R9700 is slower than llama.cpp on one card for Qwen3.6-35B-A3B, and has no DDR5 path for Flash models", "summary": "The most complete public 2x R9700 SGLang rig (v0.5.15 plus RDNA4 patches, TP=2) measured Qwen3.6-35B-A3B AWQ int4 at 67.0 tok/s short and 22.0 tok/s at 197K input. llama.cpp on a single R9700 does 165.2 tok/s (Vulkan, Q4_K_M, tg128) and our own Q5 run does 100.8 short and about 50 at 256K. Upstream SGLang still blocks gfx1201 builds, and its only CPU-expert path (KTransformers) is built around Intel AMX, so Flash-class models stay on llama.cpp or Strata.", "details": "Numbers come from different hosts and harnesses, so treat ratios as rough: 1.5x to 2.5x in llama.cpp's favour short, about 2.3x at depth, with SGLang using both cards. Where SGLang does stand out is a dense 27B FP8 model with a speculator: 34.3 / 44.5 / 41.6 / 30.1 tok/s at 24 / 7.3K / 58K / 197K input, the only speculative lane shown holding near 200K on R9700 (llama.cpp Vulkan tensor split does 45.0 tg128 on the same model at Q4_K_M). Lessons that carry over to any TP2 engine on R9700: the kernel needs CONFIG_HSA_AMD_P2P and CONFIG_PCI_P2PDMA plus iommu=pt (HSA_FORCE_FINE_GRAIN_PCIE=1 changed nothing); a hard-killed run can leave RCCL shared-memory segments that crash the next launch; and aborting a deep TP2 prefill mid-collective can hang a GPU until reset. AMD's ROCm docs say to disable AITER and use the Triton attention backend on Radeon.", "setup": { "model": "Qwen3.6-35B-A3B; Qwen3.8-27B (dense)", "quant": "AWQ int4 / FP8 (SGLang); Q4_K_M and UD-Q5_K_XL (llama.cpp)", "engine": "SGLang (patched RDNA4 fork) vs llama.cpp", "engine_version": "SGLang v0.5.15-v0.5.22; llama.cpp PR #25051 build", "gpus": "1-2x Radeon AI PRO R9700", "context": "short to 197K input", "flags": "SGLang TP=2, Triton attention, HIP graphs; llama.cpp Vulkan 1 GPU / layer / tensor" }, "metrics": [ { "label": "SGLang TP2 35B-A3B AWQ, short", "value": 67.0, "unit": "tok/s" }, { "label": "SGLang TP2 35B-A3B AWQ, 197K input", "value": 22.0, "unit": "tok/s" }, { "label": "llama.cpp Vulkan 1 card 35B-A3B Q4_K_M tg128", "value": 165.2, "unit": "tok/s" }, { "label": "llama.cpp 1 card 35B-A3B UD-Q5_K_XL, short", "value": 100.8, "unit": "tok/s" }, { "label": "SGLang TP2 27B FP8 + speculator, 197K input", "value": 30.1, "unit": "tok/s" } ], "sources": [ "https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference", "https://github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference/blob/main/benchmarks/qwen3.6-35b-moe-awq/results.json", "https://github.com/ggml-org/llama.cpp/pull/25051#issuecomment-6382846940", "https://github.com/sgl-project/sglang/issues/30599", "https://rocm.docs.amd.com/projects/ai-ecosystem/en/latest/inference/sglang.html", "https://www.lmsys.org/blog/2025-10-22-KTransformers/" ], "next": "Keep llama.cpp and Strata for the Flash lanes. Check the TP2 kernel and IOMMU prerequisites before the next vLLM tensor-parallel test." }, { "id": "2026-10-27-rocm-smi-gone-in-rocm-10", "date": "2026-10-27T14:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "rocm-smi is gone in ROCm 10.x: use amd-smi --rocm-smi, or the driver's own counters on Vulkan-only setups", "summary": "AMD's ROCm 10.1 transition guide lists ROCm SMI as removed from the Core SDK (replaced by AMD SMI, package amdrocm-amdsmi); it was dropped from TheRock builds in 10.0. On a fresh 10.x install, 'rocm-smi' prints command not found even though ROCm works. 'amd-smi --rocm-smi' prints the familiar concise table, and 'amd-smi monitor --gfx --vram-usage --watch 1' replaces 'watch -n1 rocm-smi'. Our own scan of amdai.net found 23 command lines across 15 pages still calling rocm-smi with no fallback, including the verify tutorial's pass/fail checklist; we are fixing those.", "details": "Sourced, not a new measurement. Mapping from the amd-smi 10.1 CLI docs: rocm-smi --showproductname --showmeminfo vram becomes amd-smi static --asic --vram; --showuse / --showmemuse become amd-smi metric --usage --mem-usage. --rocm-smi is a top-level flag and cannot be combined with subcommands. Readers on llama.cpp Vulkan have neither tool installed; the amdgpu kernel driver exposes mem_info_vram_total, mem_info_vram_used and gpu_busy_percent under /sys/class/drm/cardN/device/, which is enough to prove a model is on the card. ROCm SMI can still be built from source, and older legacy installs (7.2 and earlier) may still have it.", "setup": { "stack": "ROCm Core SDK 10.1.0 (released 2026-10-05) documentation", "site scan": "706 sitemap pages, 2026-10-27" }, "metrics": [ { "label": "Site pages mentioning rocm-smi", "value": 27, "unit": "pages" }, { "label": "Command lines calling rocm-smi with no amd-smi fallback", "value": 23, "unit": "lines" } ], "sources": [ "https://rocmdocs.amd.com/en/latest/about/transition-guide-TheRock.html", "https://rocm.docs.amd.com/projects/rocm_smi_lib/en/docs-10.1.0/index.html", "https://rocm.docs.amd.com/projects/amdsmi/en/docs-10.1.0/how-to/amdsmi-cli-tool.html", "https://docs.kernel.org/gpu/amdgpu/driver-misc.html", "https://docs.kernel.org/gpu/amdgpu/thermal.html" ], "next": "Confirm the replacement commands on our dual-R9700 box, then update the verify, troubleshoot, serve and run tutorials to amd-smi with a sysfs fallback for Vulkan readers." }, { "id": "2026-10-27-kv-estimate-audit-llamacpp", "date": "2026-10-27T13:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "Our KV-cache calculator vs llama.cpp's real allocator: GLM-5.3-Flash overstated 2.3x, Qwen3.8-Flash-Next 1.6 GiB short at 262K", "summary": "We checked the KV estimate for all 19 model presets in the amdai.net calculator against the model configs and llama.cpp master's memory code (Oct 27). 13 match within 10%. GLM-5.3-Flash was the big miss: only 11 of its 45 layers keep a growing cache (34 are linear-attention layers with a fixed state), so 262K context needs about 4.95 GiB at f16, not the 11.25 GiB we showed. Qwen3.8-Flash-Next needs about 7.6 GiB at 262K, not 6.0, because its sparse-attention indexer keys also live in the cache. Gemma 4's sliding-window layers were under-counted by 0.4-0.8 GiB.", "details": "Derived from source, not yet a measured allocation. GLM-5.3-Flash: config.json layer_types lists 34 linear_attention + 11 deepseek_sparse_attention layers, and the GGUF head_count_kv mask agrees. llama.cpp keeps a K-only MLA cache (kv_lora_rank 512, no RoPE part) plus an indexer cache of 3 x 128 elements per token on those 11 layers, and a fixed f32 recurrent state (~146 MiB per sequence) on the 34 linear layers. Qwen3.8-Flash-Next: 12 full-attention layers at 2 KV heads x 256 plus a 2 x 128-element indexer row per token (raw + pooled key). Gemma 4 31B/12B: global layers use 4x512 / 1x512 heads but sliding layers use 16x256 / 8x256, and llama.cpp sizes the sliding cache at window + micro-batch (1,536 cells at the default 512). Dense models (Llama 3.x, Qwen3) and Qwen3.5/3.6 hybrids matched; hybrids also hold a 50-150 MiB fixed state per slot. All figures: f16 cache, one sequence, GiB.", "setup": { "engine": "llama.cpp master 19ff817f (2026-10-27), source review", "cache": "f16 K/V, 1 sequence, -ub 512" }, "metrics": [ { "label": "Presets within 10% of llama.cpp", "value": 13, "unit": "of 19" }, { "label": "GLM-5.3-Flash KV at 262K (llama.cpp)", "value": 4.95, "unit": "GiB" }, { "label": "GLM-5.3-Flash KV at 262K (old site estimate)", "value": 11.25, "unit": "GiB" }, { "label": "Qwen3.8-Flash-Next KV at 262K (llama.cpp)", "value": 7.61, "unit": "GiB" }, { "label": "Qwen3.8-Flash-Next KV at 262K (old site estimate)", "value": 6.0, "unit": "GiB" } ], "sources": [ "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/config.json", "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json", "https://huggingface.co/google/gemma-4-31b/blob/main/config.json", "https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-model.cpp", "https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/models/glm5-next.cpp", "https://github.com/ggml-org/llama.cpp/blob/19ff817fb25e9a1b838b1d7f9f4325f7bf9779c4/src/llama-kv-cache-iswa.cpp", "https://amdai.net/calculator/" ], "next": "Confirm against llama-server startup logs on the 2x R9700 lanes, then fix the calculator presets (sparse-layer count, indexer cache, Gemma 4 window heads, fixed state per slot)." }, { "id": "2026-10-27-r9700-street-price-noon", "date": "2026-10-27T12:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 street price is now about $2,000: cheapest new card $1,949.99, 50% over MSRP and $250 up in a day", "summary": "As of Oct 27 at noon CT, every new in-stock Radeon AI PRO R9700 32 GB we found sold by Micro Center costs $1,949.99 to $1,999.99, and Newegg and B&H list $1,999.99 (ASUS Turbo). That is up $250-300 from yesterday's $1,699.99-$1,799.99. The cheapest is the Gigabyte AI TOP at $1,949.99 (Micro Center, in-store pickup only), 50% above AMD's $1,299 MSRP. Two cards (64 GB VRAM) now cost $3,899.98 before tax.", "details": "Checked on the retailer pages and on vga.live's card pages around 12:00 CT, before tax. Micro Center: Gigabyte AI TOP $1,949.99 (was $1,999.99, in-store pickup only); ASRock Creator, ASUS Turbo, Sapphire and XFX $1,999.99. Newegg: ASUS Turbo $1,999.99 sold by Newegg, but Newegg's own page showed it out of stock; Newegg marketplace sellers list the Gigabyte AI TOP from $2,079.97. B&H lists the ASUS Turbo at $1,999.99 and the Gigabyte AI TOP at $1,899.95, but vga.live shows the B&H Gigabyte listing as unavailable. Amazon: third-party sellers from $1,943.99 (Sapphire), Gigabyte $2,199.99, ASUS $2,495.00, PowerColor $2,664.51. The lowest price seen since Sep 29 is $1,599.99, so today's cheapest card is $350 above that floor. At $1,949.99 the R9700 costs $60.9 per GB of VRAM. Prices move daily; check before buying.", "setup": { "gpus": "Radeon AI PRO R9700 32 GB (6 board-partner models)", "region": "US, new condition" }, "metrics": [ { "label": "Cheapest new R9700 in stock", "value": 1949.99, "unit": "USD" }, { "label": "Premium over $1,299 MSRP", "value": 50.1, "unit": "%" }, { "label": "Change since Oct 26 snapshot", "value": 250.0, "unit": "USD" }, { "label": "Newegg (ASUS Turbo, sold by Newegg)", "value": 1999.99, "unit": "USD" }, { "label": "Two cards today, before tax", "value": 3899.98, "unit": "USD" }, { "label": "Lowest price since Sep 29", "value": 1599.99, "unit": "USD" } ], "sources": [ "https://www.microcenter.com/product/699521/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation-graphics-card", "https://www.newegg.com/asus-turbo-ai-pro-r9700-32g-32gb-graphics-card/p/N82E16814126826", "https://www.newegg.com/p/pl?d=radeon+ai+pro+r9700", "https://www.bhphotovideo.com/c/search?q=radeon%20ai%20pro%20r9700", "https://www.amazon.com/dp/B0G75W64R3", "https://vga.live/gpus/radeon-ai-pro-r9700", "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html" ], "next": "Keep the daily price snapshot; flag when the cheapest new card drops back to $1,599.99." }, { "id": "2026-10-27-llama-cpp-removed-load-flags", "date": "2026-10-27T09:40:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama.cpp removed --no-mmap, --mlock and llama-bench -mmp: old commands now stop at startup, and LLAMA_ARG_NO_MMAP is silently ignored", "summary": "Since release b10875 (2026-09-09, PR #28334) llama-server and llama-cli exit with 'invalid argument' on --no-mmap, --mmap, --mlock and --direct-io, and llama-bench rejects -mmp and -dio. Use --load-mode none (old --no-mmap), mmap+mlock (old --mlock), mlock (old --no-mmap --mlock) or dio; in llama-bench, -lm none. The env vars LLAMA_ARG_NO_MMAP, LLAMA_ARG_MMAP and LLAMA_ARG_MLOCK are no longer read at all, so Docker or systemd setups that used them quietly fall back to the default load mode (auto, which memory-maps).", "details": "Checked against llama.cpp master 27b7a8a009de (2026-10-27) and six dated snapshots of common/arg.cpp back to 2025-04-27. The load flags were folded into --load-mode on 2026-07-23 (#20834) and warned as deprecated until the 2026-09-09 removal. llama.cpp reads environment variables only for options that still exist, so a removed option's env var produces no warning. Other flags that now fail: --draft / --draft-max / --draft-min (removed in the speculative-decoding argument refactor, #22397 and #22964, April-May 2026; use --spec-draft-n-max / --spec-draft-n-min or --spec-ngram-mod-n-max), -cd / --ctx-size-draft, -cpent / --checkpoint-every-n-tokens and --spec-replace (gone between the 2026-04-27 and 2026-07-27 snapshots). -dt / --defrag-thold is still accepted but does nothing. -fa needs a value (on, off or auto); a bare -fa followed by another flag fails. We wrote a small linter for commands, scripts, systemd units and env files and ran it on every llama.cpp command shown on amdai.net (130 distinct command lines across 704 pages): none use a removed or unknown flag.", "setup": { "engine": "llama.cpp", "engine_version": "master 27b7a8a009de (2026-10-27); removal in b10875", "method": "source audit of common/arg.cpp and tools/llama-bench/llama-bench.cpp" }, "metrics": [ { "label": "First release without --no-mmap / --mlock / -mmp", "value": "b10875", "unit": "" }, { "label": "llama.cpp option entries, 2025-04-27", "value": 237, "unit": "" }, { "label": "llama.cpp option entries, 2026-10-27", "value": 361, "unit": "" }, { "label": "amdai.net command lines with a removed or unknown flag", "value": "0 of 130", "unit": "" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/28334", "https://github.com/ggml-org/llama.cpp/commit/14a9d09f75", "https://github.com/ggml-org/llama.cpp/pull/20834", "https://github.com/ggml-org/llama.cpp/pull/26135", "https://github.com/ggml-org/llama.cpp/pull/22397", "https://github.com/ggml-org/llama.cpp/pull/22964", "https://github.com/ggml-org/llama.cpp/releases/tag/b10875", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md" ], "next": "Before rebuilding llama.cpp past b10875, check every launcher, systemd unit and env file for the removed flags and env vars; switch to --load-mode / LLAMA_ARG_LOAD_MODE. The site's mmap page and glossary get updated wording (removed, not deprecated)." }, { "id": "2026-10-27-vllm-int4-r9700-gating", "date": "2026-10-27T08:15:00-05:00", "project": "dual-r9700", "kind": "result", "title": "vLLM 4-bit on the R9700: recent RDNA int4 speedups skip gfx1201, GPTQ crashes, and the default attention backend can return wrong output", "summary": "Reading vLLM main (e7fa26d5c8, 2026-10-27): the Oct 13 W4A16 decode tuning (+9-21% GEMM, up to -66% on speculative shapes) is gated to Strix Halo (gfx115x); the new int4 MoE decode kernel (+44% on a 35B-A3B, Strix Halo) and MXFP4 W4A8 GEMV are gfx11-only; upstream has no tuned int4 MoE configs for the R9700. On R9700, GPTQ checkpoints crash at startup unless you pass --linear-backend TRITON, and the default ROCM_ATTN backend gave wrong outputs or GPU faults outside head 128 / KV block 16, while TRITON_ATTN was clean.", "details": "Int4 MoE on gfx12 runs vLLM's Triton WNA16 path, which reached about 18% of memory bandwidth at batch 1 on Strix Halo. The faster W4A8 MoE kernel (open PR #62480) hard-checks for gfx11, but its core int8 dot instruction is the one llama.cpp already uses on RDNA4, so a gfx12 port looks plausible (untested). An open tuned-config PR for E=256,N=512 on the R9700 cut fused-MoE kernel time 19% at batch 1 and raised end-to-end output 3.9% at 1 request and 17.7% at 4. Int4 checkpoints usually keep lm_head in bf16; open PR #62708 quantizes it at load on gfx11 and gfx12 (1.12-1.18x on Strix Halo). Our estimate (640 GB/s spec bandwidth at 75% efficiency, not measured): about 1.5 ms per token saved on a 35B-A3B on one R9700, about 0.9 ms per card on a Flash-class model at TP2. GGUF support moved out of vLLM core in June into vllm-gguf-plugin; vLLM's docs table still lists it as AMD-supported, the plugin's PyPI wheel links the CUDA runtime (libcudart.so.13), so ROCm needs a source build, and its kernels were copied from llama.cpp b2899. For GGUF on Radeon, llama.cpp remains the path. Also open: on R9700, vLLM main runs 13.5% below v0.29 on gpt-oss-20b because of the MoE combine step (fix PR #64055 restores +17.6%).", "setup": { "engine": "vLLM (source reading)", "engine_version": "main e7fa26d5c8 (2026-10-27)", "gpus": "Radeon AI PRO R9700 (gfx1201)", "quant": "W4A16 / GPTQ / AWQ / MXFP4 / GGUF (4-bit class)" }, "metrics": [ { "label": "Int4 MoE decode kernel, 35B-A3B (Strix Halo, gfx11-only)", "value": 44, "unit": "% faster" }, { "label": "R9700 tuned int4 MoE config, kernel time at batch 1 (open PR)", "value": -19, "unit": "%" }, { "label": "ROCM_ATTN wrong outputs on R9700 (of 12,389 batches)", "value": 635, "unit": "batches" }, { "label": "ROCM_ATTN GPU faults on R9700 (of 12,389 batches)", "value": 119, "unit": "batches" }, { "label": "R9700 int4_w4a16 MoE configs in vLLM main", "value": 0, "unit": "files" } ], "sources": [ "https://github.com/vllm-project/vllm/pull/57199", "https://github.com/vllm-project/vllm/pull/62480", "https://github.com/vllm-project/vllm/pull/60413", "https://github.com/vllm-project/vllm/pull/62708", "https://github.com/vllm-project/vllm/pull/64045", "https://github.com/vllm-project/vllm/issues/63597", "https://github.com/vllm-project/vllm/pull/52637", "https://github.com/vllm-project/vllm/issues/62697", "https://github.com/vllm-project/vllm/pull/64055", "https://github.com/vllm-project/vllm/pull/39612", "https://github.com/vllm-project/vllm-gguf-plugin/issues/142", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/common.cuh" ], "next": "Measure on two R9700s: attention backend correctness, the gfx115x tuning enabled on gfx12, a tuned int4 MoE file, int4 lm_head, then try the W4A8 MoE kernel on gfx12." }, { "id": "2026-10-27-llama-server-metrics-gauges", "date": "2026-10-27T07:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Monitoring local LLM speed: llama-server's tok/s gauges read 0 mid-generation; use the counters, and compare by context depth", "summary": "In current llama.cpp, the /metrics gauges predicted_tokens_seconds and prompt_tokens_seconds are reset by every scrape and only updated when a request finishes, so dashboards show 0 while a long generation runs and two scrapers steal each other's windows. Rates from the cumulative counters are reliable. Strata's /metrics already reports a live 2-second tok/s plus per-request speeds.", "details": "Read at llama.cpp master 0c8c59ef3d (2026-10-27): each /metrics request sets metrics_reset_bucket, and generation stats reach the bucket only in the slot reset callback at request end. The 0-reading bug was filed as #27436 and auto-closed as stale; fix PRs #27760 and #30543 are still open. Use tokens_predicted_total / tokens_predicted_seconds_total and prompt_tokens_total / prompt_seconds_total deltas instead; prompt_tokens_cached_total (cache reuse) exists but is missing from the README table. The gauge and per-request timings count n_gen-1 decode steps while the counter counts n_gen, so counter-derived speed reads about 5% high on 20-token tool-call replies; speculative/MTP tokens count in both. Separately, average tok/s falls as conversations get longer, so a speed alert should compare like depths: we bin requests by context depth (0-8K, 8-32K, 32-64K, 64-128K, 128K+) using the 'stop processing: n_tokens' and 'eval time' log lines and flag a bin only when its median drops more than 5% and more than 3 robust standard errors. Strata's JSON /metrics keeps the last 500 requests with prompt size, decode tok/s and the share of experts read over PCIe, which feeds the same check.", "setup": { "engine": "llama.cpp llama-server / Strata (source read)", "engine_version": "llama.cpp 0c8c59ef3d; Strata main 2026-10-25", "gpus": "2x Radeon AI PRO R9700 (target)" }, "metrics": [ { "label": "predicted_tokens_seconds while a request is generating", "value": 0, "unit": "tok/s" }, { "label": "Counter-derived speed bias on a 20-token reply", "value": 5, "unit": "%" }, { "label": "Strata live rate window", "value": 2, "unit": "s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-common.h", "https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-context.cpp", "https://github.com/ggml-org/llama.cpp/blob/0c8c59ef3d/tools/server/server-task.cpp", "https://github.com/ggml-org/llama.cpp/issues/27436", "https://github.com/ggml-org/llama.cpp/pull/27760", "https://github.com/ggml-org/llama.cpp/pull/30543", "https://github.com/Niko1221/Strata/blob/main/serve/server.py", "https://docs.kernel.org/gpu/amdgpu/thermal.html" ], "next": "Build 7-day per-depth speed baselines for the always-on dual-R9700 lanes and check them after every driver or engine update." }, { "id": "2026-10-27-vram-per-dollar", "date": "2026-10-27T06:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "VRAM per dollar: a new R9700 is now at parity with a used RTX 3090, and wins on watts and slots", "summary": "With the cheapest new R9700 at $1,949.99 (up $250 since yesterday) it costs $60.9 per GB of VRAM, against $58.9-63.7 per GB for a used RTX 3090 at eBay sold medians of $1,413-$1,528. Per watt it wins clearly (9.4 W/GB vs 14.6), and 64 GB takes two R9700s at 600 W instead of three 3090s at 1,050 W.", "details": "For Flash-class MoE models that don't fit in VRAM, gigabytes per card decide how many experts stay on the GPU, so $/GB, W/GB and card count matter more than per-card bandwidth. At yesterday's $1,699.99 the R9700 was 10-17% cheaper per GB than a used 3090; at its $1,599.99 low since Sep 29, 15-21% cheaper. Break-even against a $1,528 used 3090 is about $2,037. The 3090 still has about 1.5x the bandwidth per card (936 vs 640 GB/s, vendor spec), which matters when the whole model fits in VRAM. Builds (GPUs only): 64 GB = 2x R9700 $3,900, 600 W vs 3x 3090 (72 GB) $4,584, 1,050 W; 128 GB = 4x R9700 $7,800, 1,200 W vs 6x 3090 (144 GB) $9,168, 2,100 W. Cheapest per GB among used cards: RX 7900 XTX ($1,004, $41.8/GB) and Quadro RTX 8000 48 GB ($2,080, $43.3/GB, 5.4 W/GB). Used RTX 4090 ($3,037) and new RTX 5090 ($5,999.99) cost $126-188/GB. Used prices carry no warranty; one tracker (getpcparts) shows a much lower $1,093 3090 average and was treated as an outlier. Prices are before tax, US.", "setup": "Prices pulled 2026-10-27 about 06:10 CT: new prices from vga.live/rtx.live store listings, used prices from eBay completed-sale medians on gpusold.com, compsniper.com and pcprice.watch. Bandwidth and board power are vendor specs. GPU cost only (no board, PSU or risers).", "metrics": [ { "label": "R9700 cheapest new in stock", "value": 1949.99, "unit": "USD" }, { "label": "Used RTX 3090 eBay sold median (gpusold, 334 sales)", "value": 1528, "unit": "USD" }, { "label": "R9700 cost per GB VRAM", "value": 60.9, "unit": "USD/GB" }, { "label": "Used RTX 3090 cost per GB VRAM", "value": 63.7, "unit": "USD/GB" }, { "label": "R9700 board power per GB", "value": 9.4, "unit": "W/GB" }, { "label": "RTX 3090 board power per GB", "value": 14.6, "unit": "W/GB" }, { "label": "R9700 price for per-GB parity with a $1,528 3090", "value": 2037, "unit": "USD" } ], "sources": [ "https://gpusold.com/", "https://compsniper.com/worth/rtx-3090", "https://www.pcprice.watch/search/RTX3090/US/GPU", "https://vga.live/gpus/radeon-ai-pro-r9700", "https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation", "https://rtx.live/gpus/geforce-rtx-5090", "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html" ], "next": "Add a VRAM-per-dollar table to the buying pages, refreshed with the price data; for a two-card purchase, wait for the R9700 to return to about $1,600." }, { "id": "2026-10-27-gfx1201-upstream-bugs", "date": "2026-10-27T04:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 upstream bug check: 15 RDNA4 fixes landed in 30 days, but the ROCm llama.cpp hang that needs a reboot still fails on the newest driver stack", "summary": "We searched 10 GitHub repos Radeon local-AI users rely on for issues naming gfx1201 or R9700. 15 RDNA4-titled issues were closed as fixed in the last 30 days, including two Tensile GEMM page faults and a ~12% Vulkan prefill regression in llama.cpp. Still open: a ROCm-only llama.cpp generation hang that resets the GPU (Vulkan passes), a cross-stream wait penalty whose size depends on MES firmware, a GFXOFF-related idle power cost, a vLLM TP=2 deadlock, and a vLLM 0.28 FP8 kernel choice that slows RDNA4 decode.", "details": "TheRock #9279: ROCm llama.cpp hang leading to MES REMOVE_QUEUE failure, GPU reset and PSP error; on 2026-10-22 a reporter showed it still fails on a TheRock 10.2 nightly with amdgpu-dkms and firmware 31.60, including a pure dual-R9700 layer-split run, while Vulkan passes. TheRock #9297: tiny kernels slow down when two or more other streams hold pending waits; reported penalty by MES firmware is +7 to +39 us at 0x89, +5 to +17 us at 0x91 and about +9 us at 0x93, and GPU_MAX_HW_QUEUES=2 or 1 removes it. TheRock #9245: an AMD contributor traced the extra idle draw to the kernel disabling GFXOFF on GC 12 once a compute queue exists (Linux commit 90505894c4ed); the reporter measured about 5-6 W more per card than Vulkan. vLLM #57838: from v0.28.0 a RowWise FP8 kernel is selected on RDNA4 and the report measures 5-24% slower decode than v0.27.1's choice. llama.cpp #28093: with an AMD iGPU enabled next to an R9700, Vulkan llama-server reserved most host RAM at idle; it was closed as stale, not fixed. Housekeeping: the ROCm/ROCm repo now redirects to ROCm/legacy-rocm-build and GitHub issue search on the old name returns an error, so new ROCm reports go to ROCm/TheRock.", "setup": "GitHub issue search API, 2026-10-27, terms gfx1201 or \"R9700\", repos: ROCm/TheRock, rocm-systems, rocm-libraries, legacy-rocm-build, llama.cpp, vLLM, PyTorch, Lemonade, Ollama, ComfyUI. Counts include issues that only mention the card in an environment dump.", "metrics": [ { "label": "RDNA4-titled issues closed as fixed, last 30 days", "value": 15, "unit": "issues" }, { "label": "Open issues mentioning gfx1201/R9700 in ROCm/TheRock", "value": 104, "unit": "issues" }, { "label": "Open in llama.cpp", "value": 35, "unit": "issues" }, { "label": "Open in vLLM", "value": 40, "unit": "issues" }, { "label": "Cross-stream wait penalty at MES 0x93 (reported)", "value": 9, "unit": "us per small kernel" } ], "sources": [ "https://github.com/ROCm/TheRock/issues/9279", "https://github.com/ROCm/TheRock/issues/9297", "https://github.com/ROCm/TheRock/issues/9245", "https://github.com/ROCm/TheRock/issues/9228", "https://github.com/vllm-project/vllm/issues/40980", "https://github.com/vllm-project/vllm/issues/57838", "https://github.com/ggml-org/llama.cpp/issues/28093", "https://github.com/ggml-org/llama.cpp/issues/29892", "https://github.com/ROCm/rocm-libraries/issues/12346", "https://github.com/ROCm/rocm-libraries/issues/7992" ], "next": "Check our own cards' MES firmware, launcher environment and iGPU state against this list, rerun the search weekly, and add an open-upstream-bugs panel to /rocm-problems." }, { "id": "2026-10-27-new-model-licenses", "date": "2026-10-27T03:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Read the LICENSE, not the tag: 1 in 4 new open-weight models had no usable license label, and Qwen3.8-Flash-Next's terms are stricter than its 2.4T sibling's", "summary": "Of 88 open-weight models released in the last 90 days, 20 were tagged only 'other' or nothing on Hugging Face. After reading each LICENSE: 62 open, 7 open with conditions, 5 commercial only under a revenue cap, 9 non-commercial or research only, 1 not for EU use, 4 still unclear. Personal and home use is fine for every Flash-class model we run.", "details": "Qwen Community License 1.0 (Qwen3.8-Flash-Next): free for personal and internal use, but any company running a Model-as-a-Service or coding/office AI-assistant business needs a separate Qwen license for commercial use, with no revenue floor; the Qwen3.8-Max license (2.4T-A95B) only requires that above US$50M revenue in 12 months. GLM-5.3: MIT-style, only Model-as-a-Service firms above US$10B revenue need a Z.AI security review. MiniMax-M3.1-Flash: the base grant is non-commercial; commercial use needs a 'Built with MiniMax M3.1-Flash' notice and a one-time email, and written approval above US$20M/year. LiquidAI's LFM Open License 1.0 (5 models): commercial use only for entities under US$10M annual revenue. Tencent ContextPilot 8B/14B/E4B ship Apache 2.0 text with an added clause limiting use to scientific research. Youtu-Parsing-Omni is 'not intended for use within the European Union'. Also, 36 of 61 repos tagged apache-2.0, mit or llama3 had no LICENSE file at the repo root; none of the 25 that did added extra terms. Plain-language reading, not legal advice.", "setup": "amdai.net /new-models snapshot of 2026-10-26 (88 models); Hugging Face model API license, license_name and base_model fields; LICENSE files read 2026-10-27.", "metrics": [ { "label": "Models with license 'other' or none", "value": 20, "unit": "of 88" }, { "label": "Still unclear after reading LICENSE files", "value": 4, "unit": "of 88" }, { "label": "Non-commercial or research only", "value": 9, "unit": "of 88" }, { "label": "Commercial only under a revenue cap", "value": 5, "unit": "of 88" }, { "label": "Permissive-tagged repos with no LICENSE file", "value": 36, "unit": "of 61" } ], "sources": [ "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE", "https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B/blob/main/LICENSE", "https://huggingface.co/zai-org/GLM-5.3/blob/main/LICENSE", "https://huggingface.co/MiniMaxAI/MiniMax-M3.1-Flash/blob/main/LICENSE", "https://huggingface.co/LiquidAI/LFM2.5-24B-A2B/blob/main/LICENSE", "https://huggingface.co/tencent/ContextPilot-8B/blob/main/LICENSE", "https://huggingface.co/tencent/Youtu-Parsing-Omni/blob/main/LICENSE" ], "next": "Add a filterable 'Use' column (open, conditions, revenue cap, non-commercial) with a LICENSE link to /new-models." }, { "id": "2026-10-27-coding-cli-prompt-tax", "date": "2026-10-27T01:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Coding CLIs on a local llama-server: 0.7K to 19K tokens before your first word, and one background call that forces a full re-read", "summary": "We captured what five coding CLIs send to a local endpoint. First-turn prompts: Claude Code 19,384 tokens, Qwen Code 15,091 (plus a 7,084-token memory call every turn), Codex CLI 8,343, OpenCode 7,158, Aider 749. Replayed on a real one-slot llama-server, Qwen Code's memory call evicted the session when the RAM prompt cache was off: turn 2 re-read 15,602 tokens instead of 21.", "details": "Method: each CLI pointed at a recording mock of /v1/chat/completions, /v1/messages and /v1/responses in a 3-file git repo, prompt 'Say done.'; tokens counted with the Qwen3 tokenizer over system text, tool schemas and messages (chat-template tokens excluded). Claude Code's 19,384 tokens are mostly tool schemas (16,465 for 24 tools). Two fresh runs of every CLI produced identical prompts; Claude Code 2.1.197's attribution block was stable, unlike older builds that changed it per request. The captured request bodies were then replayed in order on llama-server b11912 (CPU, Qwen3.5-0.8B Q8_0, a hybrid model, -np 1). Qwen Code turn 2: 21 tokens re-read (0.29 s) with the default --cache-ram, 15,602 tokens (31.0 s on this CPU) with --cache-ram 0, and 21 tokens with --cache-ram 0 when its setting memory.enableManagedAutoMemory is false. OpenCode's title call runs only on the first turn, before the main request, so turn 2 re-read 21 tokens either way. Prefill seconds on Radeon will differ; the token counts are what carry over. Also note llama-server's default port changed from 8080 to 9931 (merged 2026-10-09), so guides that assume 8080 need --port.", "setup": { "engine": "llama-server", "engine_version": "b11912 (CPU build)", "model": "Qwen3.5-0.8B", "quant": "Q8_0", "context": "65536", "flags": "-np 1, --reasoning off; arms: default --cache-ram, --cache-ram 0", "clients": "Claude Code 2.1.197, Qwen Code 0.15.10, Codex CLI 0.164.0, OpenCode 1.19.4, Aider 0.86.2" }, "metrics": [ { "label": "Claude Code first-turn prompt", "value": 19384, "unit": "tokens" }, { "label": "Qwen Code first-turn prompt", "value": 15091, "unit": "tokens" }, { "label": "Codex CLI first-turn prompt", "value": 8343, "unit": "tokens" }, { "label": "OpenCode first-turn prompt", "value": 7158, "unit": "tokens" }, { "label": "Aider first-turn prompt", "value": 749, "unit": "tokens" }, { "label": "Qwen Code turn 2 re-read, --cache-ram 0", "value": 15602, "unit": "tokens" }, { "label": "Qwen Code turn 2 re-read, default cache or memory off", "value": 21, "unit": "tokens" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "https://github.com/ggml-org/llama.cpp/pull/30159", "https://github.com/ggml-org/llama.cpp/issues/20003", "https://code.claude.com/docs/en/llm-gateway-protocol", "https://opencode.ai/docs/providers/" ], "next": "Repeat the turn-by-turn count on the dual-R9700 Flash and single-R9700 35B lanes, and publish ready-to-use configs for each CLI." }, { "id": "2026-10-26-hip-build-flags-rccl", "date": "2026-10-26T23:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp on 2x R9700: build with RCCL off for batch-1 tensor split, and three build flags that no longer do anything", "summary": "Since llama.cpp PR #27825 (merged 2026-09-15), a HIP build with GGML_HIP_RCCL=OFF (the default) uses a built-in two-GPU AllReduce for --split-mode tensor. On 2x R9700 it beat the old fallback by +17.5% prefill and +8.3% decode, and in published RCCL-on vs off tests decode was equal or slightly faster without RCCL while RCCL only helped long prefills.", "details": "Cited results: 2x R9700 (PCIe 4.0 x8/x8, ROCm 7.14), gemma4 31B Q6_K, -sm tensor, pp2048/tg512 at depth 4096: 959.9/30.0 t/s before vs 1127.5/32.5 t/s after #27825; that tester also reported frequent GPU hangs with RCCL enabled. 2x R9700, Qwen3.8-27B UD-Q4_K_XL server: RCCL on vs off gave +3.6% to +10.1% prefill at 60K-120K and identical decode at 1-3 parallel slots. 2x 7900 XTX, Qwen3.8-27B Q8_0: RCCL ~9% faster prefill, built-in AllReduce ~2.5-2.8% faster decode, also at 100K depth. Layer split (-sm layer) has no AllReduce and is unaffected. Dead or changed flags on master (2026-10-26): GGML_HIP_ROCWMMA_FATTN was removed 2026-07-24 and is silently ignored; -ffast-math and -funsafe-math were dropped from HIP builds by mid-August 2026 (unsafe math broke MTP output identity), so builds from July differ numerically; GGML_CUDA_FA_ALL_QUANTS became GGML_CUDA_FA_QUANTS, whose default covers only f16, bf16, q8_0 and q4_0 matched pairs. A mixed KV cache such as -ctk q8_0 -ctv f16 then converts K and V to f16 on every FlashAttention call, with a single log warning. Head-dim-256 models still lack the MMA FlashAttention path on RDNA4 (PR #26419 open; +21.8% prefill at 127K depth on one R9700 in its tests).", "setup": { "engine": "llama.cpp (HIP/ROCm)", "engine_version": "master 7aca0d2 (2026-10-26) source read; results from cited testers", "gpus": "2x Radeon AI PRO R9700 (and 2x RX 7900 XTX for the RCCL decode comparison)", "flags": "-sm tensor, -fa on, GGML_HIP_RCCL OFF vs ON" }, "metrics": [ { "label": "Prefill gain, built-in AllReduce vs old fallback (2x R9700, 31B Q6_K)", "value": 17.5, "unit": "%" }, { "label": "Decode gain, built-in AllReduce vs old fallback (2x R9700, 31B Q6_K)", "value": 8.3, "unit": "%" }, { "label": "Decode, RCCL on vs off (2x R9700, 27B Q4, 60K-120K)", "value": 0.0, "unit": "%" }, { "label": "Max prefill gain, RCCL on vs off (2x R9700, 27B Q4, 60K-120K)", "value": 10.1, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/27825", "https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5535620413", "https://github.com/ggml-org/llama.cpp/pull/27825#issuecomment-5699240655", "https://github.com/ggml-org/llama.cpp/pull/26046", "https://github.com/ggml-org/llama.cpp/pull/26696", "https://github.com/ggml-org/llama.cpp/pull/28079", "https://github.com/ggml-org/llama.cpp/pull/26419" ], "next": "Audit the build cache of each local lane, and A/B RCCL off vs on only if a tensor-split lane is used." }, { "id": "2026-10-26-ddr5-dimm-heat-hybrid-decode", "date": "2026-10-26T22:15:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Hot DDR5 can quietly cost hybrid GPU+RAM decode up to ~10%, and non-ECC sticks never report errors", "summary": "When a big MoE model keeps its experts in DDR5 next to two R9700s, the DIMMs work hard for hours. DDR5 doubles its refresh rate above 85 C (JEDEC refresh interval 3.9 us drops to 1.95 us). For 64 GB sticks built from 32 Gb chips, that takes the time a rank spends refreshing from about 10.5% to about 21%, so bandwidth-bound work can slow by up to ~13%. For hybrid decode we estimate up to ~7-11% fewer tokens per second once sticks pass 85 C, depending on how much of each token's time is spent reading DDR5 (an upper-bound estimate). Linux can read each stick's temperature with the spd5118 driver, so this is easy to check.", "details": "Reading temps: the spd5118 hwmon driver (Linux 6.11 and later) exposes temp1_input, temp1_crit and alarm files per DIMM; on AMD boards the SMBus driver registers the SPD hubs automatically, and an AM5 example shows sensors named spd5118-i2c-1-51 and -53 with a default 'high' of 55 C (not a problem by itself) and 'crit' of 85 C. Field reports: one owner measured DDR5-6400 RDIMMs at 95 C and above, cooled them under 60 C with shrouds and fans, and saw about 10% faster inference; a dual-3090 hybrid build reported decode falling from 17 to 8-12 tok/s after minutes from DIMM heat. Stability: consumer AM5 sticks are non-ECC, and DDR5's on-die ECC fixes bits inside the chip without telling the OS, so EDAC/rasdaemon show nothing; a marginal four-stick speed appears only as rare crashes, kernel oopses, machine-check events or wrong output. A published 4x64 GB AM5 setup was 12 h y-cruncher-stable at DDR5-4800 (36-40-40-78, about 56 GB/s), and the same settings failed after about 7 h on an older AGESA. On Linux, y-cruncher's VT3, FFTv4 and N63 components plus stressapptest are the practical tests; TestMem5 and Karhu are Windows tools.", "setup": "Calculation from JEDEC refresh timing and published reports; no measurement on our desktop yet.", "metrics": [ { "label": "DDR5 refresh interval below 85 C", "value": 3.9, "unit": "us" }, { "label": "DDR5 refresh interval at 85-95 C", "value": 1.95, "unit": "us" }, { "label": "Time refreshing, 32 Gb chips, below 85 C (calc)", "value": 10.5, "unit": "%" }, { "label": "Time refreshing, 32 Gb chips, 85-95 C (calc)", "value": 21.0, "unit": "%" }, { "label": "Hybrid decode loss above 85 C, upper-bound estimate", "value": "7-11", "unit": "%" }, { "label": "Inference gain after cooling RDIMMs from 95 C+ (field report)", "value": 10, "unit": "%" } ], "sources": [ "https://docs.kernel.org/hwmon/spd5118.html", "https://cateee.net/lkddb/web-lkddb/SENSORS_SPD5118.html", "https://lkml.indiana.edu/2405.3/07725.html", "https://www.phoronix.com/forums/forum/hardware/motherboards-chipsets/1512681-linux-6-14-to-add-sensor-monitoring-for-a-~-180-asrock-am5-motherboard/page2", "https://hivebook.wiki/wiki/dram-row-buffers-and-bank-conflicts-activate-rate-tfaw-and-refresh", "https://semiengineering.com/dram-thermal-issues-reach-crisis-point/", "https://www.reddit.com/r/LocalLLaMA/comments/1p89j2t/today_i_learned_that_ddr5_can_throttle_itself_at/", "https://forum.level1techs.com/t/ddr5-4-dimms-on-am5-whats-working-whats-not/197153?page=14", "https://pakt.cc/en/ddr5/testing/" ], "next": "Log DIMM temperatures next to tok/s during a 30-minute hybrid decode on our dual-R9700 desktop with four 64 GB sticks, and test whether better airflow changes the speed." }, { "id": "2026-10-26-four-r9700-two-desktops-power-budget", "date": "2026-10-26T21:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Power budget for four R9700s in two desktops: one 15 A circuit is not enough, even at the lowest power cap", "summary": "Before buying a second desktop for two more R9700s, we built a power budget from published measurements. Under full compute an R9700 has been measured at about 390-400 W at the card (spec TBP 300 W), while decode-bound llama.cpp is much lower: about 340-353 W for two cards together in layer split. With all four cards at 300 W, the two desktops draw an estimated ~1,780 W at the wall; even at the lowest allowed cap (210 W) it is ~1,640 W. A 15 A, 120 V circuit may carry only 1,440 W continuously (the 80% rule for loads over 3 hours), so the two boxes need separate circuits, or one 20 A circuit with cards capped at 260 W or less.", "details": "Card numbers: Igor's Lab (compute/AI full load ~390-400 W), PCGH (only ~1.5 W from the PCIe slot, so nearly all power rides the single 12V-2x6 connector), Puget Systems (two cards, sysfs power: llama.cpp Q4_K_M layer split 339 W at 1 request, 353 W at 8; vLLM TP2 FP8 310-315 W at 1 request, up to 510 W at 4-8). CPU: Ryzen 9 9950X measured at its 200 W PPT, 210-235 W with PBO. Power cap results from a published llama.cpp sweep on one R9700: 260 W cost 0.0% decode and 4.0% prefill; 210 W cost 1.9% decode and 11.2% prefill. Rest of each system (board, four DDR5 sticks, NVMe, fans, NIC) is our estimate of 60-100 W, and wall power assumes an 80 PLUS Platinum supply at 92%. Per box: two cards at 400 W plus a PBO CPU is ~1,135 W DC, which is 76% of a 1,500 W supply, 95% of 1,200 W and over 100% of 1,000 W. A UPS should be sized on input watts times 1.25 and be pure sine wave for active-PFC supplies; one desktop in hybrid Flash decode needs at least ~925 W / ~1,030 VA by that rule.", "setup": "Calculation from published measurements; rest-of-system draw estimated, not measured.", "metrics": [ { "label": "4 cards at 300 W TBP, both desktops, wall (estimate)", "value": 1782, "unit": "W" }, { "label": "4 cards capped at 210 W, worst case, wall (estimate)", "value": 1642, "unit": "W" }, { "label": "15 A / 120 V circuit, continuous limit", "value": 1440, "unit": "W" }, { "label": "R9700 full compute load, measured at card (Igor's Lab)", "value": 400, "unit": "W" }, { "label": "Two R9700s, llama.cpp layer split, 1 request (Puget)", "value": 339, "unit": "W" }, { "label": "Decode loss at 260 W cap (published sweep)", "value": 0.0, "unit": "%" } ], "sources": [ "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html", "https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/", "https://www.pcgameshardware.de/Radeon-AI-Pro-R9700-Grafikkarte-281741/Tests/Preis-Gaming-Benchmark-Release-Review-1486284/5/", "https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/", "https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/", "https://www.tomshardware.com/pc-components/cpus/amd-ryzen-9-9950x-cpu-review/4", "https://www.infobytes.de/pages/atx3.1powerrev2.1.pdf", "https://electricaltoolbox.com/continuous-load-calculator/", "https://www.se.com/us/en/faqs/FA158939/" ], "next": "Measure real CPU, GPU and rest-of-system draw per workload on our dual-R9700 desktop to replace the estimates, and pick the second desktop's supply (1,200-1,300 W with two native 12V-2x6 cables, or 1,000 W only with cards capped at 260 W)." }, { "id": "2026-10-26-llama-bench-noise-aa-test", "date": "2026-10-26T19:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama-bench A/A test: identical runs differed by up to 20%, and one run's mean +- stddev called identical runs 'different' 9 times in 15", "summary": "Before trusting a 3-5% win between two configs, we ran the same llama-bench command 6 times in a row. Inside one run, repetitions varied about 5%, but the medians of whole runs varied 6-7%, and the first run after loading was 12-16% slower than the rest. Comparing any two of the identical runs, a 95% interval built from one run's repetitions wrongly flagged a difference in 9 of 15 pairs. llama-bench's own tools don't guard against this: the generation warmup is a single token, repetitions run back-to-back by default, and compare-llama-bench.py prints a plain ratio of averages with no spread.", "details": "Measured on a shared 8-vCPU cloud VM with a CPU build, not a Radeon card, so the size of the noise here is a worst case; the pattern (repetitions within one invocation share warm caches and clock state, so their spread understates run-to-run noise) is general. Source facts are from llama.cpp master 7aca0d2: llama-bench defaults to -r 5 and --delay 0; its warmup runs one full prompt for pp tests but test_gen with 1 token for tg tests; compare-llama-bench.py compares AVG(avg_ts) only. What we do now: an A/A run of every config before an A/B sweep to measure its noise floor, at least 3 invocations per arm interleaved A B A B, one throwaway invocation after each model load, and a hierarchical bootstrap of the median ratio (resample invocations, then repetitions) with a minimum effect size, instead of mean +- stddev from one run. With only 3 vs 3 identical runs, one split still showed +9.9% (interval 1.003-1.187) because the slow first run landed on one side, which is why the throwaway run matters.", "setup": "llama.cpp b11912 CPU build, Qwen3.5-0.8B Q8_0, llama-bench -p 512 -n 64 -r 10 -t 4, 6 identical invocations plus one with --no-warmup, shared 8-vCPU cloud VM", "metrics": [ { "label": "Within-run CV, tg64 / pp512", "value": "5.2 / 5.1", "unit": "%" }, { "label": "Between-run CV of medians, tg64 / pp512", "value": "6.1 / 7.3", "unit": "%" }, { "label": "Largest gap between identical runs, tg64 / pp512", "value": "20.4 / 19.2", "unit": "%" }, { "label": "First run vs later runs, pp512", "value": "357 vs 412-426", "unit": "tok/s" }, { "label": "Identical pairs flagged 'different' by a one-run 95% CI", "value": "9 of 15", "unit": "pairs" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md", "https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/scripts/compare-llama-bench.py" ], "next": "Run the same A/A test on dual R9700s for the Flash lane and on one R9700 for the 35B lane to get the real Radeon noise floor, then use it as the minimum effect size for the Q4/Q5/Q6 dual-GPU sweep." }, { "id": "2026-10-26-openwebui-background-tasks-prompt-cache", "date": "2026-10-26T18:35:00-05:00", "project": "idea-lab", "kind": "result", "title": "Open WebUI in front of llama-server: with --cache-ram 0, its background calls force a full re-prefill of the chat on every turn", "summary": "Open WebUI 0.12.4 sends extra model calls per chat turn (follow-up suggestions every turn, plus title and tags on the first), each a fresh one-message prompt that shares no prefix with the conversation. On a one-slot llama-server they replace the slot's state. With the default host prompt cache (--cache-ram 8192 MiB) the conversation is saved and restored and each later turn re-processes only about 70 tokens; with --cache-ram 0, or a cache smaller than one conversation's saved state, every turn re-processed the whole 11K-token chat (about 40x the prefill time in this test).", "details": "Method: Open WebUI driven through its API against a logging mock backend to capture exactly what it sends, then those task prompts replayed against a real llama-server (llama.cpp b11912 CPU build, Qwen3.5-0.8B Q8_0, a hybrid Gated DeltaNet model like the Flash-class models) for a 5-turn, ~11.1K-token chat. Measured on an 8-vCPU cloud Xeon, not a Radeon card; the re-processed token counts do not depend on hardware, the seconds do. Captured calls: task prompts start with '### Task:' and carry a truncated history (follow-ups: first user message plus last 4, about 1,000 characters each); title max_tokens is 1000, follow-up and tags have no limit, and by default all run on the chat's own model. llama.cpp's cache entry for the conversation was 150.6 MiB of state, 208.4 MiB with its 3 context checkpoints, and saving plus restoring took 209 ms; each task prompt adds its own 27-30 MiB entry to the same cache. Auto slots (4 slots, unified KV, the llama-server default when -np is not set) and -np 2 also kept reuse at about 70 tokens; -np 2 without unified KV halves each slot's context.", "setup": "llama.cpp b11912 (CPU), Qwen3.5-0.8B Q8_0, -c 32768, 8 threads; Open WebUI 0.12.4 with default task settings", "metrics": [ { "label": "Tokens re-processed per later turn, default --cache-ram", "value": "68-70", "unit": "tokens" }, { "label": "Tokens re-processed per later turn, --cache-ram 0", "value": "11,201-11,396", "unit": "tokens" }, { "label": "Prefill per later turn, default --cache-ram (CPU test box)", "value": "0.52-0.56", "unit": "s" }, { "label": "Prefill per later turn, --cache-ram 0 (CPU test box)", "value": "19.8-22.9", "unit": "s" }, { "label": "Prompt cache entry for the 11.2K-token chat incl. checkpoints", "value": "208.4", "unit": "MiB" }, { "label": "Extra model calls per later turn (Open WebUI defaults)", "value": "1", "unit": "calls" } ], "sources": [ "https://github.com/open-webui/docs/blob/main/docs/features/administration/task-models.mdx", "https://github.com/open-webui/open-webui/issues/638", "https://github.com/ggml-org/llama.cpp/pull/16391", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp", "https://github.com/ggml-org/llama.cpp/releases/tag/b11912" ], "next": "Keep --cache-ram on and larger than one long conversation plus its checkpoints (check the server log's 'total state size' line), point Open WebUI's task model at a small non-reasoning model or turn follow-ups and tags off, and cap task max_tokens. Next: measure the cache entry size and restore time on a Radeon R9700 lane at 60K and 200K tokens." }, { "id": "2026-10-26-glm-flash-chat-template-render", "date": "2026-10-26T17:30:00-05:00", "project": "idea-lab", "kind": "result", "title": "GLM-5.3-Flash GGUF chat template is two upstream fixes behind: up to 17 s of template rendering per request with many parallel tool results in llama.cpp", "summary": "The chat template embedded in the current Unsloth GLM-5.3-Flash GGUF matches the upstream 2026-08-27 version and misses the 2026-10-15 upstream rewrite of tool-result reordering. Rendered with llama.cpp's own Jinja engine, one block of 100 parallel tool results takes 1.14 s with the embedded template vs 31 ms with the new one (400 results: 17.0 s vs 0.19 s), with byte-identical output. Fix without re-downloading: pass the upstream template with --chat-template-file.", "details": "Method: templates read from the GGUF headers and from upstream at each commit, rendered with llama.cpp's Jinja engine (common/jinja at master 4244ad7) and with Python Jinja2 3.1.6. Single-threaded, measured on an 8-vCPU cloud Xeon, not the R9700 desktop, so absolute times will differ; the ratio is the point. Long agent sessions: 60 turns x 4 calls 176 vs 81 ms; 30 x 16 910 vs 127 ms; 10 x 64 4,356 vs 175 ms. llama.cpp's engine was about 5x slower than Jinja2 on the same template. The 2026-09-04 upstream 'early break' change alone gave no speedup. 200 randomized conversations (shuffled, missing and duplicate tool ids, multi-turn) rendered byte-identical. Side finding on Qwen templates: the Unsloth GGUF templates for Qwen3.6-35B-A3B and Qwen3.8-Flash-Next accept a developer role, two leading system messages and a conversation with no user message, where upstream raises an error; but a system message after the first turn makes the Flash-Next GGUF template raise and the 35B GGUF template silently drop it. Keep the GGUF Qwen templates and avoid late system messages.", "setup": { "model": "GLM-5.3-Flash (also Qwen3.6-35B-A3B, Qwen3.8-Flash-Next templates)", "quant": "UD-Q6_K_XL header (template is quant-independent)", "engine": "llama.cpp Jinja engine + Python Jinja2", "engine_version": "llama.cpp master 4244ad7; Jinja2 3.1.6", "gpus": "none (CPU-only template rendering)", "context": "n/a", "flags": "--chat-template-file" }, "metrics": [ { "label": "Render, 100 parallel tool results, GGUF template", "value": 1140, "unit": "ms" }, { "label": "Render, 100 parallel tool results, upstream 2026-10-15 template", "value": 31, "unit": "ms" }, { "label": "Render, 400 parallel tool results, GGUF template", "value": 17001, "unit": "ms" }, { "label": "Render, 400 parallel tool results, upstream 2026-10-15 template", "value": 188, "unit": "ms" } ], "sources": [ "https://huggingface.co/zai-org/GLM-5.3-Flash/discussions/63", "https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF", "https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/chat_template.jinja", "https://github.com/ggml-org/llama.cpp/tree/4244ad7acd98328c1f04f1c2201d571f643370bb/common/jinja" ], "next": "Time /apply-template on the R9700 desktop with both templates and fold --chat-template-file into the GLM preset if it saves time at a normal tool fan-out." }, { "id": "2026-10-26-r9700-street-price", "date": "2026-10-26T16:03:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 street price today: $1,699.99, 31% over MSRP, and only in store", "summary": "Across the six US R9700 32 GB models, the cheapest new card in stock is $1,699.99 (Gigabyte AI TOP and XFX at Micro Center, in-store pickup), 31% above AMD's $1,299 MSRP. The lowest price seen since Sep 29 is $1,599.99. Amazon's in-stock new offers run $2,399.99 to $2,680.81.", "details": "Snapshot taken Oct 26, 2026 around 4 PM CT from vga.live's public card pages, before tax. ASRock Creator, ASUS Turbo and Sapphire are $1,799.99 at Micro Center; PowerColor is only in stock on Amazon at $2,680.81. Newegg (ASRock, $1,699.99) and B&H (Gigabyte, $1,899.95) showed out of stock. A two-card setup (64 GB VRAM) costs $3,399.98 before tax today, $200 above the recent low. Prices move daily; check before buying.", "setup": { "gpus": "Radeon AI PRO R9700 32 GB (6 board-partner models)", "region": "US, new condition, in stock" }, "metrics": [ { "label": "Cheapest new R9700 in stock", "value": 1699.99, "unit": "USD" }, { "label": "Premium over $1,299 MSRP", "value": 30.9, "unit": "%" }, { "label": "Lowest price since Sep 29", "value": 1599.99, "unit": "USD" }, { "label": "Cheapest Amazon in-stock new offer", "value": 2399.99, "unit": "USD" }, { "label": "Two cards today, before tax", "value": 3399.98, "unit": "USD" } ], "sources": [ "https://vga.live/gpus/radeon-ai-pro-r9700", "https://vga.live/cards/gigabyte-amd-radeon-ai-pro-r9700-top-single-fan-ai-workstation", "https://vga.live/cards/xfx-amd-radeon-ai-pro-r9700-single-fan-32gb-gddr6", "https://vga.live/cards/asrock-amd-radeon-ai-pro-r9700-creator-single-fan-ai-workstation", "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro.html" ], "next": "Track the cheapest new price daily and flag when it returns to the $1,599.99 floor." }, { "id": "2026-10-26-strata-mtp-q2-only-second-card", "date": "2026-10-26T14:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Strata's MTP draft layer only loads Q2_0 experts, so a Q4+-only R9700 setup loses drafts and the RDNA4 pipeline-windows split gain", "summary": "In Strata main (2026-10-25), the runtime drafter sizes its 512 experts as kernels::cpu::BLOB, which is the Q2_0 layout (1,382,400 bytes per expert, 707,788,800 for the file). mtp_pack.py can write Q4_0 or Q8_0 draft experts, but nothing loads them (open issue #3728), and --mtp-q4 only requantizes the dense head, not the experts. Anyone who keeps every file at 4-bit or more therefore runs Strata with no draft layer. On a layer split that also turns off pipeline windows, the RDNA4 default that gave +21% code decode on 2x R9700 (103 to 125 tok/s) in the maintainer's table.", "details": "Cited, not measured here: the maintainer's R9700 numbers (one card at 225 W, EPYC 7452 with 8-channel DDR4 at 106 GB/s, every expert in RAM, 40K context, MTP on) are UD-Q4 63-64 story / 85-88 code tok/s, UD-Q5 57-60 / 81-83, UD-Q6 54 / 76. Adding a second card by layer split moved UD-Q4 63 to 64 story and 88 to 96 code, UD-Q5 57 to 60 and 83 to 94; 32K prompts gain more at 3-4 cards (UD-Q4 1,110 to 2,381 tok/s on four). No two-R9700 number without a draft layer has been published. Second-card options that don't need a draft layer: layer split with per-card expert caches, STRATA_SPLIT_OVERLAP=1 (2x 7900 XTX code 53 to 70.1, about one card's speed), STRATA_STAGE_TRIM=1, helper expert caches with --remote-expert-opt (+63-132% over the plain helper path, measured only on 2x RTX 4090) and --peer-device. Also worth a deliberate choice: the default STRATA_ROUTE_TAIL_SKIP=7 moves UD-Q4's KL vs Q8_0 from 0.056 to 0.086 for +4-10% decode. Our own Strata Flash lane (UD-Q6_K_XL, one R9700, no draft layer, two-channel DDR5, engine 0.1.42) measured 32.6 tok/s; upstream is now at engine 0.1.50 with RDNA4 matrix-core prompt kernels.", "setup": "Source reading of Strata main at commit 4b9e05a; maintainer-published R9700 tables. Our measured reference: local desktop, 1x R9700, Qwen3.8-Flash UD-Q6_K_XL, Strata engine 0.1.42, no draft layer.", "metrics": [ { "label": "Q2_0 draft expert size (per expert)", "value": "1,382,400", "unit": "bytes" }, { "label": "Pipeline windows, 2x R9700 code decode (with MTP, cited)", "value": "103 -> 125", "unit": "tok/s" }, { "label": "UD-Q5 1 -> 2 R9700 code decode (with MTP, cited)", "value": "83 -> 94", "unit": "tok/s" }, { "label": "UD-Q6 1 R9700 story / code (with MTP, 8ch DDR4, cited)", "value": "54 / 76", "unit": "tok/s" }, { "label": "Our UD-Q6 lane, 1 R9700, no draft layer (measured)", "value": "32.6", "unit": "tok/s" } ], "sources": [ "https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/src/core/mtp.cpp", "https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/include/strata/kernels/cpu/expert.hpp", "https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/MULTI_GPU.md", "https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/SECOND_GPU.md", "https://github.com/Niko1221/Strata/blob/4b9e05a2a806a3fe24f7b2560d98626c37dade35/docs/QUANTS.md", "https://github.com/Niko1221/Strata/issues/3728" ], "next": "Test on the local desktop: engine 0.1.42 vs 0.1.50 on one card, then layer split, split overlap and helper-cache arms with no draft layer, gated by a correctness check. A Q8_0 or Q4_0 draft-expert runtime (#3728) would unlock drafts under a Q4+ rule." }, { "id": "2026-10-26-vllm-tp2-allreduce-rccl-only", "date": "2026-10-26T11:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "vLLM TP2 on two R9700s: every all-reduce goes through RCCL, and batch-1 decode pays it ~80-96 times per token", "summary": "In current vLLM source, the custom all-reduce, QuickReduce and AITER all-reduce paths are enabled only on MI300-class GPUs; on gfx1201 vLLM forces disable_custom_all_reduce on, so every tensor-parallel all-reduce is an RCCL call (this is why --disable-custom-all-reduce 'has no effect' on R9700). With two all-reduces per layer, Qwen3.6-35B-A3B makes 80 per token (4 KiB each) and Qwen3.8-Flash-Next 96 (5 KiB). The only published R9700 rccl-tests table (4 cards, P2P off) is flat at 31-36 us up to 32 KiB, which is about 2.9 and 3.4 ms of communication per token, before any compute.", "details": "Estimate, not a measurement: if vLLM matched llama.cpp's ~100.8 tok/s single-card decode for the 35B, an 80-100% efficient TP2 split would land near 110-128 tok/s, so TP2 is mainly a capacity tool, not a big speed win, for models that already fit on one card. RCCL's low-latency LL protocol (auto-picked for AllReduce up to 16 KiB per rank on gfx120x) is what deadlocks on RDNA4 (ROCm rocm-systems #5480, open); the NCCL_PROTO=Simple workaround avoids it but likely costs small-message latency. AMD's patch to force Simple by default was closed unmerged on 2026-10-20 after its 4-GPU hang turned out to be a hipBLASLt issue. Published R9700 TP numbers (e.g. Qwen3.6-35B-A3B-FP8 TP2 17.4 tok/s) were run with --enforce-eager. Two platform traps: with iommu=pt, cross-process peer IPC on PCIe RDNA4 returns success but reads 0xFF and drops writes; and RCCL collectives currently fail on an R9700 attached through the chipset ('AQL dispatch failed!', rocm-systems #13616, open).", "setup": { "model": "Qwen3.6-35B-A3B / Qwen3.8-Flash-Next", "quant": "n/a (communication analysis)", "engine": "vLLM (main, 2026-10-26 source) + RCCL", "engine_version": "vLLM main 7f56f45", "gpus": "2x Radeon AI PRO R9700", "context": "batch 1 decode", "flags": "tensor-parallel-size 2, NCCL_PROTO=Simple" }, "metrics": [ { "label": "All-reduces per token, Qwen3.6-35B-A3B", "value": 80, "unit": "" }, { "label": "All-reduces per token, Qwen3.8-Flash-Next", "value": 96, "unit": "" }, { "label": "Published R9700 RCCL all-reduce latency, 8 B-32 KiB", "value": "31-36", "unit": "us" }, { "label": "Comms time per token, 35B (from published table)", "value": 2.9, "unit": "ms" }, { "label": "Comms time per token, Flash-Next (from published table)", "value": 3.4, "unit": "ms" }, { "label": "Estimated TP2 decode ceiling, 35B (estimate)", "value": "110-128", "unit": "tok/s" } ], "sources": [ "https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/platforms/rocm.py", "https://github.com/vllm-project/vllm/blob/7f56f453ede3/vllm/config/parallel.py", "https://github.com/vllm-project/vllm/pull/19744", "https://github.com/ROCm/rccl-tests/issues/162", "https://github.com/ROCm/rocm-systems/issues/5480", "https://github.com/ROCm/rocm-systems/pull/13151", "https://github.com/ROCm/ROCR-Runtime/issues/355", "https://github.com/ROCm/rocm-systems/issues/13616", "https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504" ], "next": "Measure 2-card RCCL all-reduce latency (Simple vs shared-memory path vs a time-limited LL run) and vLLM TP2 decode with HIP graphs vs eager on the local desktop." }, { "id": "2026-10-26-gguf-reupload-drift", "date": "2026-10-26T10:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Stale GGUFs after a re-upload: GLM-5.3-Flash needs one 9 MiB shard, not a 272 GiB re-download, and old Flash-Next MTP heads carry a wrong value", "summary": "On 2026-10-05 Unsloth re-uploaded files in its GLM-5.3-Flash and Qwen3.8-Flash-Next GGUF repos 'due to upstream changes'. We diffed the old and new files from the Hugging Face history. For GLM-5.3-Flash only shard 1 of each quant changed, and it holds metadata only (9.0 MiB, no tensors): the architecture name went from glm5next to glm5-next, the name mainline llama.cpp merged on 2026-09-30, and one key was added (attention.indexer.index_share_mtp = true). A copy downloaded before then can be fixed by replacing that one shard instead of re-downloading 186-272 GiB. For Qwen3.8-Flash-Next the UD-Q4/Q5/Q6 model shards did not change; the three non-shared MTP head files did, by one value: the MTP layer's attention compress ratio went from 0 to 4.", "details": "Method: listed every GGUF with its LFS sha256 at the commit and at its parent via the Hugging Face API, then parsed the GGUF headers of both versions. GLM-5.3-Flash UD-Q6_K_XL shard 1: 40 architecture keys renamed with the architecture, 1 key added, tensor data shards unchanged; the pre-change file's sha256 matches revisions from 2026-08-29 to 2026-09-06. The old glm5next name came from llama.cpp PR #27752, which was closed unmerged; PR #27773 merged support as glm5-next. Flash-Next MTP heads (BF16, Q4_K_M, Q8_0; same size and same 34 tensors): qwen4exp.attention.compress_ratios entry 48 changed 0 -> 4. We have not measured what the old value does to MTP acceptance. Qwen3.6-35B-A3B GGUFs were last re-uploaded 2026-04-20. We wrote a read-only checker that compares a local GGUF's size and header with the current Hugging Face file (or its full sha256 with --full) and labels it current, stale or corrupt; it correctly flagged an old GLM shard, a current shard and a shard with one flipped byte.", "setup": { "model": "GLM-5.3-Flash, Qwen3.8-Flash-Next, Qwen3.6-35B-A3B (Unsloth GGUF)", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header diff on UD-Q6_K_XL shard 1 and MTP Q8_0)", "engine": "llama.cpp (arch names from master)", "engine_version": "master, 2026-10-26", "gpus": "n/a (file analysis)" }, "metrics": [ { "label": "GLM-5.3-Flash shard that changed (metadata only)", "value": 9.0, "unit": "MiB" }, { "label": "GLM-5.3-Flash UD-Q6_K_XL full set", "value": 271.8, "unit": "GiB" }, { "label": "GLM-5.3-Flash UD-Q4_K_XL full set", "value": 186.0, "unit": "GiB" }, { "label": "Metadata keys changed in GLM shard 1", "value": 82, "unit": "keys (40 renamed pairs + arch + 1 new)" }, { "label": "Flash-Next MTP heads changed", "value": 3, "unit": "files" } ], "sources": [ "https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF/commit/a38483c8cd5df544f53d70fb281afe97369d5ab6", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/commit/efc50b7c3195a10eebd8d3e32f83b27b2aeb239d", "https://github.com/ggml-org/llama.cpp/pull/27773", "https://github.com/ggml-org/llama.cpp/pull/27752", "https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/commits/main" ], "next": "Check our own GLM-5.3-Flash and Flash-Next MTP copies with the checker; if stale, swap the 9 MiB shard and A/B MTP acceptance with the old vs new head." }, { "id": "2026-10-26-35b-a3b-mtp-vs-draft-r9700", "date": "2026-10-26T09:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Faster Qwen3.6-35B-A3B on one R9700: use the built-in MTP head, not a draft model, and watch the 262K fit", "summary": "Published tests show a separate small draft model makes Qwen3.6-35B-A3B slower (one RTX 3090: 135.7 tok/s baseline vs 115.9-121.3 mean with drafts), while the model's own MTP head helps one user at a time: on an R9700 (Q4_K_M, Vulkan) one slot went from about 108 to about 133 tok/s, but MTP made a 3-slot job 1.58x slower. We read the headers of Unsloth's MTP GGUFs: the extra MTP block is Q8_0 attention with Q5_K/Q6_K experts in UD-Q5_K_XL, nothing below Q4.", "details": "Size cost from the Hugging Face file listings: the MTP files are 470 MiB (UD-Q4_K_XL), 540 MiB (UD-Q5_K_XL) and 732 MiB (UD-Q6_K_XL) larger than the plain files. The MTP block has its own attention layer, so its KV cache adds about 512 MiB at f16 or 272 MiB at q8_0 for 262K context (K/V width 512 from the header). Our estimate for UD-Q5_K_XL with MTP at 262K and f16 KV is about 30.9 GiB before compute buffers, which is tight or over on a 32 GiB card with ECC on (about 29.9 GiB visible); quantizing only the draft cache or capping context near 128K should fit (estimate, not measured). Other data points: a 7900 XTX on ROCm gained 1.70x at 16K falling to 1.35x at 196K and ran out of memory at 245K; Unsloth reports 1.15-1.2x on average for this MoE at 2 draft tokens, with acceptance falling from 83% to 50% at 4; the community DSpark drafter beat MTP on code (+12.7%) but lost on prose (-4.8%) on non-Radeon hardware. A later re-run of the 3090 study found the slowdown tracks low real acceptance plus draft overhead.", "setup": { "model": "Qwen3.6-35B-A3B (MTP GGUF)", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (header check); Q4_K_M and UD-Q4_K_XL in cited speed tests", "engine": "llama.cpp", "engine_version": "per source", "gpus": "1x R9700 (cited); RTX 3090 and 7900 XTX for comparison", "context": "short to 245K", "flags": "--spec-type draft-mtp --spec-draft-n-max 2, -np 1" }, "metrics": [ { "label": "R9700 Q4_K_M Vulkan, 1 slot, no MTP", "value": 108, "unit": "tok/s" }, { "label": "R9700 Q4_K_M Vulkan, 1 slot, MTP", "value": 133, "unit": "tok/s" }, { "label": "R9700, 3 slots, MTP slowdown", "value": 1.58, "unit": "x slower" }, { "label": "RTX 3090 baseline vs best draft-model mean", "value": "135.7 vs 121.1", "unit": "tok/s" }, { "label": "MTP file size increase, UD-Q5_K_XL", "value": 540, "unit": "MiB" }, { "label": "MTP draft KV at 262K, f16", "value": 512, "unit": "MiB" } ], "sources": [ "https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/discussions/14", "https://github.com/thc1006/qwen3.6-speculative-decoding-rtx3090", "https://gist.github.com/AshyIsMe/ef9dc0e57d7af5626f3935f55c38543a", "https://unsloth.ai/docs/models/tutorials/qwen3.6", "https://github.com/ggml-org/llama.cpp/discussions/26230", "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF" ], "next": "Measure MTP at 1, 2 and 3 draft tokens on our single-R9700 UD-Q5_K_XL lane at 128K, then test whether 262K fits with a q8_0 draft cache." }, { "id": "2026-10-26-multi-gpu-silent-corruption", "date": "2026-10-26T08:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "Two Radeon cards, wrong answers at full speed: which split setups corrupt text, what fixed it, and a correctness check", "summary": "Several 2026 Radeon multi-GPU setups on ROCm/HIP produced fluent gibberish or blank output at normal speed with no error, and llama-bench still reported healthy numbers. The Windows layer-split case was fixed in ROCm's runtime (PR #10553, merged 2026-09-02); Windows -sm tensor on ROCm is still open; Linux cards with no peer-to-peer path need a no-peer-copy build. Always check text, not just tok/s, after a stack change.", "details": "Windows HIP, -sm layer (2x R9700, 7900 XTX + 9070 XT and others): a staged GPU-to-GPU copy raced in-flight compute; fixed in ROCm CLR PR #10553 and confirmed by AMD on a current Lemonade build, but older Windows ROCm releases (e.g. b10767) still corrupt. Building llama.cpp with -DGGML_CUDA_NO_PEER_COPY=ON (a cmake option; an environment variable does nothing) also fixes it; one mixed-card test went from 0/7 to 7/7 exact answers at about -4.5% prefill and -7.7% decode. Windows HIP -sm tensor: open (Lemonade llamacpp-rocm #148); root cause is hipMemcpy2DAsync from memory-mapped model pages, and --load-mode none avoids it. Even when correct, Windows exposes no GPU peer access, so a second R9700 adds memory, not speed (open driver request ROCm/rocm-systems #13715). Linux, 2x RX 9060 XT on separate root ports (ROCm 7.2.4): HSA_ENABLE_PEER_SDMA=0 did not fix it; the no-peer-copy build did. Quick check: ask a few one-answer questions at temperature 0 on one card and then split across both; answers should match exactly.", "setup": { "model": "Qwen3.8-27B Q4_K_M (Windows speed test); various (corruption reports)", "quant": ">=Q4 in all cited tests", "engine": "llama.cpp (HIP and Vulkan), Strata", "engine_version": "per source", "gpus": "2x R9700; 7900 XTX + 9070 XT; 2x 9060 XT; others", "context": "short", "flags": "-sm layer / -sm tensor, -DGGML_CUDA_NO_PEER_COPY=ON, --load-mode none" }, "metrics": [ { "label": "Windows 2x R9700 Vulkan, one card (-sm none)", "value": 32.1, "unit": "tok/s" }, { "label": "Windows 2x R9700 Vulkan, -sm layer", "value": 31.3, "unit": "tok/s" }, { "label": "Windows 2x R9700 Vulkan, -sm tensor", "value": 14.9, "unit": "tok/s" }, { "label": "No-peer-copy build, decode cost (one mixed-card test)", "value": -7.7, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/issues/28676", "https://github.com/lemonade-sdk/llamacpp-rocm/issues/75", "https://github.com/ROCm/rocm-systems/pull/10553", "https://github.com/lemonade-sdk/llamacpp-rocm/issues/148", "https://github.com/ROCm/rocm-systems/issues/13715", "https://github.com/ROCm/ROCm/issues/6342" ], "next": "Run the same exact-answer check on our dual-R9700 Linux launchers (Vulkan and HIP, layer and tensor) and make it a required step before every benchmark." }, { "id": "2026-10-26-slot-restore-hybrid-checkpoints", "date": "2026-10-26T07:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama.cpp slot save/restore now works for hybrid Qwen models: resume a long agent context after a restart", "summary": "Before llama.cpp b11507, restoring a saved slot on hybrid (Gated DeltaNet) models like Qwen3.5/3.6/3.8 reported success but re-processed the whole prompt, because context checkpoints were not saved. PR #26004 now appends them to the slot file. A published AMD Vulkan test (Radeon 8050S, Qwen3.5-35B-A3B UD-Q4_K_XL, 58K tokens) cut the first request after restore from 181.9 s to 4.7 s.", "details": "Source read at llama.cpp master c6fb3b5: the slot file carries the KV payload, then a checkpoint appendix, then (since PR #30574, b11669) the draft/MTP state, so speculative decoding keeps its acceptance after a restore (37% to 49% in the PR's Qwen3.8-27B test). Older files still load; mismatched draft data is dropped instead of crashing. Testing trap: the RAM prompt cache (--cache-ram) can hide a broken disk restore, so test with --cache-ram 0. Estimate from the Qwen3.6-35B-A3B config (not measured): about 20 KiB per token of f16 KV plus about 63 MiB of recurrent state, so roughly 1.2 GiB at 58K tokens and 5.1 GiB at 262K. A test on dual R9700 is next.", "setup": { "model": "Qwen3.5-35B-A3B (published test); Qwen3.6-35B-A3B (size estimate)", "quant": "UD-Q4_K_XL (published test)", "engine": "llama.cpp llama-server", "engine_version": ">= b11507 (checkpoints), >= b11669 (draft state)", "gpus": "Radeon 8050S (published test)", "context": "58K tokens", "flags": "--slot-save-path DIR, --ctx-checkpoints, --cache-ram 0 for testing" }, "metrics": [ { "label": "First request after restore, before fix", "value": 181.9, "unit": "s" }, { "label": "First request after restore, with fix", "value": 4.7, "unit": "s" }, { "label": "Tokens re-processed after restore, with fix", "value": 516, "unit": "tokens" }, { "label": "Est. slot file, 35B-A3B at 262K, f16 KV", "value": 5.1, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/26004", "https://github.com/ggml-org/llama.cpp/pull/26004#issuecomment-5055464245", "https://github.com/ggml-org/llama.cpp/pull/30574", "https://github.com/ggml-org/llama.cpp/issues/28194", "https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json" ], "next": "Measure save, restart and restore times on dual R9700 for the 35B and the Flash llama.cpp lane, then wire save-on-stop and restore-on-start into the server units." }, { "id": "2026-10-26-thinking-budget-latency", "date": "2026-10-26T06:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Capping thinking on a slow Flash model: the stop message matters more than the token count", "summary": "At about 33 tok/s (a Flash-class MoE on 2x R9700 with experts in DDR5), every 1,000 thinking tokens is roughly 31 seconds before the first visible word. llama.cpp can cap this per request with thinking_budget_tokens plus reasoning_budget_message, and since June 2026 a client can end thinking mid-stream with POST /v1/chat/completions/control, which allows a wall-clock deadline instead of a token count. A bare cap without a stop message hurts: HumanEval on a Qwen3.5 9B fell to ~79% vs ~89% with a message (~93% uncapped), and on Qwen3.6-35B-A3B a bare 4,096-token cap ran 170 s as reasoning spilled into the answer, vs 78 s with Qwen's own stop sentence.", "details": "In llama.cpp master (2026-10-26) the per-request budget overrides the --reasoning-budget server flag unless it is -1, the sampler re-arms on every new think block, and nothing happens without --jinja because the think tags come from the chat template. Qwen's documented stop sentence is: 'Considering the limited time by the user, I have to give the solution based on the thinking directly now.' Arithmetic at measured decode speeds: 2,048 thinking tokens take about 63 s on the Flash model at 32.6 tok/s, about 128 s on a 16 tok/s fallback, and about 20 s on a 35B-A3B at 100.8 tok/s on one R9700. A 20-second deadline allows about 650 thinking tokens on the Flash model vs about 2,000 on the 35B. Whether the Strata engine honours these request fields is not yet checked.", "setup": { "model": "Qwen3.8 Flash (UD-Q6_K_XL) and Qwen3.6-35B-A3B (UD-Q5_K_XL)", "engine": "llama.cpp server (source review), Strata (speed only)", "engine_version": "llama.cpp master 8b28c23", "gpus": "2x Radeon AI PRO R9700 (Flash); 1x R9700 (35B)" }, "metrics": [ { "label": "Flash decode used for the arithmetic", "value": 32.6, "unit": "tok/s" }, { "label": "Seconds per 1,000 thinking tokens (Flash)", "value": 31, "unit": "s" }, { "label": "HumanEval, budget 1000 with message (Qwen3.5 9B)", "value": 89, "unit": "%" }, { "label": "HumanEval, budget without message (Qwen3.5 9B)", "value": 79, "unit": "%" }, { "label": "Bare 4,096 cap, 35B-A3B, 3 prompts", "value": 170, "unit": "s" }, { "label": "4,096 cap with stop message, 35B-A3B", "value": 78, "unit": "s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/20297", "https://github.com/ggml-org/llama.cpp/pull/23971", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "https://huggingface.co/ManniX-ITA/Qwen3.6-27B-A3B-Coder/commit/59aa5b59f6ecd7a7fa9cc191e1e40d064ad2b2e7", "https://arxiv.org/abs/2505.09388" ], "next": "Measure natural thinking length and capped accuracy on real agent turns for both models, then pick a default cap or deadline." }, { "id": "2026-10-26-r9700-hang-recovery-classes", "date": "2026-10-26T05:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 GPU hangs come in three kinds, and only one is fixed by restarting the server", "summary": "Public Radeon AI PRO R9700 (gfx1201) fault reports split into three recovery classes. (1) The server process dies and the driver's reset succeeds: a systemd Restart=on-failure is enough. (2) A silent wedge: the server stays up at 100% GPU busy and 0% memory busy but produces no tokens, or blocks at idle power with no error, so nothing restarts it and a /health check still answers 200. (3) A dead GPU: the reset ends with 'PSP resume failed' / ret -62 or a reset storm with a CPU soft lockup, and only a reboot helps; repeated auto-retries in that state have hard-hung an R9700.", "details": "Practical rules for unattended serving: probe with a tiny real completion and a wall-clock timeout instead of trusting /health; scan the kernel log for PSP/IP-block resume failures and stop (don't restart) when they appear; cap restarts (for example 3 per 30 minutes) so a failing load can't loop. amdgpu.cwsr_enable=0 is circulating as a fix but one report saw it turn clean faults into 'GPU Hang' on 3 of 4 R9700s. In one controlled A/B the same llama.cpp workload that wedged on ROCm 7.14 passed on Vulkan RADV, so a Vulkan build is a reasonable fallback service. The hipBLASLt page fault behind many training-time resets is fixed by rocm-libraries PR 8909 (in ROCm 10.2 nightlies as of 2026-09-21). We wrote a stdlib watchdog that encodes these rules and passes 8 of 8 simulated cases; it has not yet been run against real hangs.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201)", "engine": "llama.cpp / vLLM (published reports)" }, "metrics": [ { "label": "No-output stall before timeout (ROCm 7.14, #6630)", "value": 180, "unit": "s" }, { "label": "R9700s that degraded to GPU Hang with cwsr_enable=0", "value": "3 of 4", "unit": "" }, { "label": "Successful MODE2 resets recovered by a 10 s systemd restart (gfx1151 report)", "value": 4, "unit": "" } ], "sources": [ "https://github.com/ROCm/legacy-rocm-build/issues/6630", "https://github.com/ROCm/ROCm/issues/6413", "https://github.com/ROCm/TheRock/issues/9271", "https://github.com/lemonade-sdk/lemonade/issues/2609", "https://github.com/ROCm/ROCm/issues/6127", "https://docs.kernel.org/gpu/amdgpu/debugfs.html" ], "next": "Run the watchdog in log-only mode for three nights on the dual-R9700 desktop, then turn on automatic restart, fallback and stop-and-alert." }, { "id": "2026-10-26-endpoint-security", "date": "2026-10-26T04:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Securing a local LLM server: what llama-server's API key covers, 2026 CVEs, and why ggml RPC must stay on a private link", "summary": "llama-server binds 127.0.0.1 by default, and --api-key protects every route except /health and the web-UI files. Without a key, /props, /slots and /v1/models are open, and the default CORS setting echoes any Origin. Builds from b8227 up to but not including b11393 have a 9.2-rated use-after-free on POST /completion (CVE-2026-107183), so update to b11393 or newer. The RPC backend used for multi-machine splits has no authentication and had a 9.8-rated remote code execution bug this year (fixed in b8492), so it belongs on a dedicated point-to-point link only.", "details": "Sources: llama.cpp master server source (public endpoints list) and server/RPC READMEs; CVE records for CVE-2026-107183 (fixed b11393, PR #29942; Debian's sid package 0.7.0 still listed vulnerable), CVE-2026-43632 (tokenizer-endpoint use-after-free when --sleep-idle-seconds is set), CVE-2026-21869 (negative n_discard with --context-shift), CVE-2026-34159 (RPC GRAPH_COMPUTE RCE). vLLM's own security docs say --api-key guards only /v1, /v2, /inference and /cohere; /invocations, /pause, /abort_requests and /metrics stay open. A September 2026 internet census found 36,769 exposed self-hosted AI endpoints, of which 2.02% returned an auth challenge (llama.cpp 69 found, 0 authenticated; vLLM 4,880 found, 3 authenticated). Our posture for a dual-R9700 box: keep every server on localhost and reach it through one reverse proxy or VPN tunnel; if a LAN bind is needed use --api-key-file, set --cors-origins to the frontend, and turn off /slots; run rpc-server bound only to the inter-machine link address. We also wrote a read-only probe that checks which routes answer without a key and whether CORS reflects arbitrary origins; a bare TCP check on the RPC port can false-positive, since other services may use 50052.", "setup": { "engine": "llama.cpp llama-server, ggml RPC, vLLM", "engine_version": "llama.cpp master 9125a2e (release v0.7.0)", "gpus": "2x Radeon AI PRO R9700 (applies to any GPU)" }, "metrics": [ { "label": "CVE-2026-107183 fixed in", "value": "b11393", "unit": "" }, { "label": "CVE-2026-107183 CVSS 4.0", "value": 9.2, "unit": "" }, { "label": "CVE-2026-34159 (RPC) CVSS 3.1", "value": 9.8, "unit": "" }, { "label": "Exposed AI endpoints with auth challenge (Sept 2026 census)", "value": 2.02, "unit": "%" } ], "sources": [ "https://www.cve.org/CVERecord?id=CVE-2026-107183", "https://www.cve.org/CVERecord?id=CVE-2026-34159", "https://github.com/ggml-org/llama.cpp/security/advisories?state=published", "https://github.com/advisories/GHSA-3f2h-96vw-c84x", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md", "https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md", "https://docs.vllm.ai/en/latest/usage/security/", "https://www.mysteriumvpn.com/blog/data-and-research/we-exposed-ai-supply-chain" ], "next": "Run the read-only probe against every listener on our own box and from a second LAN machine, then pin all llama.cpp builds at b11393 or newer before the two-box RPC test." }, { "id": "2026-10-26-r9700-tts-llama-server", "date": "2026-10-26T03:12:00-05:00", "project": "idea-lab", "kind": "result", "title": "Text-to-speech next to a local LLM on R9700: llama-server now serves Qwen3-TTS, and a CPU voice would compete with offloaded experts for DDR5", "summary": "llama.cpp merged a native POST /tts endpoint for llama-server on 2026-10-20 (release b11781), with streaming, speaker-reference upload and router-mode autoload, so a TTS model can sit behind the same server as the LLM. Published R9700 results for Qwen3-TTS: the PyTorch reference runs at RTF 1.90 (slower than real time), while a custom HIP port reaches RTF 0.55 for the 1.7B model and 0.41-0.54 for the 0.6B. No llama.cpp R9700 TTS number has been published yet.", "details": "The 1.7B Base GGUFs are Q4_K_M 0.965 GiB, Q8_0 1.721 GiB and bf16 3.234 GiB, plus a 0.416-0.623 GiB mmproj, all at or above 4-bit. A review of the PR found that code2wav pads every decode batch to 72 frames, so shrinking batches for faster first audio detunes the voice (speaker similarity 0.817 vs 0.862); a not-yet-merged patch with a 12/24/48/72 frame ramp cut time to first audio from about 1.8 s to 0.4 s on Vulkan. Fixed seeds are not reproducible on the server yet. Estimate only (bandwidth roofline from the published per-frame weight reads): on an R9700 every quant is far faster than real time (RTF about 0.07-0.23), while on CPU with dual-channel DDR5 at a measured 55 GB/s Q8_0 lands near RTF 1.05 and Q4_K_M near 0.60, and that traffic competes with any MoE experts offloaded to the same DDR5.", "setup": { "model": "Qwen3-TTS-12Hz 1.7B and 0.6B", "engine": "llama.cpp llama-server /tts (b11781+); custom HIP port; PyTorch reference (published)", "gpus": "1x Radeon AI PRO R9700" }, "metrics": [ { "label": "Qwen3-TTS 1.7B, PyTorch FP32 reference, RTF", "value": 1.9, "unit": "x" }, { "label": "Qwen3-TTS 1.7B, custom HIP port, RTF", "value": 0.55, "unit": "x" }, { "label": "Qwen3-TTS 0.6B, custom HIP port streaming, RTF", "value": 0.54, "unit": "x" }, { "label": "Qwen3-TTS 1.7B Q8_0 GGUF + bf16 mmproj", "value": 2.34, "unit": "GiB" }, { "label": "Time to first audio with code2wav ramp patch (unmerged, Vulkan)", "value": 0.4, "unit": "s" }, { "label": "Estimated CPU RTF, Q4_K_M at 55 GB/s DDR5 (estimate)", "value": 0.6, "unit": "x" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/26603", "https://github.com/ggml-org/llama.cpp/pull/26254", "https://github.com/ggml-org/llama.cpp/tree/master/tools/tts", "https://huggingface.co/ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF", "https://flynnbody.com/blog/running-qwen3-tts-on-amd-r9700/", "https://github.com/piedshag/qwen3-tts-hip" ], "next": "Measure llama-server /tts on our own R9700s (Vulkan vs HIP vs CPU), then again while the Flash model decodes, to see what speaking costs the LLM." }, { "id": "2026-10-26-r9700-image-video-gen", "date": "2026-10-26T02:08:00-05:00", "project": "idea-lab", "kind": "result", "title": "Image and video generation on one R9700: step-distill LoRAs give ~6x, the hipBLASLt/MIOpen flags slow video 5-7%", "summary": "From published R9700 ComfyUI runs: SDXL 1024x1024 runs at 4.49-4.6 it/s. 4-step distillation LoRAs cut Qwen-Image-2512 from 85.9 s to 12.4 s and Wan 2.2 A14B 480p from 341.7 s to 56.6 s, far more than any runtime knob. In a same-box A/B, TORCH_BLAS_PREFER_HIPBLASLT=1 plus COMFYUI_ENABLE_MIOPEN=1 made all four video workflows 5.3-7.3% slower and left Qwen-Image within 2%.", "details": "Wan 2.2 T2V A14B Q6_K GGUF at 720p (33 frames, 4 steps) peaks at 31.8 of 31.86 GiB on one card, 27.7 GiB at 480p, so video cannot share a card with any resident LLM layers; even SDXL peaks at 12.87 GiB. Video runs drew 500-568 W at 97-104 C junction vs 342-503 W for image runs (same source). Cold starts dominate short jobs: Qwen-Image 4-step 33.4 s cold vs 12.4 s warm. The flag A/B is one warm run per config, so differences under ~2% are noise; the video slowdown is consistent in sign across Wan 2.2 I2V/T2V and HunyuanVideo 1.5 I2V/T2V. Qwen-Image-Edit 20-step was 7.0% faster with the flags. All weights cited are 4-bit or higher (FP8, Q6_K, BF16).", "setup": { "model": "SDXL 1.0, Qwen-Image-2512 FP8, Wan 2.2 A14B (FP8 and Q6_K), HunyuanVideo 1.5", "engine": "ComfyUI (published)", "engine_version": "per source (ROCm 7.1-7.2.4)", "gpus": "1x Radeon AI PRO R9700" }, "metrics": [ { "label": "SDXL 1024x1024 sampling", "value": 4.49, "unit": "it/s" }, { "label": "Qwen-Image-2512 FP8, 20 steps, warm", "value": 85.9, "unit": "s" }, { "label": "Qwen-Image-2512 FP8, 4-step LoRA, warm", "value": 12.4, "unit": "s" }, { "label": "Wan 2.2 A14B Q6_K 480p 33 fr, 4-step LoRA", "value": 56.6, "unit": "s" }, { "label": "Wan 2.2 A14B Q6_K 720p VRAM peak", "value": 31.8, "unit": "GiB" }, { "label": "Wan 2.2 I2V with hipBLASLt+MIOpen flags", "value": 5.3, "unit": "% slower" } ], "sources": [ "https://kyuz0.github.io/amd-r9700-comfy/", "https://kyuz0.github.io/amd-r9700-comfy/benchmark_results.json", "https://brookesia.app/boards/ai-hardware/76", "https://brookesia.app/boards/ai-hardware/78", "https://rocm.blogs.amd.com/artificial-intelligence/comfyui-radeon-9000/README.html" ], "next": "Measure an SDXL baseline, an ABBA test of the two flags, and two ComfyUI instances (one per card) on our own pair of R9700s." }, { "id": "2026-10-26-r9700-lora-pagefault", "date": "2026-10-26T01:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "LoRA fine-tuning on the R9700: the hipBLASLt page-fault trap and what fixes it", "summary": "The R9700 (gfx1201) is officially supported for LoRA/QLoRA training (Unsloth lists RDNA4 as Full; TheRock lists gfx1201 Release Ready), but stock hipBLASLt GEMMs can fault in the LoRA backward pass at a fixed step. On ROCm 7.2.4 a reporter saw bf16 LoRA on Qwen3-8B fault at step 85 and QLoRA NF4 at step 799. The fix (rocm-libraries PR 8909, merged 2026-08-14) ran clean in independent validations at about 1.5% extra step time.", "details": "Cause: RDNA4's transpose global load has no hardware bounds field, so some Tensile GEMMs read past the end of a tensor; PyTorch's caching allocator places tensors next to unmapped pages, so the over-read faults. The fault step depends on data order and resume point, so resuming from a checkpoint with the same order dies at the same step. On stock libraries, PYTORCH_HIP_ALLOC_CONF=roundup_power2_divisions:16 had no measurable cost and strongly reduced faults but did not remove them; disabling hipBLASLt (TORCH_BLAS_PREFER_HIPBLASLT=0, ROCBLAS_USE_HIPBLASLT=0, DISABLE_ADDMM_HIP_LT=1) was clean but about 2.5x slower. HSA_ENABLE_SDMA=0, HSA_XNACK=1 and max_split_size_mb did not help. Other reported tips: plain adamw_torch instead of paged or 8-bit optimizers, frequent checkpoints, change the data seed on resume, and reboot after a fault before relaunching. Multi-GPU (RCCL) training is Linux-only per Unsloth. We have not yet seen the fix named in a stable 7.2.x release note.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201), published reports", "engine": "PyTorch ROCm + PEFT / Unsloth", "engine_version": "ROCm 7.2.4, torch 2.13.0 (issue 6600)", "model": "Qwen3-8B, Qwen3.6-27B (reproducers)" }, "metrics": [ { "label": "bf16 LoRA fault step, stock ROCm 7.2.4", "value": 85, "unit": "step" }, { "label": "QLoRA NF4 fault step, stock ROCm 7.2.4", "value": 799, "unit": "step" }, { "label": "12-step run, stock vs PR 8909 build", "value": "726 vs 737", "unit": "s" }, { "label": "hipBLASLt-off workaround step time", "value": 2.5, "unit": "x slower" } ], "sources": [ "https://github.com/ROCm/legacy-rocm-build/issues/6600", "https://github.com/ROCm/rocm-libraries/pull/8909", "https://unsloth.ai/docs/basics/amd", "https://github.com/ROCm/TheRock/blob/main/SUPPORTED_GPUS.md" ], "next": "Run a one-card LoRA smoke test on the local desktop comparing stock, allocator rounding and hipBLASLt-off, then QLoRA NF4." }, { "id": "2026-10-26-embed-rerank-pick", "date": "2026-10-26T00:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Local RAG for an R9700 agent: Qwen3-Embedding-4B Q5_K_M is the value point, 0.6B pair fits in ~1.2 GiB", "summary": "From official GGUF sizes and vendor scores: Qwen3-Embedding-4B Q5_K_M (2.69 GiB) scores 69.45 on MTEB multilingual, +5.1 over the 0.6B Q8_0 (0.60 GiB, 64.33), while the 8B Q5_K_M (5.05 GiB) adds only +1.1 more. Qwen3-Reranker-0.6B (0.60 GiB at Q8_0) lifts retrieval to 65.80 MTEB-R from 61.82, well ahead of bge-reranker-v2-m3 at 57.03.", "details": "llama.cpp serves both: --embedding --pooling last for Qwen3-Embedding and --reranking for /v1/rerank. Bidirectional rerankers such as bge-reranker-v2-m3 must fit the whole prompt in --ubatch-size; causal Qwen3 rerankers do not have that limit. Qwen reports that adding a query-side instruction ('Instruct: ...\\nQuery: ...') improves retrieval by about 1-5%. Scores are vendor-reported (Qwen model cards), not measured here; reranker scores are on the top-100 candidates from Qwen3-Embedding-0.6B. Whether the small pair is fast enough on CPU, and how much a GPU embedder slows a co-resident Flash model, is still to be measured.", "setup": { "model": "Qwen3-Embedding 0.6B/4B/8B, Qwen3-Reranker-0.6B, bge-reranker-v2-m3", "quant": "Q8_0 / Q5_K_M", "engine": "llama.cpp server (docs at master d854d98bae)", "gpus": "Radeon AI PRO R9700 (planned placement)" }, "metrics": [ { "label": "Qwen3-Embedding-0.6B Q8_0 size", "value": 0.595, "unit": "GiB" }, { "label": "Qwen3-Embedding-4B Q5_K_M size", "value": 2.691, "unit": "GiB" }, { "label": "Qwen3-Embedding-8B Q5_K_M size", "value": 5.05, "unit": "GiB" }, { "label": "MTEB multilingual mean, 0.6B / 4B / 8B", "value": "64.33 / 69.45 / 70.58", "unit": "" }, { "label": "MTEB-R, Qwen3-Reranker-0.6B vs bge-reranker-v2-m3", "value": "65.80 vs 57.03", "unit": "" } ], "sources": [ "https://huggingface.co/Qwen/Qwen3-Embedding-8B", "https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF", "https://huggingface.co/Qwen/Qwen3-Reranker-0.6B", "https://huggingface.co/ggml-org/Qwen3-Reranker-0.6B-Q8_0-GGUF", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md" ], "next": "Measure CPU vs one-R9700 placement for the 0.6B pair and the 4B embedder, including the decode drop on a co-resident Flash model." }, { "id": "2026-10-25-strix-halo-vs-r9700", "date": "2026-10-25T23:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Strix Halo vs Radeon AI PRO R9700 on the same benchmark harness: ~2x decode, 3-4x prefill when the model fits", "summary": "Joining kyuz0's two public llama-bench grids (same scripts, best backend per row, 8 overlapping Q4+ models), one or two R9700s lead a 128 GB Ryzen AI Max+ 395 by a median ~2.0x on decode (tg128) and ~2.9x on prompt processing (pp512), widening to ~3.6-4.1x prefill at 32K-64K depth. Strix Halo's edge is capacity: it runs 80-90 GiB Q4/Q5 MoE models entirely in unified memory.", "details": "gpt-oss-120b MXFP4 (59 GiB, two R9700s): 98.0 vs 56.6 tok/s decode, 1956 vs 720 tok/s pp512. Qwen3.6-35B-A3B UD-Q4_K_XL (one R9700): 108.3 vs 60.4 tok/s decode; at 64K depth 82.4 vs 43.2 tok/s decode and 2048 vs 548 tok/s prefill. Strix Halo runs Qwen3.5-122B-A10B UD-Q5_K_XL (85.6 GiB) at 22.3 tok/s and Nemotron-3-Super-120B-A12B UD-Q4_K_XL (78 GiB) at 14.9 tok/s; those exceed 64 GB of R9700 VRAM and need DDR5 expert offload on a desktop. Caveats: different llama.cpp builds (b8990 vs b9187/b9193) and OS versions; best backend chosen separately per platform; Q1-Q3 rows excluded.", "setup": { "model": "8 overlapping models (gpt-oss-120b, Qwen3.6-35B-A3B, Gemma 4, others)", "quant": "Q4 and up only", "engine": "llama.cpp llama-bench (published grids)", "engine_version": "b8990 (R9700), b9187/b9193 (Strix Halo)", "gpus": "1-2x Radeon AI PRO R9700 vs Ryzen AI Max+ 395 128 GB", "context": "0, 32K and 64K depth", "flags": "fa on, ngl 99, best of ROCm and Vulkan backends" }, "metrics": [ { "label": "Median decode lead, R9700 vs Strix Halo (tg128)", "value": 2.0, "unit": "x" }, { "label": "Median prompt lead (pp512)", "value": 2.9, "unit": "x" }, { "label": "Median prefill lead at 64K depth", "value": 4.0, "unit": "x" }, { "label": "gpt-oss-120b decode, 2x R9700", "value": 98.0, "unit": "tok/s" }, { "label": "gpt-oss-120b decode, Strix Halo", "value": 56.6, "unit": "tok/s" }, { "label": "Qwen3.5-122B-A10B UD-Q5_K_XL decode, Strix Halo", "value": 22.3, "unit": "tok/s" } ], "sources": [ "https://kyuz0.github.io/amd-r9700-ai-toolboxes/", "https://kyuz0.github.io/amd-strix-halo-toolboxes/", "https://github.com/lhl/strix-halo-testing/tree/main/llm-bench" ], "next": "Measure Qwen3.5-122B-A10B UD-Q5_K_XL on two R9700s with experts in DDR5, with the same tests, against Strix Halo's 22.3 tok/s." }, { "id": "2026-10-25-model-swap-router", "date": "2026-10-25T22:07:00-05:00", "project": "idea-lab", "kind": "result", "title": "Swapping big models on one box: llama.cpp router mode evicts by count, not VRAM; llama-swap can budget per card", "summary": "llama-server's router mode (start it with no model and a --models-preset INI) loads models on demand, but it decides what to unload only by counting: --models-max defaults to 4 and the scheduler never checks VRAM or RAM. On a dual-R9700 box where one Flash-class model fills both cards and DDR5, run it with --models-max 1, or a request naming a second big model will try to load it alongside the first. llama-swap's capacity router lets you declare per-card and RAM budgets instead, and can also host non-llama.cpp servers.", "details": "Read from llama.cpp master d854d98bae (2026-10-25): server_lru_sched::has_capacity() compares the number of running models with models_max, and a sleeping model still counts as running, so it holds a slot after its memory is freed. --sleep-idle-seconds (PR #18228) unloads the model and KV cache after the idle time; GET /health, /props, /models and /metrics neither wake it nor reset the timer, so monitoring doesn't keep the cards busy. Presets accept any CLI flag as an INI key plus load-on-startup and stop-timeout (default 10 s). Router mode only runs llama.cpp and can't express rules like 'these two single-card models may coexist, nothing else may'. llama-swap v266 runs any OpenAI-compatible server as a child process and has three routing engines: group (one model at a time by default), matrix (expressions listing allowed combinations, evicting by evict_cost) and capacity (declared pools such as gpu0, gpu1 and ram, with a requirement per model). The capacity engine does no automatic measurement, so per-model footprints have to be measured first. Load, swap and wake times on R9700s are not measured yet.", "setup": { "engine": "llama.cpp master d854d98bae and llama-swap v266 (source and docs read)", "hardware": "planned for 2x Radeon AI PRO R9700 + DDR5" }, "metrics": [ { "label": "router --models-max default", "value": 4, "unit": "models" }, { "label": "preset stop-timeout default", "value": 10, "unit": "s" }, { "label": "llama-swap routing engines", "value": 3, "unit": "engines" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/server-models.cpp", "https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/tools/server/README.md", "https://github.com/ggml-org/llama.cpp/pull/18228", "https://github.com/mostlygeek/llama-swap/blob/v266/docs/config.example.yaml", "https://github.com/mostlygeek/llama-swap/blob/v266/README.md" ], "next": "Measure each preset's per-card VRAM and DDR5 footprint, cold-load, swap-in and wake times on 2x R9700, then decide whether the always-on agent model goes behind the swap endpoint." }, { "id": "2026-10-25-llamacpp-grammar-sampling-cost", "date": "2026-10-25T21:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Tool calls and JSON mode in llama.cpp: lazy grammars, rejection sampling, and why -bs switches off", "summary": "In current llama.cpp, a request with tools and tool_choice \"auto\" uses a lazy grammar that stays idle until the model starts a tool call, while tool_choice \"required\" or a response_format JSON schema constrains every token from the start. Sampling checks only the chosen token against the grammar and computes the full-vocabulary mask only when that token is rejected. Experimental backend (GPU) sampling, -bs, is turned off for any request that carries a grammar, lazy tool grammars included, so agent requests always sample on the CPU.", "details": "Read from llama.cpp master d854d98bae (2026-10-25): common/sampling.cpp logs \"backend sampling is not compatible with grammar, disabling\" and also disables it when a reasoning budget is active; rejection sampling was restored in PR #18137 because grammar sampling is widely used and slow. Published -bs results are CUDA-only: no change at 1 slot and +48% at 32 slots (issue #27050), about 8% with MTP speculative decoding on an RTX 5090 (PR #25532); PR #17004 recommends CUDA for now. In MaskBench (128K-token vocab, one core), llama.cpp's native grammar mask averaged 16.6 ms vs 64 us for LLGuidance, and llama.cpp handled 6,505 of 11,306 schemas vs 8,909. Building llama.cpp with LLGuidance helps JSON Schema requests only; tool-call grammars from the chat parser stay GBNF. Qwen3.6 has a 248,320-token vocab, about 1.9x the benchmark's. Not yet measured on R9700s; the cost should show most on fast models (~10 ms per token) and speculative lanes, least on DDR5-hybrid Flash.", "setup": { "engine": "llama.cpp master d854d98bae (source read)", "models": "Qwen3.6 tokenizer (248,320 vocab)" }, "metrics": [ { "label": "llama.cpp native grammar mask, MaskBench avg", "value": 16.6, "unit": "ms" }, { "label": "LLGuidance mask, MaskBench avg", "value": 0.064, "unit": "ms" }, { "label": "-bs throughput gain at 32 slots (RTX 5090)", "value": 48, "unit": "%" }, { "label": "Qwen3.6 vocab size", "value": 248320, "unit": "tokens" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/sampling.cpp", "https://github.com/ggml-org/llama.cpp/blob/d854d98bae598388ecea1ed84396587e5edc326d/common/chat-auto-parser-generator.cpp", "https://github.com/ggml-org/llama.cpp/pull/18137", "https://github.com/ggml-org/llama.cpp/pull/17004", "https://github.com/ggml-org/llama.cpp/pull/25532", "https://github.com/ggml-org/llama.cpp/issues/27050", "https://github.com/guidance-ai/jsonschemabench/blob/main/maskbench/README.md", "https://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json" ], "next": "Measure plain vs tools (auto and required) vs JSON-schema decode speed on a fast single-R9700 model and on dual-R9700 Flash, with and without -bs." }, { "id": "2026-10-25-whisper-stt-rdna4", "date": "2026-10-25T20:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Speech-to-text next to a local LLM on R9700s: use whisper.cpp, and mind the endpoint", "summary": "For voice input on RDNA4 cards, whisper.cpp (HIP or Vulkan) is the working path: a test on an R9700 ran the 11 s JFK sample with tiny.en in 206 ms, and an RX 9070 XT on Vulkan transcribed a 19.3-minute talk with large-v3-turbo in 48 s (about 24x real time, 1.5% WER). One RDNA4 tester reports the CTranslate2 ROCm forks behind faster-whisper produce wrong transcripts, so avoid that stack. The turbo model is 0.81 GiB at q8_0. whisper-server only serves /inference unless you pass --inference-path /v1/audio/transcriptions, which OpenAI-style agent clients expect.", "details": "Cited results, not yet measured here. Strix Halo (gfx1151) on an 11 s clip: large-v3-turbo 0.84 s on Vulkan vs 0.63 s on ROCm/HIP. A single RX 9070 tester ran whisper.cpp on Vulkan and a ROCm diarizer on the same card at once without meaningful contention, and saw large-v3-turbo crash with a UTF-8 decode error (large-v3 was fine). whisper-server (master 231638a) turns flash attention on by default, picks the card with -dev N or WHISPER_ARG_DEVICE, and has built-in VAD. llama.cpp's own /v1/audio/transcriptions route (PR #21863) is for audio-capable LLMs, not Whisper models. Model sizes: large-v3-turbo f16 1.513 GiB, q8_0 0.814, q5_0 0.535; large-v3 f16 2.882, q5_0 1.007. On a dual-R9700 Flash setup those GiB compete with experts kept in VRAM, so the CPU lane is worth testing first.", "setup": { "engine": "whisper.cpp master 231638a (docs/source), llama.cpp PR #21863", "models": "Whisper large-v3 / large-v3-turbo ggml (f16, q8_0, q5_0)" }, "metrics": [ { "label": "R9700 tiny.en, 11 s clip, total", "value": 205.72, "unit": "ms" }, { "label": "RX 9070 XT Vulkan large-v3-turbo, 19.3 min talk", "value": 48, "unit": "s" }, { "label": "RX 9070 XT large-v3-turbo WER (TED talk)", "value": 1.5, "unit": "%" }, { "label": "Strix Halo large-v3-turbo 11 s clip, Vulkan vs HIP", "value": "0.84 / 0.63", "unit": "s" }, { "label": "large-v3-turbo q8_0 file", "value": 0.814, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/whisper.cpp/pull/3823", "https://pypi.org/project/babelscribe/", "https://github.com/collabora/WhisperLive/issues/520", "https://github.com/maherr/onnxruntime-migraphx-rdna4/blob/main/witness/README.md", "https://github.com/ggml-org/whisper.cpp/blob/master/examples/server/server.cpp", "https://github.com/ggml-org/llama.cpp/pull/21863", "https://huggingface.co/ggerganov/whisper.cpp" ], "next": "Measure HIP, Vulkan and CPU-only Whisper lanes on a dual-R9700 box, and how much a loaded speech model slows Flash decode." }, { "id": "2026-10-25-flash-next-vision-cost", "date": "2026-10-25T19:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Vision on Qwen3.8-Flash-Next with R9700s: the encoder is cheap, the image tokens are not", "summary": "Flash-Next reads images through a separate 0.84 GiB projector (27-layer ViT, patch 16, merge 2, so one image token per 32x32 pixels). On an R9700 the GPU encoder takes 57 ms for a 260-token picture and 0.40 s for 972 tokens (3.13 s and 14.1 s on a CPU encoder). With llama.cpp's default cap of 4,096 tokens per image, a 1920x1080 screenshot becomes 2,040 tokens, so on a hybrid setup that prefills around 200 tok/s the image tokens, not the encoder, dominate time to first token. Capping images at 1,024 tokens halves that.", "details": "Estimate, not a measurement: at a 205 tok/s hybrid prefill a default-cap 1080p screenshot needs about 0.9 s of encoding plus about 10 s of prefill; at a 1,024-token cap about 5.3 s. llama.cpp allows 8 to 4,096 tokens per Qwen-VL image by default and warns that grounding tasks want at least 1,024; Strata caps pictures at 1,024 by default. Strata's GPU reader reserves about 1.4 GiB of VRAM (fewer cached experts, a few percent slower text) and can start on demand and stop after 120 s idle. llama.cpp can pin the encoder to one card with --mmproj-device or keep it on the CPU with --no-mmproj-offload. On a DGX Spark, GGUF plus projector scored 58/60 on a 60-image synthetic set, the same as NVFP4; both misses were output-length truncations.", "setup": { "model": "Qwen3.8-Flash-Next GGUF + mmproj-BF16/F16 (Unsloth)", "engine": "llama.cpp master d854d98bae (mtmd), Strata v0.2.6 docs" }, "metrics": [ { "label": "Projector size (F16)", "value": 0.842, "unit": "GiB" }, { "label": "R9700 GPU encode, 972-token picture", "value": 0.4, "unit": "s" }, { "label": "CPU encode, 972-token picture (EPYC)", "value": 14.1, "unit": "s" }, { "label": "1920x1080 screenshot at llama.cpp default cap", "value": 2040, "unit": "image tokens" }, { "label": "Strata GPU reader VRAM reserve", "value": 1.4, "unit": "GiB" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/clip.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/common/arg.cpp", "https://github.com/Niko1221/Strata/blob/main/docs/AMD_HIP.md", "https://github.com/Niko1221/Strata/blob/main/docs/DETAILS.md", "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md", "https://0xbakeer.com/flash-next/vision" ], "next": "Measure time to first token, VRAM per card and text decode with the encoder on and off on two R9700s at Q5 and Q6, at 1,024 and 4,096 image-token caps." }, { "id": "2026-10-25-r9700-scoreboard-band", "date": "2026-10-25T18:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 reference band on llama.cpp's community scoreboards (Llama 2 7B Q4_0)", "summary": "Published single-card R9700 results for the standard llama-bench test: Vulkan tg128 126-146 t/s (FA off) and 131-153 (FA on), pp512 5,000-5,900; ROCm/HIP 112.7 / 121.6 tg128 at build 8ea2902. A card well under this band points to a driver, power or PCIe problem, not the hardware. The ROCm scoreboard's R9700 row is stale (93.84 tg128 at bd4ef13); newer builds are about 20% faster.", "details": "On one rig and one build (8ea2902), Vulkan decode was ~8% faster than HIP on this dense model (125.96 vs 112.72 FA off). R9700 Vulkan decode is about 76-89% of an RTX 3090 on Vulkan (171.61 FA on). The same-die RX 9070 XT posted 142.00 FA on (Vulkan, 2026-10-25), so the extra VRAM costs no decode speed. Splitting a model that fits one card across two hurts: a 2x 7900 XTX owner saw tg128 drop from 170.69 to 149.04 with layer split and to 56.35 with Vulkan tensor split.", "setup": { "model": "llama-2-7b.Q4_0.gguf (3.56 GiB)", "command": "llama-bench -ngl 99 -fa 0,1 -sm none (one GPU)", "engine": "llama.cpp Vulkan and HIP, various builds" }, "metrics": [ { "label": "R9700 Vulkan tg128, FA on (best published)", "value": 152.7, "unit": "tok/s" }, { "label": "R9700 Vulkan tg128, FA on (8ea2902)", "value": 131.19, "unit": "tok/s" }, { "label": "R9700 ROCm tg128, FA on (8ea2902)", "value": 121.62, "unit": "tok/s" }, { "label": "R9700 ROCm scoreboard row tg128, FA off (stale bd4ef13)", "value": 93.84, "unit": "tok/s" }, { "label": "RTX 3090 Vulkan tg128, FA on", "value": 171.61, "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/10879", "https://github.com/ggml-org/llama.cpp/discussions/15021" ], "next": "Measure both local R9700s with the same test (Vulkan and HIP, each card alone) and check them against this band before further dual-card Flash tuning." }, { "id": "2026-10-25-r9700-driver-stack-traps", "date": "2026-10-25T17:15:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 'regressions' that are really the driver stack: AMDVLK vs RADV, Mesa version, and a silent SMU hang", "summary": "A review of public R9700 (gfx1201) reports found that the biggest swings come from the driver stack, not model flags. AMDVLK's Vulkan driver can silently take priority over RADV: one reported -39% MoE prefill regression was really AMDVLK, and on an R9700 AMDVLK's dense prefill was about 4x slower than RADV (207 vs 823 tok/s on a 27B). One tester saw Mesa 25.3.6 give +48% prefill on a 35B MoE over Ubuntu's 25.2.8. Two alarming R9700 Vulkan reports (an 8x decode drop tied to one llama.cpp PR, and a Qwen3.6-35B decode cliff after ~32 tokens) were never confirmed. The real safety gap: a hung power-management controller can leave a card unreadable by every sensor while it can still draw 300 W.", "details": "Check the llama-bench device line: RADV prints '(RADV GFX1201) (radv)' with shared memory 65536, AMDVLK prints '(AMD open-source driver)' with 32768; pin AMD_VULKAN_ICD=RADV. The dmesg line 'SMU driver if version not matched' (driver 0x2E vs card 0x32/0x33) shows up on every R9700 and AMD staff call it harmless. AMD says MES firmware 0x8B fixes a stuck-busy idle state (100% busy, ~100 W at idle), but one dual-R9700 owner saw about 10.8% lower decode on that package. On a three-card host, one card's SMU hung while idle and only a reboot cleared it; health checks that skip unreadable sensors go blind, so treat an unreadable sensor as an alarm. Docked or external cards can pick the wrong runtime power method; amdgpu.runpm=0 fixed one case. Our own measured single-card Qwen3.6-35B Vulkan decode (~101 tok/s) shows the decode-cliff report does not apply to our current build. Mesa and kernel deltas are single-tester results.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201)", "engine": "llama.cpp Vulkan (RADV / AMDVLK), amdgpu kernel driver", "engine_version": "per source", "model": "various (per source)", "quant": "per source" }, "metrics": [ { "label": "AMDVLK dense 27B prefill on R9700", "value": 207, "unit": "tok/s" }, { "label": "RADV dense 27B prefill on R9700", "value": 823, "unit": "tok/s" }, { "label": "Mesa 25.3.6 vs 25.2.8, 35B MoE pp2048 (one tester)", "value": 48, "unit": "%" }, { "label": "Decode drop after MES 0x8B package (one dual-R9700 report)", "value": 10.8, "unit": "%" }, { "label": "Power a sensor-blind card can still draw", "value": 300, "unit": "W" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/21043", "https://github.com/ggml-org/llama.cpp/issues/22375", "https://github.com/ggml-org/llama.cpp/issues/26679", "https://github.com/ggml-org/llama.cpp/issues/26795", "https://github.com/ROCm/ROCm/issues/6101" ], "next": "Fingerprint the stack (kernel, Mesa, ICD, firmware, llama.cpp build) on both desktops and record it with every benchmark; A/B RADV vs AMDVLK only if AMDVLK is installed." }, { "id": "2026-10-25-ep-vs-tp-moe-imbalance", "date": "2026-10-25T16:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "vLLM expert parallel on 2x R9700: no extra PCIe traffic, but ~25% routing imbalance at batch 1", "summary": "On one box, vLLM's --enable-expert-parallel with TP=2 and DP=1 still uses all-reduce, not all-to-all, so it adds no PCIe traffic; it only changes how experts are placed. A routing simulation for Qwen3.8-Flash-Next (512 experts, top-10) shows the busier card reads 1.25x the routed-expert bytes of plain sharded TP at batch 1 (p95 1.60x), shrinking to 1.04x at 32 concurrent streams. On a 4-card setup it is 1.68x at batch 1. For single-stream chat, keep plain -tp 2; try EP only for many-slot batch serving, and never DP+EP over PCIe.", "details": "Sharded TP gives each card half of every active expert, so the work is equal every token. EP gives each card full copies of the experts it owns, and the all-reduce waits for the card that drew more of the token's 10 experts. Analytic check: E[max(X,10-X)]/5 with X~Binomial(10,0.5) = 1.246. Round-robin placement and EPLB do not help at batch 1 because the imbalance is per-token chance, not a hot-expert bias; EPLB also costs VRAM. Estimate: routed experts are roughly 40% of Flash-Next's per-token weight reads by parameter count, so the single-stream penalty would be about 10% on 2 cards and about 27% on 4, unless the half-width sharded expert kernel is much slower on RDNA4. This is a model, not a measurement; a TP vs TP+EP benchmark is planned.", "setup": { "model": "Qwen3.8-Flash-Next (config), 256-expert top-8 class for comparison", "engine": "vLLM (behavior per docs and AMD blog)", "gpus": "2x Radeon AI PRO R9700 (also modeled 4x)", "flags": "-tp 2 vs -tp 2 --enable-expert-parallel" }, "metrics": [ { "label": "EP/TP slowest-card expert bytes, 2 GPUs, batch 1", "value": 1.25, "unit": "x" }, { "label": "same, batch 32", "value": 1.04, "unit": "x" }, { "label": "same, 4 GPUs, batch 1", "value": 1.68, "unit": "x" }, { "label": "AMD MI300X: TP8+EP vs DP8+EP throughput at concurrency 64 (DeepSeek-R1)", "value": 52, "unit": "% higher" } ], "sources": [ "https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html", "https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/", "https://github.com/vllm-project/vllm/blob/main/vllm/config/parallel.py", "https://0xsero.github.io/blackwell-gpu-wiki/interconnect/moe-parallelism/" ], "next": "Benchmark plain TP2 vs TP2+EP in vLLM at 1, 4, 16 and 32 concurrent streams on a 4-bit-or-better MoE that fits two R9700s." }, { "id": "2026-10-25-new-models-moe-active-params-audit", "date": "2026-10-25T15:25:00-05:00", "project": "idea-lab", "kind": "result", "title": "Estimating MoE active parameters and GGUF sizes from Hugging Face: the k/E shortcut is ~40% low, and MTP files fake a Q4", "summary": "An audit of the 84 models on our /new-models page found that the common shortcut active = total x (experts per token / experts) underestimates active parameters by 33-43% on four models with published counts, while subtracting only the routed expert weights lands within +1 to +25%. It also found that configs nesting MoE keys under text_config made 9 of 30 MoE models (including Qwen3.8-Flash-Next, GLM-5.3-Flash and DeepSeek-V4.1-Flash) display as fully active.", "details": "Shape method: active = total - 3 x hidden x moe_intermediate x experts x MoE layers x (1 - k/E), counting MTP layers as MoE layers. Checks against official active counts: DeepSeek-V3 37B (shortcut 21.4, shape 40.1), Qwen3-30B-A3B 3.3B (1.91 vs 3.35), Qwen3-235B-A22B 22B (14.7 vs 22.2), GLM-4.5-Air 12B (6.9 vs 15.0; its official count appears to leave out MTP and embeddings). For file sizes, the Hugging Face model API returns per-file sizes only when called with ?blobs=true. Split GGUFs must have their shards summed per quant folder, and helper files must be skipped: Qwen3.8-Flash-Next's GGUF repo carries a 3.9 GiB MTP draft head named mtp-...-Q4_K_M.gguf that a filename match reads as the model's Q4_K_M (the real UD-Q4_K_XL is 103.7 GiB). Checkpoints with FP4-packed experts also undercount parameters in the safetensors metadata: MiniMax-M3.1-Flash estimates to 124.5 GiB at Q4 but the measured UD-Q4_K_XL is 232.3 GiB. With measured sizes and a rough rule (2x 32 GB VRAM less ~6 GiB for KV/compute, experts in DDR5 with 16 GB left for the OS), 5 Flash-class MoE models fit two R9700s plus 128 GB DDR5 as a hybrid and 5 more need 256 GB; dense models of that size are not worth running from DDR5. A corrected /new-models data script is prepared.", "setup": { "engine": "Hugging Face Hub API + config.json", "gpus": "fit tiers computed for 2x Radeon AI PRO R9700 (32 GB each) + DDR5" }, "metrics": [ { "label": "k/E shortcut error vs official active params (4 models)", "value": "-33 to -43", "unit": "%" }, { "label": "Shape-method error vs official active params (4 models)", "value": "+1 to +25", "unit": "%" }, { "label": "MoE models shown as fully active (nested text_config)", "value": "9 of 30", "unit": "models" }, { "label": "Qwen3.8-Flash-Next UD-Q4_K_XL, shards summed", "value": "103.7", "unit": "GiB" }, { "label": "MTP draft head misread as Q4_K_M", "value": "3.9", "unit": "GiB" }, { "label": "MiniMax-M3.1-Flash Q4: estimate vs measured", "value": "124.5 vs 232.3", "unit": "GiB" } ], "sources": [ "https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/config.json", "https://huggingface.co/Qwen/Qwen3-30B-A3B/blob/main/config.json", "https://huggingface.co/Qwen/Qwen3-235B-A22B/blob/main/config.json", "https://huggingface.co/zai-org/GLM-4.5-Air/blob/main/config.json", "https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/config.json", "https://huggingface.co/api/models/unsloth/Qwen3.8-Flash-Next-GGUF?blobs=true", "https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF" ], "next": "Ship the corrected script, then show measured vs estimated sizes and two DDR5-hybrid fit columns (128 GB, 256 GB) on /new-models." }, { "id": "2026-10-25-measured-quant-quality-speed", "date": "2026-10-25T15:15:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Measured: Qwen 3.8 Flash quants on dual R9700, quality vs speed", "summary": "Measured on the local desktop (2x R9700, 262K f16 KV, reference Q8_0). Strata 0.2.3.1 UD-Q5_K_XL decodes 58 tok/s short / 52 at 36K with 2,192 tok/s prefill and 24/26 tool calls, so it stays the default. llama.cpp hybrid decode was only 17-22 tok/s (UD-Q5 17.6/17.1/14.2 at 1K/36K/163K; UD-Q4 21.2/19.9/16.0); our earlier bandwidth-model estimate of ~34-45 tok/s was far too optimistic. UD-Q5 drifts least from Q8_0 (KLD 0.036 text / 0.019 code vs 0.060/0.034 for UD-Q4).", "details": "llama.cpp (decode ~1K/36K/163K; prefill 36K/163K; KLD text/code; tools): UD-Q5_K_XL 147 GB 17.6/17.1/14.2, 226/195, 0.036/0.019, top-1 92.2%/96.3%, 24/26. UD-Q4_K_XL 104 GB 21.2/19.9/16.0, 314/273, 0.060/0.034, top-1 90.1%/95.1%, 24/26. AP-Q5_K_M 104 GB 20.1/20.3/16.5, 330/301, 0.065/0.033, 24/26. AP-Q4_K_XL 94 GB 22.2/21.8/17.6, 416/359, 0.070/0.038, 24/26. Strata UD-Q5: 25/26 tools with thinking on; greedy exact match vs Q8_0 37.5% (llama.cpp UD-Q5 43.8%). vLLM community MXFP4: best tools 25/26 but lowest exact match 18.8%; KLD run failed, not speed-benched. KLD = llama-perplexity KL vs Q8_0 on held-out post-Oct-1 corpus (152K text / 51K code tokens); 26-case tool-call eval. Q3 estimate: not applicable for this project." }, { "id": "2026-10-25-strata-mmap-experts-ab", "date": "2026-10-25T15:14:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Measured: Strata mmap-experts on vs off", "summary": "On the local desktop with Strata UD-Q5_K_XL, mmap-experts=on left decode unchanged, halved load time (~27 s vs 48-61 s) and cut process RAM from 109 to 19 GB, but prefill dropped 37% (1,377 vs 2,189 tok/s at 36K). Use it when RAM or load time matters more than long-prompt ingest." }, { "id": "2026-10-25-llamacpp-fit-autoplace", "date": "2026-10-25T14:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama.cpp --fit on dual R9700 + DDR5: what it changes silently, and how to A/B it", "summary": "llama.cpp's automatic placement (--fit) is on by default. It only adjusts arguments you left unset: set any of -ngl, -ts or -ot and placement is entirely yours. When a model does not fit it shrinks the context first (down to 4096 by default) before moving tensors to RAM, so a launcher without an explicit -c can quietly lose its long context.", "details": "For MoE models it fills the last GPU with dense-only layers (experts in system RAM), then converts layers to full GPU layers front-to-back, letting parts of a layer overflow to the next GPU or RAM. Default margin is 1024 MiB per device and can be set per device (-fitt 512,1024). llama-bench now accepts -fitt and -fitc, so fitted and hand-written placements can be compared in one tool; llama-fit-params prints the chosen -c/-ngl/-ts/-ot. It is not available with -sm tensor. Integrated GPUs are eligible targets, so a desktop CPU's iGPU that shows up as a ROCm or Vulkan device should be excluded with --device. A user report on two Arc Pro B70s found an explicit -ts 1,1 silently spilling to one card plus RAM with fit left on (about 10x slower) until -fit off. Upstream's own CUDA table shows a second GPU can lower prefill while the model is still RAM-bound (Qwen3 Next q8_0 pp4096 904 to 574 t/s, tg 24.3 to 28.3). Fit measures memory before warm-up, so leave headroom for long prompts.", "setup": { "engine": "llama.cpp", "engine_version": "master 6050b467 (2026-10-25)", "gpus": "2x Radeon AI PRO R9700 (applies to any multi-GPU + RAM setup)" }, "metrics": [ { "label": "Default --fit-ctx floor", "value": 4096, "unit": "tokens" }, { "label": "Default --fit-target margin per device", "value": 1024, "unit": "MiB" }, { "label": "Upstream 2x vs 1x RTX 4090 pp4096, Qwen3 Next q8_0 (RAM-bound)", "value": "574 vs 904", "unit": "t/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/18049", "https://github.com/ggml-org/llama.cpp/pull/18679", "https://github.com/ggml-org/llama.cpp/issues/22592", "https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md" ], "next": "Measure fitted vs hand-tuned placement for Flash-class Q4/Q5/Q6 on two R9700s with DDR5 experts, at 0 and 32K depth." }, { "id": "2026-10-25-hip-graphs-r9700", "date": "2026-10-25T13:10:00-05:00", "project": "dual-r9700", "kind": "progress", "title": "HIP graphs on R9700 in llama.cpp: the real off switch, the new knobs, and a build trap", "summary": "llama.cpp has captured decode as HIP graphs by default since April 2026 (upstream cites +3-14% decode on other AMD GPUs). The working runtime off switch is GGML_CUDA_DISABLE_GRAPHS; a dual-gfx1201 bug report concluded env vars can't disable graphs, but it tried GGML_CUDA_NO_GRAPHS, which the code never reads. Two new knobs are worth an A/B: GGML_CUDA_GRAPH_CHUNK_SIZE and GGML_CUDA_GRAPH_OPT=1.", "details": "From llama.cpp master source (2026-10-25): the only graph compatibility check left is the MUL_MAT_ID sync fallback, and batch-1 quantized MoE decode takes the graph-safe path. Each GPU split gets its own graph, which needs two identical calls to warm up and resets on any shape change, so with experts in DDR5 a GPU's work is cut into many small graphs and the gain there is unmeasured. gfx1201 has no VMM, so llama.cpp uses its legacy pool, and every pool flush forces all graphs to be captured again. GGML_CUDA_GRAPH_CHUNK_SIZE (default 512 nodes) landed 2026-10-23 with +1-2% decode on CUDA; GGML_CUDA_GRAPH_OPT=1 (concurrent streams) has been allowed per split on multi-GPU since 2026-09-03, +3.4-3.7% MoE decode on CUDA. GGML_CUDA_GRAPH_DEBUG=1 with verbose logging prints warm-up, reset and reuse events. Build trap: CUDA/HIP builds fail between the chunked-graphs merge and its fix (commit 9ff0d7e90f), so don't pin a commit in that window. Graphs only exist in the HIP/CUDA backend; Vulkan launchers are unaffected. No R9700 measurement yet; an A/B is planned.", "setup": { "engine": "llama.cpp (source read)", "engine_version": "master 6050b467 (2026-10-25)", "gpus": "Radeon AI PRO R9700 (gfx1201)", "flags": "GGML_HIP_GRAPHS=ON (default)" }, "metrics": [ { "label": "Upstream decode gain from HIP graphs, qwen3moe 30B-A3B Q4_K_M (gfx908)", "value": 12, "unit": "%" }, { "label": "Upstream decode gain from HIP graphs, gfx1100 range", "value": "3-7", "unit": "%" }, { "label": "GRAPH_OPT=1 MoE decode gain, 2 GPUs (CUDA)", "value": "3.4-3.7", "unit": "%" }, { "label": "Graph chunking decode gain (CUDA)", "value": "1-2", "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/22254", "https://github.com/ggml-org/llama.cpp/issues/23579", "https://github.com/ggml-org/llama.cpp/pull/30804", "https://github.com/ggml-org/llama.cpp/pull/31053", "https://github.com/ggml-org/llama.cpp/pull/28198", "https://github.com/ggml-org/llama.cpp/pull/30327", "https://github.com/ggml-org/llama.cpp/issues/25082", "https://github.com/ggml-org/llama.cpp/blob/6050b467c07a95c7b87debd4ddac2f8ba8d9b807/ggml/src/ggml-cuda/ggml-cuda.cu" ], "next": "A/B on R9700: default vs graphs off vs chunk 0/128 vs GRAPH_OPT=1, single-GPU MoE and the dual-R9700 Flash hybrid, with graph event counts." }, { "id": "2026-10-25-easy-apps-dual-r9700-traps", "date": "2026-10-25T12:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "Ollama, LM Studio and Lemonade on two R9700s: what they do with the second card, and five traps", "summary": "From the apps' docs and bug trackers: Ollama puts a model that fits on one GPU onto one GPU only and defaults to a 4096-token context. A Ryzen desktop iGPU (gfx1036) can show up as an extra ROCm device, and llama.cpp-based apps that auto-split onto it segfault; list only the cards with --device ROCm0,ROCm1. Ollama MoE models crash on mixed R9700 + other Radeon pairs because the rocBLAS fix is not in any ROCm 7.2.x release, which Ollama bundles.", "details": "Ollama's Linux ROCm v7 list includes the R9700, but its Windows ROCm list has no RDNA4 cards, so Windows R9700 users run on Ollama's Vulkan backend. The mixed-card MoE crash ('no kernel image is available') was reproduced on Ollama 0.32.14 through 0.34.1 with R9700 + RX 9060 XT and R9700 + RX 7900 XTX; two identical R9700s are not affected. The rocBLAS fix (rocm-libraries #4781) merged 2026-03-02. On kernels without the gfx12 MES idle-power fix (in 6.12+ stable), ROCm runtimes can keep a card 100% busy at up to ~300 W between prompts until the model unloads (reported on dual R9700 in LM Studio). LM Studio's ROCm runtime has an open report of a 27B dense model hanging at 99% load on an R9700 where the Vulkan runtime loads it. llama.cpp's own multi-GPU doc marks -sm row deprecated and -sm tensor experimental with no performance guarantees outside CUDA, so layer split remains the safe default. Not measured on our hardware yet.", "setup": { "gpus": "2x Radeon AI PRO R9700", "engine": "Ollama / LM Studio / Lemonade (llama.cpp-based)", "engine_version": "per source" }, "metrics": [ { "label": "Ollama default context", "value": 4096, "unit": "tokens" }, { "label": "Ollama default parallel requests per model", "value": 1, "unit": "" }, { "label": "Kernel with gfx12 idle-power MES fix", "value": "6.12+", "unit": "" }, { "label": "Reported idle draw with the bug (per card)", "value": 300, "unit": "W" } ], "sources": [ "https://docs.ollama.com/gpu", "https://docs.ollama.com/faq", "https://github.com/ollama/ollama/issues/18162", "https://github.com/lemonade-sdk/llamacpp-rocm/issues/96", "https://lemonade-server.ai/docs/dev/backends-reference/", "https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1991", "https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1916", "https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md" ], "next": "Check whether the iGPU is visible as a device on our dual-R9700 desktop, and measure Ollama against raw llama.cpp on the same 35B Q5 file." }, { "id": "2026-10-25-longctx-quant-evidence", "date": "2026-10-25T11:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Short-context KL can't settle Q4 vs Q5 vs Q6 for Flash models at 128K+", "summary": "Published studies show the 4-bit penalty grows with context. 8-bit stays near-lossless, but 4-bit methods lose up to 16% on RULER at 64K+ inputs and decline further past 75K. The public long-context evidence for Qwen3.8-Flash-Next at Q4 vs Q6 is only 4 items each, so our UD-Q5_K_XL sweet-spot pick is unproven past short context.", "details": "arXiv 2505.20276 tested FP8, int8, AWQ/GPTQ int4 and nf4 on Llama 3.1 and Qwen 2.5. The average 8-bit drop was about 0.8%, the worst 4-bit cell dropped 59%, and the authors suggest RoPE rounding error accumulating over long context as a cause. These are not GGUF k-quants. Red Hat measured INT W4A16 accuracy recovery of 85-88% at 128K on RULER, against an already-low baseline. One 2026 Qwen3-8B study at 4K found Q8_0 through Q4_K_M within noise of BF16 and first damage at Q3_K_S. In the same study, distance hurt more than precision: the BF16 two-fact join fell from 0.538 to 0.066 between 4K and 23K. For Flash-Next, Unsloth's short-context mean KLD is 0.0469 (UD-Q4_K_XL) and 0.0304 (UD-Q5_K_XL), with top-1 agreement of 92.26%, 93.68% and 94.09% (Q6). One 2x RTX 3090 review at 131K scored Q4 and Q6 4/4 each on long-context items. Flash-Next mixes Gated DeltaNet layers with QSA sparse attention (a 2048-token budget), so results from dense models may not transfer. We built a paired 160-item probe (literal needle, near-duplicate keys, 3-hop chain, no-word-overlap join, count) at 16K-245K to measure it on 2x R9700.", "metrics": [ { "label": "Worst 4-bit drop on RULER/OneRULER, 64K+ inputs (published)", "value": 16, "unit": "%" }, { "label": "Average 8-bit accuracy drop, long-context study", "value": 0.8, "unit": "%" }, { "label": "Flash-Next UD-Q4_K_XL mean KLD (Unsloth)", "value": 0.0469, "unit": "" }, { "label": "Flash-Next UD-Q5_K_XL mean KLD (Unsloth)", "value": 0.0304, "unit": "" } ], "sources": [ "https://arxiv.org/html/2505.20276v3", "https://developers.redhat.com/articles/2024/02/03/how-well-do-quantized-models-handle-long-context-tasks", "https://egordmitriev.dev/blog/2026-08-12-does-precision-loss-break-retrieval-or-does-distance", "https://unsloth.ai/docs/models/qwen3.8-next", "https://peaster.io/articles/local-LLMs/qwen-3-8-flash-next-first-impressions" ], "next": "Run the paired probe on UD-Q6, UD-Q5 and UD-Q4 with the same llama.cpp build and f16 KV cache, and publish the per-depth accuracy." }, { "id": "2026-10-25-llama-rpc-two-box-build-parity", "date": "2026-10-25T10:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Two-box R9700 with llama.cpp RPC: the official Docker images can't do it, and both boxes need the identical build", "summary": "Checked llama.cpp master (2026-10-25): the official ROCm and Vulkan Docker images are built without GGML_RPC, which defaults to OFF, so they contain no RPC backend or ggml-rpc-server. RPC also needs byte-identical builds on both machines: the protocol sends raw op enum values, mismatched builds have crashed after a successful handshake, and the protocol version (now 8.0.1, 102 ops) is bumped on every op change. Build one pinned image with RPC on and run it on both boxes.", "details": "The upstream ROCm Dockerfile pins ROCm 7.2.1 and builds from rocm/dev-ubuntu-24.04:-complete. The newest -complete tag is 7.2.4; ROCm 7.14, 10.0 and 10.1 bases are published as -full, so changing only the version points at a tag that doesn't exist. It also compiles 11 GPU targets; gfx1201 alone is enough for R9700. ggml-rpc-server binds to 127.0.0.1 unless given -H and has no authentication, so bind it only to the direct 10GbE/USB4 link address. Its -c tensor cache lives in LLAMA_CACHE (default ~/.cache/llama.cpp/rpc); inside a container that must be a persistent volume or every restart re-sends the worker's weights. RDMA is only negotiated with RoCEv2 NICs; ordinary onboard 10GbE stays on TCP. Estimate, not measured: re-sending ~48 GiB (half of a ~97 GiB GPU-side Q5 Flash pack) at ~1.1 GiB/s practical 10GbE is ~45 s per load without the cache. Any local patch (for example a carried llama.cpp PR) must be identical on both sides too.", "setup": { "engine": "llama.cpp RPC", "engine_version": "master 2280f0f (2026-10-25)", "gpus": "planned 2x2 Radeon AI PRO R9700 over 10GbE" }, "metrics": [ { "label": "RPC protocol version on master", "value": "8.0.1", "unit": "" }, { "label": "ggml op count the protocol is pinned to", "value": 102, "unit": "" }, { "label": "GPU targets in upstream ROCm image", "value": 11, "unit": "" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/.devops/rocm.Dockerfile", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/CMakeLists.txt", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/include/ggml-rpc.h", "https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md", "https://github.com/ggml-org/llama.cpp/issues/20259", "https://github.com/ggml-org/llama.cpp/pull/20330", "https://hub.docker.com/r/rocm/dev-ubuntu-24.04/tags" ], "next": "Build a pinned gfx1201 image with RPC on, confirm it matches native speed on one box, then measure RPC loopback overhead and cold vs warm load time before the second desktop joins." }, { "id": "2026-10-25-vllm-r9700-moe-config-names", "date": "2026-10-25T09:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "vLLM on R9700: tuned MoE configs named 'AI_PRO_R9700' never load, and Flash-Next has no tuned config", "summary": "vLLM looks up fused-MoE tile configs by exact filename. Since PR #38086 (merged 2026-04-02) it names the R9700 'AMD_Radeon_R9700' (PCI id 0x7551), so config files still circulating as 'AMD_Radeon_AI_PRO_R9700' are silently ignored. Upstream ships only three R9700 MoE configs and none for Qwen3.8-Flash-Next's 512-expert shapes, and the stock tuner script rejects the Flash-Next architecture.", "details": "Rename old files (device_name=AMD_Radeon_AI_PRO_R9700 -> AMD_Radeon_R9700) and confirm the server log says 'Using configuration from' instead of 'Using default MoE config'. Flash-Next keys (int4_w4a16, the group size is not part of the key): tensor parallel 2 -> E=512,N=320; TP2 with expert parallel -> E=256,N=640; TP4 -> E=512,N=160; TP4 with EP -> E=128,N=640. On gfx1201, compressed-tensors W4A16 int packs reach the Triton MoE kernel that uses these files (the RDNA3 HIP kernel is gfx1100-only and Marlin is CUDA-only). benchmark_moe.py has no Qwen4Exp branch and stops with 'has no MoE experts'; a small wrapper that reads num_experts/moe_intermediate_size from text_config fixes it, and tuning needs only config.json. Payoff on other R9700 shapes from the cited PRs: 1.05-1.46x per MoE layer (int4, M=1-32), -4.2% ms/token single stream and +13.5% tok/s at 4 concurrent; FP8 TPOT -8% to -24%. Flash-Next numbers are not measured yet.", "setup": { "model": "Qwen3.8-Flash-Next (W4A16 int4 packs)", "engine": "vLLM main (source read 2026-10-25)", "gpus": "Radeon AI PRO R9700 (gfx1201)" }, "metrics": [ { "label": "Upstream R9700 fused-MoE config files", "value": 3, "unit": "files" }, { "label": "Per-layer speed-up, tuned R9700 int4 MoE (other model)", "value": "1.05-1.46", "unit": "x" }, { "label": "TPOT change, tuned R9700 FP8 MoE (other model)", "value": "-8 to -24", "unit": "%" } ], "sources": [ "https://github.com/vllm-project/vllm/pull/38086", "https://github.com/vllm-project/vllm/pull/36659", "https://github.com/vllm-project/vllm/pull/60893", "https://github.com/vllm-project/vllm/issues/50436", "https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py", "https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py" ], "next": "Tune the four Flash-Next keys on R9700 and measure default vs tuned per-layer time." }, { "id": "2026-10-25-rocm-pinned-memory-thp", "date": "2026-10-25T08:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "ROCm's 'pinned' host memory is pageable, and transparent huge pages can make a big-MoE start take 5 minutes instead of 35 seconds", "summary": "On Linux, ROCm implements hipHostMalloc as an anonymous mmap with MADV_HUGEPAGE plus a KFD userptr mapping, so the pages are not locked (VmLck and VmPin stay 0). In a reported 2x RX 7900 XTX test, page-locking a 28.47 GiB expert cache took a median 300.5 s with THP at the default madvise setting versus 34.0 s with THP disabled for the engine process (4 interleaved starts each), with decode and prompt speed unchanged. For anyone keeping 30-100 GiB of MoE experts in DDR5 next to Radeon GPUs, starting the engine with THP off is a free fix for slow, variable starts.", "details": "The slow starts showed thousands of EAGAIN retries of AMDKFD_IOC_ALLOC_MEMORY_OF_GPU (about 3,770 in one 198.6 s start). The reporter's unconfirmed explanation is that 2 MB huge-page faults trigger compaction, which invalidates the HMM walk so ROCr retries the whole range. Workaround: a tiny launcher that calls prctl(PR_SET_THP_DISABLE) and execs the server (no root); allocating in 2 GiB pieces was also fast in the good state (3.6 s). Separately, because RDNA GPUs have no XNACK, reclaim or compaction of these pageable pages evicts all of the process's GPU queues; an open Strata PR that mlocks every GPU-mapped host range reported no stalls in 30 minutes of heavy memory pressure (twice) versus stalls after 103 s and 63 s without it, and needs LimitMEMLOCK=infinity. The older HSA_USERPTR_FOR_PAGED_MEM=0 workaround avoids the stall but stops ROCr pinning the arena and roughly halved prompt reads in that test. AMD's IOMMU guidance is to keep the default remapping mode on Radeon GPUs that use peer-to-peer; iommu=pt is recommended for Instinct and for Radeon without P2P. THP itself is a small speed lever for hybrid inference; the risk it brings here is start time and stalls.", "setup": { "gpus": "2x RX 7900 XTX (start-time test); 1x RX 7900 XTX (stall test)", "engine": "Strata (HIP build)", "engine_version": "0.1.44.1 (start test), 0.1.48.1 + patch (stall test)", "rocm": "7.2.4 (start test)", "flags": "THP enabled=madvise, defrag=madvise" }, "metrics": [ { "label": "Expert cache page-lock, THP default (median of 4)", "value": 300.5, "unit": "s" }, { "label": "Expert cache page-lock, THP off for the process (median of 4)", "value": 34.0, "unit": "s" }, { "label": "Load to ready, THP default (median of 4)", "value": 366.5, "unit": "s" }, { "label": "Same allocation in 15 x 2 GiB pieces (fast state)", "value": 3.6, "unit": "s" } ], "sources": [ "https://github.com/Niko1221/Strata/issues/2850", "https://github.com/Niko1221/Strata/pull/3913", "https://man7.org/linux/man-pages/man2/PR_SET_THP_DISABLE.2const.html", "https://rocm.docs.amd.com/en/docs-6.3.2/conceptual/iommu.html", "https://lists.openwall.net/linux-kernel/2026/06/25/875" ], "next": "A/B the same THP-off launcher on dual R9700 for the Q6 and Q5 Flash hybrid lanes (start time, decode and prefill), then a memlock pressure test." }, { "id": "2026-10-25-r9700-fp8-fp4-kernel-paths", "date": "2026-10-25T07:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 has no FP4 hardware, and vLLM 0.28+ picks a slower FP8 kernel on it by default; one env var gets ~10% back", "summary": "RDNA4 (gfx1201) has matrix instructions for FP16/BF16 (191 TFLOPS), FP8 and INT8 (383) and INT4 (766 TOPS), but none for FP4, so MXFP4/NVFP4 models are unpacked to 16-bit inside the kernel: you get the memory savings, not 4-bit math. On vLLM 0.28 and later, per-token FP8 checkpoints on an R9700 select RowWiseTorchFP8ScaledMMLinearKernel; setting VLLM_DISABLED_KERNELS=RowWiseTorchFP8ScaledMMLinearKernel switches to ChannelWise and, in a reported single-R9700 test, raised decode from 34.71 to 38.48 tok/s (+10.9%) and prefill from 4,152 to 4,531 tok/s (+9.1%) with GSM8K unchanged within noise.", "details": "Measurements are from vLLM issue #57838 (gemma-4-12B FP8 compressed-tensors, AMD rocm/vllm ROCm 10.1 image, vLLM 0.29.1.dev, fp8 KV, single stream, eight boots 38.43-38.54). The fix is not merged as of 2026-10-25; open PR #61091 adds a native Triton per-token FP8 backend (+14-17% per-request output rate vs RowWise at TP1, GSM8K -0.3 pt). Block-scaled FP8 checkpoints (128x128 weight blocks) are not affected by this kernel choice; they use AITER Triton blockscale for shapes in the gfx1201 tuned list (PR #43615) and generic Triton otherwise. MXFP4 MoE on RDNA4 runs through Triton's tl.dot_scaled decomposed onto 16-bit WMMA (PR #37826). llama.cpp's HIP backend runs GGUF k-quants on INT8 WMMA (PR #17156). No mainstream LLM engine uses the INT4 matrix rate yet; it needs W4A4 rotation and calibration, not a normal Q4 file. Separately, AITER unified attention on gfx12 gave silently wrong image answers (text fine) in the same issue thread; use --attention-backend TRITON_ATTN for vision models.", "setup": { "engine": "vLLM (ROCm) and llama.cpp HIP, cited", "engine_version": "vLLM 0.28-0.29.1.dev", "gpus": "Radeon AI PRO R9700 (gfx1201)", "quant": "FP8 W8A8 per-token; MXFP4; GGUF k-quants (all >= 4-bit)" }, "metrics": [ { "label": "Decode, RowWise default", "value": 34.71, "unit": "tok/s" }, { "label": "Decode, RowWise disabled (ChannelWise)", "value": 38.48, "unit": "tok/s" }, { "label": "Prefill, RowWise default", "value": 4152, "unit": "tok/s" }, { "label": "Prefill, RowWise disabled", "value": 4531, "unit": "tok/s" }, { "label": "FP8 matrix peak (dense)", "value": 383, "unit": "TFLOPS" }, { "label": "FP4 matrix peak", "value": 0, "unit": "TFLOPS (no FP4 instruction)" } ], "sources": [ "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html", "https://rocm.docs.amd.com/projects/rocWMMA/en/develop/api-reference/data-type-support.html", "https://github.com/vllm-project/vllm/issues/57838", "https://github.com/vllm-project/vllm/pull/61091", "https://github.com/vllm-project/vllm/pull/43615", "https://github.com/vllm-project/vllm/pull/37826", "https://github.com/ggml-org/llama.cpp/pull/17156" ], "next": "Check the startup log of the dual-R9700 vLLM Flash lane for which FP8 and MXFP4 kernels it selects, and A/B any slow pick." }, { "id": "2026-10-25-rdna4-q6k-hip-prefill", "date": "2026-10-25T06:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp HIP prefills Q6_K at about half speed on RDNA4, which makes UD-Q6 Flash packs ~1.6x heavier than Q5 (fix in an open PR)", "summary": "On RDNA4 (gfx1201, same chip as the R9700), llama.cpp's HIP matrix kernel for Q6_K runs at 36.6 TFLOPS versus 75-80 for Q4_K, Q5_K and Q8_0. Open PR #25940 lifts it to 69.7 (1.9x), with prefill gains of +68-82% on a pure Q6_K 8B model and +28-37% on R9700 Q6_K models; token generation is unchanged. For Qwen3.8-Flash-Next, 80% of UD-Q6_K_XL's expert-matmul time is Q6_K, so by our estimate its GPU expert prefill work is ~1.58x that of UD-Q4_K_XL on HIP master (UD-Q5_K_XL 0.94, AP-Q5_K_M 0.93), dropping to ~0.97 with the PR.", "details": "Kernel numbers are test-backend-ops MUL_MAT at n=512 on an RX 9070 XT from the PR author; the author's 2026-09-09 re-run on master shows a pure Q6_K Llama 8B at pp2048 2,320 -> 3,891 t/s (ub 512) and 2,425 -> 4,422 (ub 2048), Q4_K_M and Q5_K_M +10% (their Q6_K tensors), Q8_0, Q4_0, IQ4_NL and IQ4_XS flat. An R9700 user on ROCm 7.2.4 measured Qwen3.6-27B Q6_K pp512 684 -> 935 t/s (+36.7%) and a 35B MoE Q6_K 2,153 -> 2,757 (+28.1%). The merged RDNA4 MMQ tuning (PR #26199, July) predates those master numbers, and llama.cpp master as of 2026-10-25 has no fix. The Flash pack estimate uses real per-type tensor sizes from the GGUF headers and covers routed-expert matmuls only (attention and PCIe copies excluded); offloaded layers that are PCIe-bound will hide part of the gap. Practical takeaway: on the HIP backend prefer Q5 packs for Flash until #25940 merges, and don't assume HIP's measured Q4_K_M prefill lead over Vulkan carries over to Q6. Vulkan has a smaller Q6_K penalty (open PR #22970 adds +15% on Q6_K). Strata uses its own kernels and is not covered.", "setup": { "model": "Qwen3.8-Flash-Next (estimate); Llama 8B, Qwen3.6-27B, 35B MoE (cited)", "quant": "Q4_K / Q5_K / Q6_K / Q8_0 and UD/AP packs, all >= 4-bit", "engine": "llama.cpp HIP (ROCm)", "engine_version": "master vs PR #25940", "gpus": "RX 9070 XT and Radeon AI PRO R9700 (gfx1201)", "context": "pp512-pp4096", "flags": "-ngl 99, -ub 512 / 2048" }, "metrics": [ { "label": "Q6_K MUL_MAT n=512, HIP master", "value": 36.62, "unit": "TFLOPS" }, { "label": "Q6_K MUL_MAT n=512, with PR #25940", "value": 69.66, "unit": "TFLOPS" }, { "label": "Q4_K MUL_MAT n=512, HIP master", "value": 77.69, "unit": "TFLOPS" }, { "label": "R9700 Qwen3.6-27B Q6_K pp512, master -> PR", "value": "684 -> 935", "unit": "tok/s" }, { "label": "Est. UD-Q6_K_XL expert prefill work vs UD-Q4_K_XL, HIP master", "value": 1.58, "unit": "x" }, { "label": "Est. same with PR #25940", "value": 0.97, "unit": "x" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/25940", "https://github.com/ggml-org/llama.cpp/pull/25587", "https://github.com/ggml-org/llama.cpp/issues/27181", "https://github.com/ggml-org/llama.cpp/pull/26199", "https://github.com/ggml-org/llama.cpp/pull/22970", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF" ], "next": "Measure HIP master vs master + PR #25940 on Flash-Next UD-Q6_K_XL and UD-Q5_K_XL hybrid prefill on 2x R9700 (decode expected flat)." }, { "id": "2026-10-25-r9700-vs-arc-pro-b70", "date": "2026-10-25T04:10:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 vs Arc Pro B70: same 32 GB, but the R9700 is ~1.7-1.9x faster on MoE and ~2x on dual-card Flash at 4-bit+", "summary": "Both cards have 32 GB on a 256-bit bus (640 vs 608 GB/s), yet published results split by model type. Qwen3.6-35B-A3B Q4 on one card: R9700 165.2 tok/s (Vulkan) vs B70 97.7 (SYCL) or 87.3 (Vulkan, Mesa 26). Dense 27B Q4 is close (27.1 vs 21.4-25.3), and one dual-B70 tensor-parallel fork beats the R9700 Vulkan tensor PR on dense 27B (51.1 vs 45.0). For Qwen3.8-Flash-Next at 4-bit or higher on two cards with experts in RAM, the best B70 result we found is 23-33 tok/s against 52-65 (Q5, our measured) and 87-94 at 36K (vLLM MXFP4, our measured) on two R9700s.", "details": "B70 notes: Mesa version is a 2x lever (Vulkan MoE decode 39.0 on Mesa 25.3.6 vs 86.1 on Mesa 26.2.5); SYCL is far faster for long prompts; upstream vLLM XPU serves FP16 only and has no BF16 path, with INT4/FP8/GPTQ via Intel's llm-scaler container. With MTP depth 2 on dense 27B a comparison video reports 46.2 (R9700) vs 28.4 (B70), consistent with the R9700's higher matrix throughput helping multi-token verification. A single-B70 Flash-Next result of 41.7 tok/s exists but uses a 3.05-bit pack, so it is excluded from this 4-bit-and-up comparison. Power: 230 W TBP (B70) vs 300 W (R9700). Newegg listings seen 2026-10-25: R9700 $1,749-$1,999.99, B70 $1,299.99-$2,878, so the B70's launch-price edge has mostly closed at that store. Cost per decode tok/s on two cards for Flash-Next 4-bit GGUF works out to roughly $55 (R9700) vs $78-117 (B70); this is an estimate built from different testers, engines and hosts, so treat all ratios as rough.", "setup": { "model": "Qwen3.6-35B-A3B, Qwen3.8-27B, Qwen3.8-Flash-Next", "quant": "Q4 to Q5 GGUF, MXFP4 (all >= 4-bit)", "engine": "llama.cpp Vulkan/SYCL/ROCm, Strata, vLLM (per source)", "engine_version": "per source", "gpus": "1-2x Radeon AI PRO R9700 vs 1-2x Arc Pro B70", "context": "short unless noted", "flags": "per source" }, "metrics": [ { "label": "35B-A3B Q4 decode, 1 card, R9700 Vulkan", "value": 165.2, "unit": "tok/s" }, { "label": "35B-A3B Q4 decode, 1 card, B70 SYCL / Vulkan Mesa 26", "value": "97.7 / 87.3", "unit": "tok/s" }, { "label": "27B Q4 decode, 1 card, R9700 / B70", "value": "27.1 / 21.4-25.3", "unit": "tok/s" }, { "label": "27B Q4_K_M decode, 2 cards tensor, R9700 / B70", "value": "45.0 / 51.1", "unit": "tok/s" }, { "label": "Flash-Next 4-bit+ decode, 2 cards, best B70 found", "value": "23.0-33.3", "unit": "tok/s" }, { "label": "Flash-Next decode at 36K, 2x R9700 vLLM MXFP4 (measured)", "value": "87-94", "unit": "tok/s" }, { "label": "Est. USD per decode tok/s, 2-card Flash GGUF, R9700 / B70", "value": "55 / 78-117", "unit": "USD" } ], "sources": [ "https://riceharvest.github.io/awesome-b70/performance.html", "https://www.pugetsystems.com/labs/articles/intel-arc-pro-b70-multi-gpu-ai-inference-performance/", "https://github.com/0xSero/qwen38-b70", "https://www.intelinside.ai/runtimes/llamacpp", "https://www.youtube.com/watch?v=pNYNRUOnPeU", "https://github.com/ggml-org/llama.cpp/pull/25051", "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md", "https://www.newegg.com/p/pl?N=100008333+601191745" ], "next": "Watch for a 4-bit-or-higher dual-B70 Flash-Next result on vLLM or llm-scaler and refresh prices before turning this into a full comparison page." }, { "id": "2026-10-25-flash-next-cold-ngram-table", "date": "2026-10-25T03:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Flash-Next's 50 GiB n-gram table is read lazily from disk cache: a cold cache can halve prefill, and 256 GB of DDR5 avoids it", "summary": "Qwen3.8-Flash-Next GGUFs carry one huge per-layer n-gram embedding table (50.66 GiB in UD-Q5_K_XL, read from the file headers). By default llama.cpp does not load it; any tensor over 4 GiB is read row by row from the memory-mapped file during prefill. Published tests show a cold table costs 2.2x prefill on real text (2825 vs 1293 tok/s at UD-Q4_K_XL) and ~5% decode. The direct-read fix was closed unmerged, so on a box with enough RAM the options are keeping the table warm in page cache or loading it with --lazy-mode off. Separately, llama.cpp removed --no-mmap, --mmap, --mlock and -dio in September; launchers must use --load-mode.", "details": "Header-only read of unsloth's UD-Q5_K_XL: per_layer_token_embd.weight is 50.66 GiB of 147.41 GiB total and the only tensor above the 4 GiB lazy threshold; UD-Q4_K_XL's table is 26.82 GiB (IQ4_NL). Current master: --lazy-mode auto reads tensors > 4 GiB on demand and still uses mmap even with --load-mode none; --load-mode auto means mmap on discrete GPUs (only iGPUs opt out), so dual R9700 setups get the lazy path by default. Cited costs, NVIDIA hosts: UD-Q4_K_XL, 24,576-token varied prompt, table in RAM 2825 tok/s prefill and 90.0 tok/s decode vs cold mmap 1293 and 85.9. A low-RAM box that can never keep the table cached saw cold 584 vs warm 2019 tok/s at 3.3K tokens, with the gap shrinking to almost nothing past 130K. Benchmark trap: prompts built from repeated filler text touch only a few table rows and always measure the warm case. The direct-read PRs (#28136, then #29030 '--lazy-mode on-direct') were both closed without merging. Flag change: #20834 folded the old flags into --load-mode (none = old --no-mmap, dio = old -dio) and #28334 removed the old flags on 2026-09-09. Not yet measured on R9700; a cold / warm / lazy-off A/B is planned.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL", "engine": "llama.cpp (cited results on NVIDIA; source read on master b3fa69955e)", "engine_version": "master 2026-10-25", "gpus": "cited: RTX PRO 6000; target: 2x Radeon AI PRO R9700", "context": "24,576-token prompt (cited)", "flags": "--lazy-mode auto vs off; --load-mode" }, "metrics": [ { "label": "UD-Q5_K_XL n-gram table size", "value": 50.66, "unit": "GiB" }, { "label": "Prefill, table in RAM (cited, UD-Q4_K_XL)", "value": 2825, "unit": "tok/s" }, { "label": "Prefill, cold mmap table (cited, UD-Q4_K_XL)", "value": 1293, "unit": "tok/s" }, { "label": "Decode, table in RAM vs cold (cited)", "value": "90.0 vs 85.9", "unit": "tok/s" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF/tree/main/UD-Q5_K_XL", "https://github.com/ggml-org/llama.cpp/pull/28136", "https://github.com/ggml-org/llama.cpp/pull/29030", "https://github.com/ggml-org/llama.cpp/pull/26081", "https://github.com/ggml-org/llama.cpp/pull/20834", "https://github.com/ggml-org/llama.cpp/pull/28334", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model-loader.cpp" ], "next": "A/B on 2x R9700 at Q5: cold table vs pre-warmed vs --lazy-mode off vs --lazy-mode off with --load-mode none, using varied real prompts at 3K/13K/30K tokens." }, { "id": "2026-10-25-r9700-thermal-gating", "date": "2026-10-25T01:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "One badly cooled R9700 can silently cap a dual-card setup: watch the hotspot-to-edge gap", "summary": "A public 2x R9700 investigation traced a ~6-10% per-card prefill deficit to a poor thermal interface, not weak silicon or power delivery. The bad card hit its ~110 C junction limit at ~230 W instead of ~300 W, with a 47 C hotspot-to-edge gap versus ~37-38 C on a healthy card. Because tensor-parallel runs move in lockstep, it dragged the pair's result down too, and the stock fan curve never reacted because it follows edge temperature.", "details": "Deep prefill (pp131072, ROCm 7.14): defective card junction 110 C, edge 62 C, gap 47 C (peak 62), ~229-230 W at ~2790-2867 MHz, 297-300 t/s; healthy card gap ~38 C, ~297 W, ~3050 MHz. Swapping PSU cables between cards left the deficit with the card. After warranty service the same card measured a 37.4 C gap, 297 W, 3057 MHz and 323.2 t/s (+8.4% vs its own earlier run). The author treats the pair's tp=2 prefill of ~1841-1894 t/s as a lower bound next to a ~1965 t/s community 2x R9700 run, and gives ~37 C as healthy and a drift toward ~47 C as a warning. On Navi 48, fan control lives in gpu_od/fan_ctrl/fan_curve (five hotspot-temperature points, 25% minimum); staged points read back as if applied but only take effect after writing 'c' to commit. For scale, single-card sustained loads in reviews: 3,200-3,400 rpm and 57.9 dB(A) (igor'sLAB), 99 C hotspot without throttling in a 7.7-minute FurMark run, 98 C junction on a 32-minute agentic 35B run, and a 210 W cap cutting 97 C to 87 C for a 1.2% token-generation loss. Practical takeaway: before trusting any dual-R9700 benchmark, log each card's junction, edge, power and clock under a long load and compare the cards; a gap of 45 C or more, or one card stuck well below its power cap at a hot junction, points at the cooler rather than the software. Our own earlier note: hybrid DDR5-offload decode draws only ~180-190 W per card, so this mostly bites long prefill and fully in-VRAM tensor-parallel runs.", "setup": { "model": "per source (vLLM FP8 models; 35B MoE GGUF)", "quant": "per source", "engine": "vLLM ROCm tp=2, llama.cpp (published)", "engine_version": "ROCm 7.14, amdgpu DKMS 6.19.4 (thermal investigation)", "gpus": "2x Radeon AI PRO R9700 (investigation); 1x in reviews", "context": "pp131072 deep prefill", "flags": "stock fan curve, 300 W cap" }, "metrics": [ { "label": "Hotspot-to-edge gap, defective card", "value": 47, "unit": "C" }, { "label": "Hotspot-to-edge gap, same card after service", "value": 37.4, "unit": "C" }, { "label": "Sustained board power, defective card", "value": 230, "unit": "W" }, { "label": "Sustained board power, after service", "value": 297, "unit": "W" }, { "label": "Deep prefill gain after service", "value": 8.4, "unit": "%" } ], "sources": [ "https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-mem-vendor-bios-variance.md", "https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit", "https://docs.kernel.org/gpu/amdgpu/thermal.html", "https://www.igorslab.de/en/amd-radeon-ai-r9700-alias-rx-9070xt-in-the-big-workstation-graphics-card-test-what-are-the-benefits-of-double-the-memory-at-work/14/", "https://pendakwah.tech/reviews/r9700ai/", "https://branetechnologies.com/blog/local-35b-model-gpu-thermals-waterblock", "https://slybase.com/en/gpu-inference-tuning-same-performance-quieter-cheaper/" ], "next": "Baseline both of our R9700s under a 20-minute in-VRAM prefill load (per-card gap, power, clock, plus single-card runs) and re-check monthly." }, { "id": "2026-10-25-gpu-max-hw-queues", "date": "2026-10-25T00:40:00-05:00", "project": "dual-r9700", "kind": "result", "title": "GPU_MAX_HW_QUEUES on the R9700: one variable behind idle-busy power, spawn-lottery decode and slow small kernels", "summary": "Four public R9700 (gfx1201) reports point at HIP's hardware-queue count. With it uncapped, idle HIP servers have sat at 100% gfx and 92-95 W per card (=1 gave 21-22 W), vLLM decode landed at 33 or 26 tok/s depending on process start, and small kernels ran ~17 us slower behind pending cross-stream waits. Capping it cost nothing measurable in the one test that checked (0.1% decode, 0.3% prefill).", "details": "Idle: ROCm #5706 single R9700 (MES 0x84, llama.cpp ROCm image) 92-95 W idle with a model loaded; GPU_MAX_HW_QUEUES=1 brought it to 3% / 21-22 W, while --no-warmup, --no-mmap and smaller context changed nothing. PyTorch resident processes stayed pinned even with =1. On ROCm 7.14 (MES 0x8b) a smaller residual remains: HIP 18-19 W vs Vulkan 12-13 W per card on the same 3x R9700 box (ROCm #6453, open). Speed: vLLM on one R9700 was bimodal per spawn (33.1-33.2 vs 25.7-26.2 tok/s); with =1, 8 of 8 cold spawns landed at 33.43-33.52 and the KFD queue count dropped 7 to 3. Two or more streams with pending hipStreamWaitEvent waits added ~17 us to every small kernel (gone at 2 or fewer queues; gfx1030 unaffected). Strata reported its MTP draft step bimodal per start (about 6% of decode on PCIe x16, up to ~25% on a chipset x4 link); =2 fixed every start on both links, =1 was slow on x4, =16 always slow (default 4). Estimate: a 24/7 resident two-card server stuck in the idle-busy state uses ~1,660 kWh/yr of card power (~$264/yr at the July 2026 Texas residential average) vs ~385 kWh/yr (~$61) with the cap. Single-GPU reports only so far; whether a dual-R9700 layer split shows a slow-start mode is untested, so we pin =2 before further A/B runs.", "setup": { "model": "various (Qwen3-14B FP8, Gemma 4 12B FP8, Qwen3.x 35B GGUF, Flash-Next via Strata)", "quant": "per source", "engine": "llama.cpp HIP, vLLM ROCm, Strata (published)", "engine_version": "ROCm 7.2.x to 10, per source", "gpus": "1-3x Radeon AI PRO R9700", "context": "per source", "flags": "GPU_MAX_HW_QUEUES unset vs 1 vs 2" }, "metrics": [ { "label": "Idle power per card, HIP idle-busy (model loaded)", "value": 95, "unit": "W" }, { "label": "Idle power per card, GPU_MAX_HW_QUEUES=1", "value": 22, "unit": "W" }, { "label": "Idle power per card, Vulkan reference (ROCm 7.14 report)", "value": 12.5, "unit": "W" }, { "label": "vLLM decode, fast vs slow spawn (single R9700)", "value": "33.2 / 26.0", "unit": "tok/s" }, { "label": "Extra time per small kernel with pending cross-stream waits", "value": 17, "unit": "us" }, { "label": "Cost of =1 on decode (measured, noise-level)", "value": 0.1, "unit": "%" }, { "label": "Est. yearly card energy, 2 cards idle-busy vs capped", "value": "1664 / 385", "unit": "kWh" } ], "sources": [ "https://github.com/ROCm/ROCm/issues/5706", "https://github.com/ROCm/ROCm/issues/6453", "https://github.com/ROCm/legacy-rocm-build/issues/6347", "https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a" ], "next": "Spawn-lottery A/B on 2x R9700 (unset vs 1 vs 2, 8 cold starts each) for llama.cpp UD-Q5 (then Q4, Q6) and Strata Q6, plus idle watts with the model resident." }, { "id": "2026-10-24-pp-host-experts-gate", "date": "2026-10-24T23:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp PR #29963 runs multi-GPU pipeline parallelism with MoE experts in RAM (+155% prefill at 32K on ROCm), but the usual --n-cpu-moe layout can never trigger it", "summary": "Today llama.cpp switches pipeline parallelism off whenever tensor overrides put MoE experts in system RAM, so expert uploads go through the first GPU and the GPUs take turns. Open PR #29963 lets each GPU upload its own layers' experts. A ROCm test on 4x MI50 (DeepSeek-V4-Flash MXFP4) went from 122.5 to 312.5 tok/s prefill at a 32K prompt (+155%) and +28% at 8K, with decode up about 1.5%. Reading the PR code, it only engages when no single GPU's layer range owns more than 2/3 of the RAM-resident expert bytes. On two GPUs, --n-cpu-moe K puts layers 0..K-1 in RAM, and those all belong to the first card, so the gate fails for every Q4-Q6 Flash-Next layout. A spread override list (half the RAM-resident layers taken from each card's slice) passes it.", "details": "Pinned host memory is also needed: in the PR's own tests the default mmap load gave no gain (2029 to 2088 tok/s) while --load-mode none went from 2705 to 5486 on Qwen3.5-35B-A3B with all experts in RAM. Merged PR #30449 splits CUDA-family pinned buffers into 4 GiB pieces so 50+ GiB of experts can be pinned. Other requirements: -sm layer, all layers offloaded, op offload and KQV offload on, and async plus events on every device; the Vulkan backend reports both, but no Vulkan result exists yet. CUDA results from the PR author on 3 GPUs with Qwen3.8-Flash-Next Q8_0: 52K-prompt prefill 1009 to 1808 tok/s and pp32768 1334 to 2474, with tg64 46.2 to 47.1. For Flash-Next on two cards (48 layers, 0-23 on card 1), a spread list that keeps the high-imatrix-energy layers 36-47 in VRAM also balances VRAM per card, so -ts 1,1 should fit. Not yet measured on R9700; expect decode within 2% and prefill gains that grow with prompt length.", "setup": { "model": "Qwen3.8-Flash-Next (layout analysis); DeepSeek-V4-Flash and Qwen3.5-35B-A3B (cited tests)", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (analysis); MXFP4, Q8_0 (cited)", "engine": "llama.cpp PR #29963 (head 3ca7eed, open)", "gpus": "2x Radeon AI PRO R9700 (analysis); 4x MI50 ROCm 7.14 and 3x RTX PRO 4000 (cited)", "context": "131072-262144", "flags": "-sm layer, experts in RAM via -ot spread over both cards, --load-mode none" }, "metrics": [ { "label": "ROCm 4x MI50 prefill at 32K prompt, master", "value": 122.5, "unit": "tok/s" }, { "label": "ROCm 4x MI50 prefill at 32K prompt, PR #29963", "value": 312.5, "unit": "tok/s" }, { "label": "ROCm 4x MI50 prefill gain at 8K prompt", "value": 28.0, "unit": "%" }, { "label": "ROCm 4x MI50 decode change", "value": 1.5, "unit": "%" }, { "label": "Share of RAM experts on card 1 with --n-cpu-moe 24 (2 cards)", "value": 100, "unit": "%" }, { "label": "Gate limit per card", "value": 66.7, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/29963", "https://github.com/ggml-org/llama.cpp/pull/30449", "https://github.com/ggml-org/llama.cpp/pull/24156" ], "next": "A/B on 2x R9700 with HIP: master --n-cpu-moe vs the PR with a spread override list and --load-mode none, Q4 then Q5 then Q6, pp8192/pp32768/tg64." }, { "id": "2026-10-24-r9700-ecc-vram", "date": "2026-10-24T22:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "R9700 ECC hides about 2 GiB of VRAM per card; turning it off is worth an estimated 4-12% on Flash-Next across two cards", "summary": "Radeon AI PRO R9700s ship with VRAM ECC (GECC) on, so Linux shows about 30,576 MiB per card instead of 32,768. Disabling it with the amdgpu.ras_enable=0 kernel parameter (two reboots) frees about 2.1 GiB per card. On a calibrated bandwidth model for Qwen3.8-Flash-Next with experts in DDR5, that moves 2-3 MoE layers onto the GPUs and raises the decode ceiling 4-7% from capacity alone, or 8-12% if a published +9% single-card decode gain from ECC off also holds.", "details": "Estimated decode ceilings, ECC on vs off (capacity only / capacity plus +9% GPU bandwidth): UD-Q4_K_XL 43.2 vs 46.1 / 48.5 tok/s, UD-Q5_K_XL 32.1 vs 34.2 / 35.4, UD-Q6_K_XL 28.7 vs 29.9 / 30.9. GPU-resident MoE layers go from 29/22/20 to 32/25/22 of 48. Our earlier dual-R9700 ceilings (45/34/30) assumed the full 32 GiB, so with ECC left on they are about 3-5% too high. The +9% figure is one source (Qwen3.6-35B-A3B Q4_K, 101 to 110 tok/s on one R9700 in a Vulkan engine) and is not yet reproduced. The trade-off: VRAM bit flips go undetected, so stress-test first and keep ECC on if the kernel log ever reported ECC errors. Check the state with cat /sys/module/amdgpu/parameters/ras_enable (-1 is the default, ECC on) and dmesg | grep GECC.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (model, not measured)", "gpus": "2x Radeon AI PRO R9700", "context": "262144, f16 KV", "flags": "layer split, experts of the first N layers in DDR5 (-ncmoe), no MTP" }, "metrics": [ { "label": "Visible VRAM per card, ECC on", "value": 30576, "unit": "MiB" }, { "label": "Visible VRAM per card, ECC off", "value": 32768, "unit": "MiB" }, { "label": "UD-Q5_K_XL decode ceiling, ECC on (estimate)", "value": 32.1, "unit": "tok/s" }, { "label": "UD-Q5_K_XL decode ceiling, ECC off (estimate)", "value": 34.2, "unit": "tok/s" }, { "label": "Published single-R9700 decode gain, ECC off", "value": 9, "unit": "%" } ], "sources": [ "https://github.com/ROCm/ROCm/issues/5832", "https://lunnova.dev/articles/amdgpu-disabling-ecc/", "https://forum.level1techs.com/t/sapphire-radeon-r9700-not-32gb/244649", "https://zolotukhin.ai/zinc/docs/rdna4-tuning/", "https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md" ], "next": "Measure ECC on vs off with llama-bench on two R9700s at Q4 and Q5, 32K depth, with a soak test before keeping it off." }, { "id": "2026-10-24-placement-aware-quant", "date": "2026-10-24T21:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Q5 experts on GPU, Q4 in DDR5: placement-aware Flash-Next quant only wins if late layers matter more", "summary": "A custom Flash-Next GGUF with UD-Q5 experts on the VRAM-resident layers and UD-Q4 on the DDR5 layers is estimated at ~39 tok/s ceiling on 2x R9700 (UD-Q4 46, UD-Q5 34), but with equal per-layer sensitivity its quality lands right on the Q4-Q5 line: extra GPU bits cost VRAM, which pushes layers to DDR5. Unsloth's public imatrix shows routed-expert inputs in layers 36-47 carry 4-19x the median energy, and llama.cpp's --n-cpu-moe keeps exactly those on the GPU, so it may land near Q5 quality at +14% speed. Needs a KLD measurement.", "details": "Two concrete facts from the llama.cpp source: --tensor-type / --tensor-type-file take regexes and the first match wins, and Flash-Next's expert down tensors have 640 columns, so 256-block k-quants silently fall back (Q4_K to Q5_0, Q5_K to Q5_1, Q6_K to Q8_0). That is why Unsloth's UD files use Q5_1/Q8_0 downs, and why a hand-made 'Q4_K everywhere' recipe still gets 5.5-bit downs; IQ4_NL or Q4_0 are the only 4.5-bit options. Unsloth's own Q8_0-down layer picks (2, 4, 30, 46, 47) do not follow imatrix energy, so energy is a proxy, not a sensitivity measure. Keeping stock UD-Q4's Q8_0-down layers on GPU with -ot is a slight loss in the model (45.4 vs 46.1 tok/s). All speeds are roofline estimates (GPU at 75% of 640 GB/s, 55 GB/s DDR5, 262K f16 KV, layer split), not benchmarks.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / custom mix (all >=4-bit)", "engine": "llama.cpp (model)", "gpus": "2x Radeon AI PRO R9700", "context": "262K", "flags": "layer split, --n-cpu-moe, f16 KV" }, "metrics": [ { "label": "GPU-Q5 / DDR5-Q4 mix, decode ceiling (est.)", "value": 39.1, "unit": "tok/s" }, { "label": "UD-Q4_K_XL ceiling (est.)", "value": 46.1, "unit": "tok/s" }, { "label": "UD-Q5_K_XL ceiling (est.)", "value": 34.2, "unit": "tok/s" }, { "label": "Mix KLD, equal-layer est.", "value": 0.038, "unit": "" }, { "label": "Mix KLD, energy-weighted est.", "value": 0.033, "unit": "" }, { "label": "Layer 47 expert-down input energy vs median", "value": 18.7, "unit": "x" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://unsloth.ai/docs/models/qwen3.8-next" ], "next": "Build the mix from BF16 with Unsloth's imatrix and measure held-out KLD and decode against UD-Q4 and UD-Q5 on the local desktop; adopt only if KLD is within 15% of UD-Q5 at 10%+ more speed." }, { "id": "2026-10-24-flash-depth-decay", "date": "2026-10-24T20:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Qwen3.8-Flash-Next at long context on R9700: keep f16 KV, get the QSA indexer fixes", "summary": "Flash-Next's sparse attention should keep decode nearly flat with depth, but llama.cpp issues show three separate slowdowns: a HIP-only CPU top-k fallback past 1K context (about 20 to 6 tok/s, fixed by PR #27466), an indexer that re-pooled the whole cache every token (pooled-key cache PR #28699, +9% at 63K-114K), and top-k applied as a dense mask. Quantized KV does not help: q8_0 decoded 24-30% slower than f16 at 128K-250K on this model, and on gfx1201 batch-1 quantized KV at head 256 takes a slower kernel.", "details": "Bandwidth estimate for 2x R9700 with experts in DDR5 (UD-Q4, 45 tok/s at short context): about 35 tok/s at 131K and 29 at 262K with dense attention and f16 KV, versus about 44 and 43 if the sparse path and pooled cache both apply. The f16 KV cache at 262K is only about 6 GiB across both cards (q8_0: 3.2 GiB), so the VRAM it would free is worth a few percent at most. Sparse flash attention (PR #28770) was built for CUDA; whether HIP batch-1 decode reaches it on gfx1201 is not yet verified. A patched CUDA fork went from 10.2 to 33.7 tok/s at 250K on 6x RTX 3090, which shows how much is at stake. The k-pool indexer layout (#29751) doubles the indexer cache and mirrors it on every GPU under tensor split (issue #30005).", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "llama.cpp (HIP)", "engine_version": "per source", "gpus": "2x Radeon AI PRO R9700 (estimate); cited runs on gfx1151, gfx1201, A100, RTX 3090", "context": "0-262K", "flags": "layer split, -fa on, --parallel 1" }, "metrics": [ { "label": "HIP decode before top-k fix, d0 to d1024 (gfx1151)", "value": "19.6 -> 6.1", "unit": "tok/s" }, { "label": "Pooled-key cache gain at 63K / 114K (CUDA)", "value": "+9.3 / +9.4", "unit": "%" }, { "label": "q8_0 vs f16 KV decode at 128K / 250K (A100)", "value": "-24 / -30", "unit": "%" }, { "label": "q4_1 vs f16 KV decode at 32K, gfx1201 27B", "value": "20.55 vs 28.48", "unit": "tok/s" }, { "label": "Est. UD-Q4 decode at 131K, dense f16 vs sparse+pooled (2x R9700)", "value": "35 vs 44", "unit": "tok/s (estimate)" }, { "label": "f16 KV cache at 262K, both cards", "value": 6.0, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/issues/27856", "https://github.com/ggml-org/llama.cpp/issues/28012", "https://github.com/ggml-org/llama.cpp/pull/28699", "https://github.com/ggml-org/llama.cpp/pull/28770", "https://github.com/ggml-org/llama.cpp/issues/28734", "https://github.com/ggml-org/llama.cpp/issues/27796", "https://github.com/ggml-org/llama.cpp/issues/30005" ], "next": "Measure decode at 0, 32K, 128K and 200K on 2x R9700 with f16 and q8_0 KV on a current HIP build. If 128K decode is below 75% of short-context decode, look at bringing the sparse decode path to HIP." }, { "id": "2026-10-24-flash-model-shootout", "date": "2026-10-24T19:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Which Flash MoE suits 2x R9700 + DDR5 offload? Qwen3.8-Flash-Next by 2.6-4x (bandwidth estimate)", "summary": "From byte counts read out of the Unsloth GGUF headers: Qwen3.8-Flash-Next reads 6.1-6.7 GiB per token, DeepSeek-V4-Flash 10.1, GLM-5.3-Flash 13.6-16.0 and MiniMax-M3.1-Flash 21.3. On two R9700s with experts spilled to DDR5, that puts the decode ceiling (estimate) at 45/34/30 tok/s for Qwen Q4/Q5/Q6, 17 for DeepSeek-V4 Q4, 11/9/7 for GLM Q4/Q5/Q6 and about 6 for MiniMax.", "details": "Same model as our earlier roofline: GPU at 75% of 640 GB/s, measured 55 GB/s DDR5 read, 2.5 GiB reserve per card, 262K f16 KV, layer split, no MTP. These are upper-bound estimates, not benchmarks; llama.cpp has measured 55-66% of the ceiling on Flash-Next. GLM-5.3-Flash is attention-heavy (6.15 GiB of attention weights read every token at UD-Q4; 12 MLA + 34 linear layers). DeepSeek-V4-Flash and MiniMax-M3.1-Flash store experts as native 4-bit (MXFP4 / NVFP4) in every Unsloth file, so their Q8 files only upgrade the dense part and there is no Q5/Q6 choice. MiniMax keeps full KV in 60 layers (about 30 GiB at 262K, estimate). DeepSeek-V4.1-Flash-1017 is 466 GiB at UD-Q4, too big for 256 GB RAM plus 64 GB VRAM, and its architecture is not in llama.cpp master yet.", "setup": { "model": "Qwen3.8-Flash-Next, GLM-5.3-Flash, DeepSeek-V4-Flash, MiniMax-M3.1-Flash", "quant": "UD-Q4_K_XL to UD-Q6_K_XL (UD-Q8_K_XL where experts are native 4-bit)", "engine": "bandwidth model (llama.cpp layout)", "gpus": "2x Radeon AI PRO R9700", "context": "262144", "flags": "layer split, f16 KV, experts spilled to DDR5, no MTP" }, "metrics": [ { "label": "Qwen3.8-Flash-Next UD-Q5_K_XL ceiling", "value": 34.1, "unit": "tok/s (est.)" }, { "label": "DeepSeek-V4-Flash UD-Q4_K_XL ceiling", "value": 17.0, "unit": "tok/s (est.)" }, { "label": "GLM-5.3-Flash UD-Q5_K_XL ceiling", "value": 8.8, "unit": "tok/s (est.)" }, { "label": "MiniMax-M3.1-Flash UD-Q4_K_XL ceiling", "value": 6.4, "unit": "tok/s (est.)" }, { "label": "Bytes read per token, Qwen vs GLM (UD-Q5)", "value": "6.5 vs 14.6", "unit": "GiB" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF", "https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF", "https://huggingface.co/unsloth/MiniMax-M3.1-Flash-GGUF", "https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp" ], "next": "Measure DeepSeek-V4-Flash UD-Q4 against GLM-5.3-Flash UD-Q5 on the local desktop to check the ~1.6x estimate; Qwen3.8-Flash-Next stays the main dual-R9700 lane." }, { "id": "2026-10-24-reap-q4plus", "date": "2026-10-24T18:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Expert pruning could fit a Q4 Qwen3.8-Flash-Next fully on two R9700s, but it costs world knowledge", "summary": "Community REAP builds of Qwen3.8-Flash-Next drop the least-used experts (512 to 320 or 288 per layer) and keep the survivors bit-for-bit. Pruning alone does not speed up decode: with all experts on CPU, a 256-expert build decoded at 22.0 vs 21.5 tok/s for the full model (prefill 353 vs 223 tok/s). The gain comes only from fitting more experts in VRAM. Almost all public REAP files are Q2/Q3; the only Q4+ ones we found are a REAP-288 Q4_K_M (78.05 GiB) and Q5_K_M (87.37 GiB). Because each layer's experts are contiguous in a GGUF, a published 320-expert keep-list can also be applied to Unsloth's UD-Q4_K_XL by byte copy, giving an estimated 76.8 GiB file whose compute path fits two 32 GB cards (about 95% VRAM at 262K context). Our roofline estimate: about 74 tok/s decode vs about 45 for the unpruned UD-Q4_K_XL with 17 expert layers in DDR5, the same gain we estimated for adding a third card. The cost: the REAP-320 author measured general-knowledge fabrication at 25% vs 0% for the unpruned model on a 12-question probe, while HumanEval held (96.3% vs 95.7%).", "details": "Builders report rare-token reliability and clean termination degrading below about 288-320 kept experts, and that retained routing mass does not rank two prunes; only behavioral tests do. One also reported a silent slowdown on a 32 GB CUDA card: a fully resident config at 98.7% VRAM ran 10.82 tok/s at 98K context vs 116 tok/s at 8K, with only a fit warning in the log, so leave headroom. The REAP-288 Q4_K_M looks faster in our estimate (about 100 tok/s) mainly because its attention tensors are quantized lower than Unsloth's (1.75 vs 3.04 GiB), which is a quality trade too. File sizes from the Hugging Face API; tensor sizes from the GGUF headers; tok/s figures are our estimates, not measurements on R9700.", "setup": "Cited third-party measurements (RTX 5090 and CPU) plus our bandwidth model for 2x Radeon AI PRO R9700: 75% of 640 GB/s per card, 55 GB/s DDR5, 2.5 GiB reserve per card, 262K f16 KV, layer split, no MTP. Not yet measured on our own desktop.", "metrics": [ { "label": "Est. decode, UD-Q4_K_XL unpruned vs 320-expert slice (2x R9700)", "value": "45 -> 74", "unit": "tok/s (estimate)" }, { "label": "Est. decode, UD-Q5_K_XL unpruned vs 288-expert slice + q8_0 KV", "value": "34 -> 67", "unit": "tok/s (estimate)" }, { "label": "CPU-only decode, 256-expert prune vs full model", "value": "22.0 vs 21.5", "unit": "tok/s" }, { "label": "General-knowledge fabrication, REAP-320 vs unpruned (12 questions)", "value": "25 vs 0", "unit": "%" }, { "label": "HumanEval-164, REAP-320 vs unpruned UD-IQ1_M", "value": "96.3 vs 95.7", "unit": "%" } ], "sources": [ "https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-320-GGUF", "https://huggingface.co/AnonimousA/Qwen3.8-Flash-Next-REAP-256-duo-GGUF", "https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-GGUF", "https://github.com/CerebrasResearch/reap" ], "next": "Slice UD-Q4_K_XL with the published 320-expert list, measure decode at 8K and 98K on two R9700s, and run a tool-call plus fabrication gate against the unpruned file before using it for anything beyond coding agents." }, { "id": "2026-10-24-rdna4-fa-head256", "date": "2026-10-24T16:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Qwen 3.5/3.6/3.8 on R9700: make sure your llama.cpp ROCm build has PR #28102, or long-context prefill can run ~2x slower", "summary": "Every Qwen 3.5/3.6/3.8 model (including the Flash-Next MoE) uses attention head size 256. After llama.cpp removed its rocWMMA FlashAttention kernel (PR #26046), head 256 on RDNA4 had no tensor-core attention path and fell back to a scalar tile kernel. Reported on a single R9700 with Qwen3.6-35B-A3B Q4_K_M: pp4096 at 127K context dropped from 777 to 395 tok/s; on dual R9700s with a 122B-A10B, pp512 at 65K dropped from 571 to 344, and rocprofv3 put attention at 25.4 s vs 65.2 s. PR #28102 (merged 2026-09-11, commit 16378d93f) enabled and tuned the WMMA FlashAttention kernel for head 256 on RDNA4: Qwen3.8-27B IQ4_XS pp512 at 150K went from 164 to 457 tok/s on an R9700, and a dual-R9700 27B Q4_K_M tensor-split run measured 747.5 tok/s at 127K vs 341 before (2.2x), beating the old rocWMMA revert (600). Token generation is unchanged by both changes. If you build llama.cpp for ROCm, check `git merge-base --is-ancestor 16378d93f HEAD`.", "details": "Open caveat (issue #28867): the new dispatch sends small multi-token batches (speculative-decoding verify batches, merged parallel slots) to the WMMA kernel once n_q x gqa_ratio_eff > 16. One R9700 owner measured MTP decode on Qwen3.8-27B (24 Q / 4 KV heads) fall from 51.7 to 41.4 tok/s, recovered by raising the threshold to 64 with no prefill loss; another owner on a tuned fork saw the opposite. Reading the quoted gate, Flash-Next (24 Q / 2 KV heads, effective GQA factor 4) keeps plain decode and single-stream MTP with up to 3 draft tokens on the tile kernel, so it should not hit this; two or more parallel slots with MTP would. This is a reading of quoted source, not a measurement. The old rocWMMA path also crashed around 106K tokens of prefill with HIP graphs on gfx1201 (#24961; GGML_CUDA_DISABLE_GRAPHS=1 avoided it), another reason to move to a post-#28102 build instead of reverting.", "setup": "Cited third-party measurements on Radeon AI PRO R9700 (gfx1201), ROCm 7.2.x to 10.x, llama.cpp HIP backend, f16 or q8_0 KV. Not yet measured on our own desktop.", "metrics": [ { "label": "27B pp512 @ 150K, before vs after #28102 (1x R9700)", "value": "164 -> 457", "unit": "tok/s" }, { "label": "27B Q4_K_M pp4096 @ 127K, 2x R9700 tensor split, old master vs post-#28102", "value": "341 -> 747.5", "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M pp4096 @ 127K, rocWMMA vs post-removal tile kernel", "value": "777 -> 395", "unit": "tok/s" }, { "label": "27B MTP decode regression reported in #28867 (threshold 16 vs 64)", "value": "41.4 vs 51.7", "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/28102", "https://github.com/ggml-org/llama.cpp/issues/26220", "https://github.com/ggml-org/llama.cpp/issues/28867", "https://github.com/ggml-org/llama.cpp/issues/24961" ], "next": "Measure Flash-Next UD-Q5_K_XL prefill at 0/32K/128K on two R9700s: pre- vs post-#28102 ROCm builds vs Vulkan, with a kernel trace to confirm which attention kernel runs (its sparse-attention indexer path is unverified)." }, { "id": "2026-10-24-two-box-4x-r9700-link", "date": "2026-10-24T15:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Pooling 4 R9700s across two desktops: use 10GbE, not USB4, and it pays off most at Q5/Q6 (estimates + cited link data)", "summary": "On X870 boards the USB4 ports use an ASMedia ASM4242. Linux host-to-host networking on that chip drops every packet unless the kernel carries a fix that turns off transmit end-to-end flow control; with the fix it measured 0.28 ms RTT but only 4.2-5.2 Gb/s. A direct 10GbE link gives similar latency (0.17-0.30 ms) at ~9.4 Gb/s. For a llama.cpp RPC layer split with one boundary between boxes, decode is latency-bound, so the estimated network tax is about 5-11% on either link; 10GbE mainly helps prefill (est. ~4% vs ~9% tax) and needs no kernel patch. Estimated Flash-Next decode, 262K context: two boxes x 2 cards keep every >=Q4 pack fully in VRAM (UD-Q5_K_XL est. ~62-66 tok/s vs est. ~51 on three cards in one box; for reference, Strata UD-Q5_K_XL measures 49-52 tok/s on two cards today, while a llama.cpp roofline estimate for two cards is ~34). At UD-Q4_K_XL a third card in one box is slightly better (~74 vs ~66-70) because Q4 already fits in 3 cards.", "details": "Estimates use a bandwidth roofline (75% of 640 GB/s per card, 55 GB/s measured DDR5 read), 2-4 network round trips per token, and a ~3% per-token RPC software tax reported for a 1 local + 1 RPC GPU layer split. Do not put only the experts on the remote box: that layout makes two round trips per layer per token. In vLLM, keep tensor parallel inside each box and use pipeline parallel across boxes; tensor parallel across a network link all-reduces every layer. llama.cpp -sm tensor currently asserts on Qwen3 MoE with 2x R9700 on ROCm, so use -sm layer. Not yet measured on hardware.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimate)", "engine": "llama.cpp RPC, -sm layer (estimate)", "gpus": "2 desktops x 2 Radeon AI PRO R9700", "context": "262K, f16 KV, single request" }, "metrics": [ { "label": "ASM4242 USB4 host-to-host throughput (cited)", "value": "4.2-5.2", "unit": "Gb/s" }, { "label": "10GbE direct link throughput (cited)", "value": "9.41", "unit": "Gb/s" }, { "label": "Est. UD-Q5_K_XL decode, 4 cards over 10GbE", "value": "62-66", "unit": "tok/s" }, { "label": "Est. UD-Q5_K_XL decode, 3 cards one box", "value": "51", "unit": "tok/s" }, { "label": "Est. UD-Q4_K_XL decode, 3 cards one box", "value": "74", "unit": "tok/s" } ], "sources": [ "https://github.com/torvalds/linux/commit/1881f2efbf7f78dc0a79a387b29fde6ff56d3731", "https://qiita.com/faliye/items/2a2d8def5298f6c1c10e", "https://github.com/kjaiswal/llama-cpp-distributed-benchmarks", "https://github.com/ggml-org/llama.cpp/pull/24122", "https://github.com/ggml-org/llama.cpp/issues/22307", "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/", "https://jingchaozhang.github.io/LLM-Inference-Across-NVLink-and-InfiniBand/", "https://www.asrock.com/MB/AMD/X870%20Taichi%20Creator/" ], "next": "Measure it: iperf3 and RTT on the 10GbE link, then llama-bench with --rpc at Q4, Q5 and Q6 once the second desktop is built." }, { "id": "2026-10-24-card-count-resident", "date": "2026-10-24T14:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "3 or 4 R9700s: when Flash-Next fits fully in VRAM, and what 4-card owners report", "summary": "A cited 4x R9700 vLLM tensor-parallel-4 run of Qwen3.8-Flash-Next (MXFP4 experts, FP8 dense, ROCm 10) reports 219 tok/s at 32K context, one request. Our fit model (estimate) says UD-Q4_K_XL becomes fully VRAM-resident on 3 cards at 262K context, and UD-Q5/Q6_K_XL need 4.", "details": "Estimate from measured GGUF header sizes, keeping the n-gram embedding table in system RAM, 262K f16 KV, 2.5 GiB reserve per card, 75% of 640 GB/s, 55 GB/s DDR5. GPU bytes needed: UD-Q4_K_XL 82.9 GiB, UD-Q5_K_XL 102.8, UD-Q6_K_XL 113.0, versus about 59 free on 2 cards, 88.5 on 3 and 118 on 4. Layer-split decode ceilings with no MTP: Q4 45 tok/s on 2 cards vs ~74 resident on 3; Q5 34 vs ~69 resident on 4; Q6 30 vs ~67 on 4 (only ~5 GiB slack). Once resident, the Q4/Q5/Q6 ceilings converge, so a 4-card box would favor Q5 or Q6. Layer split only removes the DDR5 bottleneck; the much larger 219 tok/s figure comes from 4-way tensor parallel. Stock vLLM requires the tensor-parallel size to divide the attention-head count and recommends pipeline parallel for uneven GPU counts, so 3-way TP likely needs a non-stock engine. A cited 3x R9700 tensor-parallel run (~182-191 tok/s) used a fine-tune with an unreported quant, so it is not counted here. Different rigs; not a measurement on our hardware.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (model); MXFP4 experts + FP8 dense (cited)", "engine": "llama.cpp layer split (model); vLLM TP4 (cited)", "gpus": "2-4x Radeon AI PRO R9700", "context": "262K (model); 32K (cited)" }, "metrics": [ { "label": "Cited 4x R9700 vLLM TP4 decode, 1 request", "value": 219, "unit": "tok/s" }, { "label": "Cited 4x R9700 vLLM TP4 prefill, 11.8K input", "value": 3571, "unit": "tok/s" }, { "label": "UD-Q4_K_XL GPU bytes at 262K (estimate)", "value": 82.9, "unit": "GiB" }, { "label": "UD-Q4_K_XL ceiling, 3 cards resident (estimate)", "value": 74, "unit": "tok/s" }, { "label": "UD-Q4_K_XL ceiling, 2 cards + DDR5 (estimate)", "value": 45, "unit": "tok/s" }, { "label": "UD-Q5_K_XL ceiling, 4 cards resident (estimate)", "value": 69, "unit": "tok/s" } ], "sources": [ "https://amd-pro-bench.com/gpus/r9700/4-gpus", "https://amd-pro-bench.com/gpus/r9700/3-gpus", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://docs.vllm.ai/en/stable/serving/parallelism_scaling/", "https://github.com/vllm-project/vllm/issues/11797" ], "next": "Measure a 3-card resident UD-Q4_K_XL run and a 4-card vLLM TP4 run if extra cards are pooled into one box." }, { "id": "2026-10-24-alt-engines-gfx1201", "date": "2026-10-24T13:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Other engines for Flash MoE on R9700 (gfx1201): AMD's HRX backend isn't supported yet, ik_llama.cpp is untested, KTransformers ROCm is stale", "summary": "AMD's native llama.cpp backend (ggml-hrx, draft) does not run on gfx1201 yet: a user hit 'no HRX dispatch registry for target gfx1201' on 2026-10-15 and AMD said support is planned with no timeline. ik_llama.cpp's HIP build was fully broken until a repair merged 2026-08-25, validated only on RDNA3 (gfx1101); it now supports Qwen3.8-Flash-Next and MTP, but nobody has published a gfx1201 run. KTransformers lists ROCm as historical documentation only.", "details": "HRX early data is from RDNA3/RDNA3.5: AMD claims about 30-50% prefill uplift and parity to +15% non-MTP decode vs the faster of Vulkan and HIP. Community Strix Halo (gfx1151) runs on Qwen3-30B-A3B Q4_K_M show tg128 85.1 vs 68.0 tok/s for HIP (+25%), but another run had Vulkan ahead (89.5 vs 85.3); HRX could not chunk prefill past 1024 tokens, and one development branch produced garbage text while llama-bench still reported healthy speed, so check output, not just tok/s. Lemonade ships an HRX engine recipe (merged 2026-10-02) without MTP. The ik_llama.cpp HIP repair also fixed AMD GPUs being detected as NVIDIA (NaNs in MoE matmul) and a software byte-permute that made 4-bit table quants about 15x slower on HIP (IQ4_XS 17.16 to 266.31 tok/s on a small model, RDNA3). Its maintainer says ROCm has not been a project goal and he cannot test it, so expect breakage. Takeaway for dual-R9700 owners today: stay on mainline llama.cpp, Strata or vLLM; ik_llama.cpp is the one alternative worth a gated try for the CPU-expert path.", "setup": { "gpus": "Radeon AI PRO R9700 (gfx1201); sources cover gfx1100/1101/1151", "engine": "ggml-hrx draft, ik_llama.cpp, KTransformers (per sources)" }, "metrics": [ { "label": "HRX vs HIP tg128, Qwen3-30B-A3B Q4_K_M, gfx1151 (community)", "value": 25, "unit": "% faster" }, { "label": "HRX prefill uplift claimed by AMD (RDNA3/3.5)", "value": "30-50", "unit": "%" }, { "label": "ik_llama.cpp IQ4_XS tg before/after HIP fix, small model, RDNA3", "value": "17.16 / 266.31", "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/discussions/27219", "https://github.com/ggml-org/llama.cpp/pull/27218", "https://github.com/lemonade-sdk/lemonade/pull/3723", "https://github.com/ikawrakow/ik_llama.cpp/pull/2339", "https://github.com/ikawrakow/ik_llama.cpp/discussions/562", "https://ktransformers.net/en/docs/hardware-platforms/cpu-gpu-requirements" ], "next": "Gated test of ik_llama.cpp (HIP) vs mainline on 2x R9700 with a Q5 Flash model: build, correctness check, then speed with identical expert placement. Watch for HRX gfx1201 support." }, { "id": "2026-10-24-toolcall-eval-noise", "date": "2026-10-24T12:25:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Q4 vs Q5 for agents: small tool-call evals can't tell the quants apart", "summary": "A published Qwen3.8-Flash-Next engine comparison found llama.cpp UD-Q4_K_XL at 73.0% on 300 BFCL v4 tool-calling samples, but its three repeats ranged 70.7 to 76.3 points, and the multi-turn subset swung by plus or minus 14. Unsloth's quality gap between UD-Q4_K_XL and UD-Q5_K_XL is mean KLD 0.047 vs 0.030. A standard McNemar sample-size calculation shows that confirming a 3-point tool-call gap between two quants needs about 870 paired cases (at 10% disagreement), and a 5-point gap about 310. Before picking the dual-R9700 Flash quant on agent accuracy, compare quants with paired runs on the same cases, require the same sign in every repeat, and treat 'no difference found' as 'gap under about 8 points', not 'equal'.", "details": "The same comparison saw a failure that BFCL missed: one engine printed a tool call into its reply text instead of dispatching it, in a real two-turn agent job. Tool-call checks should count calls leaked into text separately. A pruned community GGUF of the same model also reports a run-to-run band of about 3 cases out of 40 at temperature 0 across backends. Sample sizes are from Connor's approximation (alpha 0.05 two-sided, 80% power) and are estimates; repeats of the same case are not independent, so real needs are higher. We built a 40-case paired tool-call harness for OpenAI-compatible servers (llama.cpp, Strata, vLLM) and will run Q4/Q5/Q6 on two R9700s.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL (published BFCL run); UD-Q4/Q5 KLD from Unsloth", "engine": "llama.cpp (published)", "gpus": "published run on one RTX 6000 Blackwell, not R9700", "context": "131K served", "flags": "temp 0.6, 3 repeats, Q8_0 KV, MTP 3 draft tokens" }, "metrics": [ { "label": "BFCL v4, llama.cpp UD-Q4_K_XL, mean of 3", "value": 73.0, "unit": "%" }, { "label": "BFCL v4 repeat range", "value": 5.67, "unit": "points" }, { "label": "UD-Q4_K_XL mean KLD", "value": 0.046893, "unit": "" }, { "label": "UD-Q5_K_XL mean KLD", "value": 0.030415, "unit": "" }, { "label": "Paired cases to confirm a 3-point gap (estimate, 10% disagreement)", "value": 870, "unit": "cases" }, { "label": "Paired cases to confirm a 5-point gap (estimate)", "value": 312, "unit": "cases" } ], "sources": [ "https://www.demietrich.com/blog/q3.8fn_engine_comparison/", "https://unsloth.ai/docs/models/qwen3.8-next", "https://huggingface.co/Cyronius/Qwen3.8-Flash-Next-131B-A6B-GGUF" ], "next": "Run the paired 40-case tool-call harness on Q4, Q5 and Q6 Flash on two R9700s with identical server flags, 5 repeats each, and publish pass rates, leaked-call counts and McNemar results." }, { "id": "2026-10-24-local-vs-api-cost-flash", "date": "2026-10-24T11:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Dual R9700 vs the hosted Qwen3.8 Flash API: the vLLM 4-bit lane is cheaper per token on power, the Strata Q5 lane is not", "summary": "Using our measured speeds, cited prices and an estimated 465-615 W at the wall, one stream of the vLLM TP2 MXFP4 lane (about 90 tok/s) costs about $0.23-0.35 in electricity per million output tokens, against $0.47 on OpenRouter, and runs about 1.7x the hosted median speed (47-53 tok/s). Strata UD-Q5_K_XL (about 49 tok/s) costs about $0.42-0.64, roughly break-even or worse. Two cards only pay for themselves after 14-16.5 billion output tokens, about 5-6 years running 24x7 at one stream.", "details": "API: OpenRouter qwen/qwen3.8-flash on 24 Oct 2026 lists $0.15 per million input, $0.47 output and $0.016 cached input; the weighted average actually paid for input is $0.0278 because of caching. Electricity: EIA July 2026 residential averages, Texas 15.88 and US 18.31 cents per kWh. Power is an estimate, not measured on our machine: 315 W for both cards (Puget Systems measured 310-315 W for two R9700s at one user under vLLM TP=2 on a 27B model) plus 150 W for a consumer desktop or Puget's ~300 W workstation overhead. Prefill is where local wins: about $0.005-0.014 per million input tokens versus $0.15 uncached, though cached API input ($0.016) is close. If batching scaled the way Puget measured on the 27B model (also an estimate, not measured on Flash), 4-8 streams would bring local to about $0.05-0.09 per million output tokens. Card prices: $1,699.99-$1,999.99 per R9700 on Newegg in October 2026. Idle draw matters for an always-on server: a ROCm 7.14 issue reports 18-19 W per idle R9700 versus 12-13 W on Vulkan.", "setup": { "model": "Qwen3.8 Flash", "quant": "MXFP4-FP8-GPTQ (vLLM) / UD-Q5_K_XL (Strata)", "engine": "vLLM TP2, Strata", "gpus": "2x Radeon AI PRO R9700, local desktop", "context": "36K test" }, "metrics": [ { "label": "vLLM MXFP4 lane, electricity per 1M output tokens (estimate)", "value": "0.23-0.35", "unit": "USD" }, { "label": "Strata UD-Q5 lane, electricity per 1M output tokens (estimate)", "value": "0.42-0.64", "unit": "USD" }, { "label": "Hosted API, per 1M output tokens", "value": 0.47, "unit": "USD" }, { "label": "Hosted API median throughput", "value": "47-53", "unit": "tok/s" }, { "label": "Local prefill, electricity per 1M input tokens (estimate)", "value": "0.005-0.014", "unit": "USD" }, { "label": "Output tokens to pay back two cards (vLLM lane, 1 stream)", "value": "14-16.5", "unit": "billion" } ], "sources": [ "https://openrouter.ai/qwen/qwen3.8-flash", "https://www.eia.gov/electricity/monthly/epm_table_grapher.php?t=epmt_5_6_a", "https://energyfactbook.com/electric-power-monthly/", "https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/", "http://www.nowinstock.net/computers/videocards/amd/aipror9700/", "https://github.com/ROCm/ROCm/issues/6453" ], "next": "Measure GPU and CPU power on our machine during vLLM and Strata runs, plus vLLM aggregate speed at 2, 4 and 8 streams, to replace the estimates." }, { "id": "2026-10-24-ngram-spec-hybrid-offload", "date": "2026-10-24T10:30:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp n-gram speculation on Flash-Next with experts in RAM: off by default, and why", "summary": "Reading llama.cpp master (71255fe, 2026-10-24): ngram-mod only drafts 48-64-token chains, so each verify batch is 49-65 tokens. That is above GGML_OP_OFFLOAD_MIN_BATCH (32), so RAM-resident experts get copied over PCIe for that step instead of being read by the CPU. Also, n-gram types never arm recurrent rollback, so on a hybrid model like Qwen3.8-Flash-Next every drafted round takes a full, host-serialized state checkpoint and a partial accept restores it and replays the accepted tokens. Our bandwidth model (estimate) says default ngram-mod is a net loss for this setup except on near-pure file rewrites.", "details": "Sourced data points: llama.cpp PR #28123 (merged 2026-09-01) gave Flash-Next native recurrent rollback; with an MTP head at n-max 3 on an RTX PRO 6000 it went from 108 tok/s (no draft) to 183 on code and 144 on prose, vs 123 / 83 before. Open PR #28118 measured the host checkpoint at about 600 ms of an 825 ms round on Strix Halo (Vulkan, Flash-Next Q4_K_M, MTP): 32.4 tok/s without drafts, 6.2 with host checkpoints, 41.5 with on-device checkpoints. Open issue #28019: ngram-map-k4v with 3 parallel slots on a GB10 gave about 50 tok/s aggregate vs 66-70 with speculation off, and reports a recurrent-rollback test mismatch on master that affects MTP when slots are reused across conversations. Elsewhere, ngram-mod took a 177B MoE from 22.09 to 77.56 tok/s on a file edit (90.3% accepted) with no change on prose. Our estimate for UD-Q5_K_XL on 2x R9700 (bandwidth model, uniform routing, single slot): rewrite turns 0.87x at defaults, about 1.9x with GGML_OP_OFFLOAD_MIN_BATCH=256 if checkpoints are cheap, about 1.0x if they cost 600 ms; mixed agent turns 0.47x at defaults. Not measured on R9700 yet.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q5_K_XL (model), Q4-Q6 lane", "engine": "llama.cpp", "engine_version": "master 71255fe (source read)", "gpus": "2x Radeon AI PRO R9700", "context": "262K", "flags": "-sm layer, routed experts partly in DDR5, -np 1" }, "metrics": [ { "label": "MTP n-max 3 after #28123, code (RTX PRO 6000, published)", "value": 183, "unit": "tok/s" }, { "label": "No draft, same rig (published)", "value": 108, "unit": "tok/s" }, { "label": "Host checkpoint share of a speculative round (Strix Halo, published)", "value": 600, "unit": "ms of ~825" }, { "label": "ngram-mod verify batch size", "value": "49-65", "unit": "tokens" }, { "label": "Estimated UD-Q5 rewrite speedup, defaults (2x R9700)", "value": 0.87, "unit": "x" }, { "label": "Estimated UD-Q5 rewrite speedup, MIN_BATCH=256, cheap checkpoints", "value": 1.9, "unit": "x" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp", "https://github.com/ggml-org/llama.cpp/blob/master/tools/server/server-context.cpp", "https://github.com/ggml-org/llama.cpp/pull/28123", "https://github.com/ggml-org/llama.cpp/pull/28118", "https://github.com/ggml-org/llama.cpp/issues/28019", "https://ai-muninn.com/en/blog/qwen4exp-177b-three-2080ti", "https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md" ], "next": "Measure on 2x R9700: baseline vs ngram-mod (defaults, MIN_BATCH=256, n-max 30) on rewrite, agent and prose turns, plus the checkpoint cost. MTP with native rollback is the likelier win once a Q4-or-better MTP head is confirmed." }, { "id": "2026-10-24-vllm-tp2-mxfp4-flash-fastest", "date": "2026-10-24T09:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "New fastest Flash setup on dual R9700: vLLM TP2 with a 4-bit MXFP4 pack, 87-94 tok/s decode at 36K", "summary": "vLLM 0.29.0.dev0 (tcclaviger/vllm:latest) with tensor parallel 2 and the tcclaviger MXFP4-FP8-GPTQ Qwen3.8 Flash pack decodes 83-114 tok/s short (story 81-94, code 100-114) and 87-94 tok/s at 36K, with about 3,800 tok/s prefill at 36K. That is about 1.8x Strata Q5 decode (about 49 at 36K) and 1.6x its prefill (about 2,300), and well above the best published 4-bit-or-higher dual-3090 llama.cpp result (37-41 tok/s).", "details": "Pack revision 6fe3e74: experts MXFP4 at 4.25 bpw, n-gram table int6, MTP head 4.25 bpw, so every tensor is 4-bit or higher. 262K context, fp8 KV, card-default max-num-batched-tokens 4096. VRAM 27.55 / 27.40 GB, system RAM 90 GB including a 38.7 GB pinned n-gram table. Startup takes about 10 minutes. MTP acceptance 51.8%. Tool calls parse cleanly and reasoning comes back in a separate 'reasoning' field. 36K prefill runs were 3809 / 3804 / 3789 tok/s on a clean relaunch. An earlier run right after a 110 GB download got only about 1,030 tok/s prefill and logged a 'couldn't lock n-gram table' warning; page-cache pressure is the likely cause but this is unproven, so relaunch on a quiet system before benchmarking. Failed: max-num-batched-tokens 8192 at 262K, 16384 at 131K and 8192 at 131K all fail KV-cache sizing, and no flag to cap it was found. Quality (KL divergence vs a reference) of this pack has not been measured yet. For reference, a CUDA vLLM fork with an expert cache reports 85-90 tok/s on two 3090s, but with a W4A16 fine-tune of the model, so it is not a like-for-like comparison.", "setup": { "model": "Qwen3.8 Flash", "quant": "MXFP4-FP8-GPTQ (tcclaviger), rev 6fe3e74", "engine": "vLLM", "engine_version": "0.29.0.dev0 (tcclaviger/vllm:latest)", "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "context": "262K", "flags": "tensor parallel 2, fp8 KV, MNBT 4096, MTP" }, "metrics": [ { "label": "Decode, short (story / code range)", "value": "83-114", "unit": "tok/s" }, { "label": "Decode at 36K", "value": "87-94", "unit": "tok/s" }, { "label": "Prefill at 36K", "value": 3800, "unit": "tok/s" }, { "label": "Strata Q5 decode at 36K (same machine)", "value": 49, "unit": "tok/s" }, { "label": "Strata Q5 prefill at 36K (same machine)", "value": 2300, "unit": "tok/s" }, { "label": "MTP acceptance", "value": 51.8, "unit": "%" } ], "sources": [], "next": "Measure KL divergence / top-token match of the MXFP4 pack against a Q8_0 reference before calling it the default." }, { "id": "2026-10-24-strata-q5-tuning-ab", "date": "2026-10-24T09:04:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Strata Q5 tuning on dual R9700: the defaults are already about right", "summary": "STAGE_TRIM=1 with layer-split=23 raised 36K decode from 48.5 to 50.3 tok/s (+3.7%) but cut 36K prefill from 2374 to 2171 tok/s (-8.6%), so the default stays. Strata's auto split point (K=23) beat 19, 21, 25 and 27. Speculation depth 3, 5 and 6 were no better than 4, SPLIT_MTP_BATCH had no effect, and adapt-async is refused in layer-split mode.", "details": "Qwen3.8 Flash UD-Q5_K_XL. The n-gram table (per_layer_token_embd, Q8_0, 50.66 GiB) stays in the DDR5 page cache, with only small buffers on the GPUs, so it costs no VRAM.", "setup": { "model": "Qwen3.8 Flash", "quant": "UD-Q5_K_XL", "engine": "Strata", "engine_version": "0.2.3", "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "context": "36K test", "flags": "layer split, MTP" }, "metrics": [ { "label": "36K decode, default", "value": 48.5, "unit": "tok/s" }, { "label": "36K decode, STAGE_TRIM=1 + split 23", "value": 50.3, "unit": "tok/s" }, { "label": "36K prefill, default", "value": 2374, "unit": "tok/s" }, { "label": "36K prefill, STAGE_TRIM=1 + split 23", "value": 2171, "unit": "tok/s" } ], "sources": [], "next": "No further Strata tuning planned; vLLM MXFP4 is now faster." }, { "id": "2026-10-24-llamacpp-moe-cache-mib-measured", "date": "2026-10-24T09:03:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp --moe-cache-mib on dual R9700: about +20-25% decode, but still about 2.5x behind Strata", "summary": "With the HIP dsv4 build on Qwen3.8 Flash UD-Q5_K_XL (262K, f16 KV), the VRAM expert cache lifted warm decode from 18.2 to 23.5 tok/s (story) and 36K decode from 17.0 to 19.6, at a 93.7% hit rate. Prefill fell from 231 to 162 tok/s at 36K because prompt batches only hit the cache 8-15% of the time.", "details": "Warm story/code decode, then 36K decode/prefill, then hit rate: fit only 18.2/18.2, 17.0/231; +6544 MiB 22.0/19.6, 18.0/201, 79.75%; +12000 MiB 23.4/21.0, 19.4/183, 88.74%; -cmoe +40000 MiB 23.5/21.5, 19.6/162, 93.72%. Q6 at the -cmoe +40000 setting: 22.3/20.4, 18.8/147. --moe-cache-mib is a total split across both GPUs, and it can't be combined with tensor split ('MoE cache requires a GPU backend').", "setup": { "model": "Qwen3.8 Flash", "quant": "UD-Q5_K_XL (Q6 for comparison)", "engine": "llama.cpp (HIP, dsv4 build)", "engine_version": "dsv4 build", "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "context": "262K", "flags": "f16 KV, --moe-cache-mib 6544 / 12000 / 40000 with -cmoe" }, "metrics": [ { "label": "Warm story decode, fit only", "value": 18.2, "unit": "tok/s" }, { "label": "Warm story decode, -cmoe +40000", "value": 23.5, "unit": "tok/s" }, { "label": "36K prefill, fit only", "value": 231, "unit": "tok/s" }, { "label": "36K prefill, -cmoe +40000", "value": 162, "unit": "tok/s" }, { "label": "Cache hit rate, -cmoe +40000", "value": 93.72, "unit": "%" } ], "sources": [], "next": "Use a modest cache size if prefill matters; the prompt-batch hit rate is the limiter." }, { "id": "2026-10-24-glm53-flash-q5-dual-r9700", "date": "2026-10-24T09:02:00-05:00", "project": "dual-r9700", "kind": "result", "title": "GLM 5.3 Flash Q5_K_M on dual R9700: about 6 tok/s decode, HIP prefill 2.3x Vulkan, 524K context loads", "summary": "bartowski GLM 5.3 Flash Q5_K_M (227 GiB) on llama.cpp-dsv4 HIP decodes 6.4 tok/s short, 6.0 at 36K (prefill 112.7) and 5.1 at 113K (prefill 74.4). HIP prefill is about 2.3x Vulkan (48.8). The largest context that loads is 524K. Strata and the RDNA4 llama.cpp fork don't support GLM yet.", "details": "q8 KV, ubatch/batch 2048, -t 16, 262K context. VRAM 24.8 / 27.8 GB. Failed: Vulkan with ubatch 4096 hit an amdgpu ring timeout (device lost) at about 29K tokens; stay on HIP or a smaller ubatch.", "setup": { "model": "GLM 5.3 Flash", "quant": "Q5_K_M (bartowski), 227 GiB", "engine": "llama.cpp (HIP, dsv4 build)", "engine_version": "dsv4 build", "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "context": "262K", "flags": "q8 KV, -ub 2048 -b 2048, -t 16" }, "metrics": [ { "label": "Decode, short", "value": 6.4, "unit": "tok/s" }, { "label": "Decode at 36K", "value": 6.0, "unit": "tok/s" }, { "label": "Prefill at 36K", "value": 112.7, "unit": "tok/s" }, { "label": "Decode at 113K", "value": 5.1, "unit": "tok/s" }, { "label": "Prefill at 113K", "value": 74.4, "unit": "tok/s" }, { "label": "Vulkan prefill (for comparison)", "value": 48.8, "unit": "tok/s" } ], "sources": [], "next": "Revisit if Strata or vLLM adds GLM 5.3 support." }, { "id": "2026-10-24-deepseek-v4-flash-q4-dual-r9700", "date": "2026-10-24T09:01:00-05:00", "project": "dual-r9700", "kind": "result", "title": "DeepSeek V4 Flash UD-Q4_K_XL on dual R9700: about 10 tok/s decode, and a full 1M context loads", "summary": "unsloth DeepSeek V4 Flash UD-Q4_K_XL (144.4 GiB) on llama.cpp-dsv4 HIP decodes 10.0 tok/s short (prefill 105-147), 9.1 at 36K (prefill 168.6) and 7.9 at 112K (prefill 107.9), about 1.5x GLM 5.3 Flash. A full 1M context loads, with short decode at 9.4 tok/s.", "details": "Same build and settings as the GLM run, 262K context for the main numbers. VRAM 29.1 / 29.0 GB.", "setup": { "model": "DeepSeek V4 Flash", "quant": "UD-Q4_K_XL (unsloth), 144.4 GiB", "engine": "llama.cpp (HIP, dsv4 build)", "engine_version": "dsv4 build", "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "context": "262K (1M also loads)", "flags": "q8 KV, -ub 2048 -b 2048, -t 16" }, "metrics": [ { "label": "Decode, short", "value": 10.0, "unit": "tok/s" }, { "label": "Decode at 36K", "value": 9.1, "unit": "tok/s" }, { "label": "Prefill at 36K", "value": 168.6, "unit": "tok/s" }, { "label": "Decode at 112K", "value": 7.9, "unit": "tok/s" }, { "label": "Prefill at 112K", "value": 107.9, "unit": "tok/s" }, { "label": "Decode, short, 1M context", "value": 9.4, "unit": "tok/s" } ], "sources": [], "next": "None planned." }, { "id": "2026-10-24-systemd-oomd-mmap-kill", "date": "2026-10-24T09:00:00-05:00", "project": "dual-r9700", "kind": "failure", "title": "systemd-oomd killed a benchmark: a 227 GiB mmap'd model plus a big rsync crossed the memory-pressure limit", "summary": "Running a 227 GiB mmap'd model while a large rsync was copying files pushed memory pressure past 50%, and systemd-oomd killed the whole scope the job ran in. Fix: run heavy model jobs as their own systemd --user units so they are isolated from other work.", "details": "systemd-oomd acts on memory pressure (PSI), not just free RAM, and kills the whole cgroup it picks. Page-cache churn from a large copy next to a huge mmap'd model is enough to trip it. Launch each server with systemd-run --user --unit=NAME (or a .service file) and avoid big copies during benchmarks.", "setup": { "gpus": "2x Radeon AI PRO R9700, local desktop (dual-channel DDR5)", "engine": "llama.cpp (HIP, dsv4 build)" }, "metrics": [], "sources": [], "next": "Fixed by running heavy jobs as separate systemd --user units." }, { "id": "2026-10-24-llama-bench-json-path-privacy", "date": "2026-10-24T08:20:00-05:00", "project": "idea-lab", "kind": "progress", "title": "Sharing llama-bench JSON? It includes the full model path you typed", "summary": "llama-bench's json/jsonl output copies the -m argument verbatim into model_filename, so a shared raw file usually shows your home directory and username. There is no hostname field, and cpu_info/gpu_info are only device names. Trim model_filename to the file name before posting.", "details": "Read from llama.cpp tools/llama-bench/llama-bench.cpp at master df9beac (2026-10-24): each row has 43 fields plus per-run samples_ns/samples_ts; model_type comes from llama_model_desc (e.g. 'qwen3next 80B.A3B Q5_K - Medium'), so the quant can be checked without the GGUF; test_time is UTC. For like-for-like comparisons, rows must match on model_type, n_prompt, n_gen, n_depth, type_k/type_v, flash_attn and n_ubatch. We wrote a converter that keeps only setup fields, reduces the path to the file name, rejects quants below Q4 and flags noisy rows (over 5% spread), fewer than 3 runs and missing long-context rows; it will back a result-submission form when the site's next build lands.", "setup": { "engine": "llama.cpp llama-bench", "engine_version": "master df9beac" }, "sources": [ "https://github.com/ggml-org/llama.cpp/blob/df9beac/tools/llama-bench/llama-bench.cpp" ], "next": "Suggested sharing command: llama-bench -m MODEL.gguf -fa 1 -p 512 -n 128 -d 0,8192,32768 -r 3 -o jsonl" }, { "id": "2026-10-24-cpu-expert-path-repack-mlock", "date": "2026-10-24T07:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "The CPU half of Flash decode in llama.cpp: no x86 repack for Q5/Q6 experts, keep the host buffer, and use -lm mlock", "summary": "Reading llama.cpp master (df9beac, Oct 24): x86 CPU weight repack exists only for Q4_0, Q4_K, IQ4_NL and MXFP4, so Unsloth UD-Q5_K_XL and UD-Q6_K_XL experts (Q5_K, Q6_K, Q8_0) get none, while AP-Q4_K_XL is 100% eligible and UD-Q4_K_XL 58%. With a ROCm or Vulkan GPU loaded, -ncmoe experts go to the GPU host buffer and are never repacked unless --no-host is set, and repacked weights can't be offloaded to the GPU for prefill. For pinned, swap-proof experts, -lm mlock now means no mmap plus mlock again.", "details": "Why: with Flash-class experts in DDR5, the CPU reads most of every token's bytes, so its path matters. Repack: ggml_repack_get_optimal_repack_type() has x86 (AVX2) paths for Q4_0, Q4_K, IQ4_NL and MXFP4 (Q2_K on AVX-512); Q5_K, Q6_K and Q8_0 repack is ARM-only, and the AVX-512 Q6_K repack PR (#19706) is still open. make_cpu_buft_list() lists the GPU host buffer before the repack buffer, and under mmap the loader swaps the host buffer for a plain CPU buffer; ggerganov confirmed repack never triggers with CUDA_Host and -ncmoe (PR #19738). The repack buffer isn't host memory and the scheduler only offloads ops whose weights are host memory, so --no-host trades prefill offload for repack; where repack did trigger (Apple, gpt-oss 20B MXFP4) he measured pp512 0.64x and tg32 1.07x. Verdict: keep --no-host off on dual R9700. Load mode: after the July --load-mode refactor, --load-mode none was reported swapped out under load (25-27 to 7.55 tok/s) and mmap+mlock with -ncmoe was OOM-killed from double residency (issue #26110); PR #26135 (merged Jul 27) made -lm mlock mean no mmap plus mlock, which keeps the pinned host buffer that fast PCIe prefill copies need. Raise the memlock limit (LimitMEMLOCK=infinity) or the lock fails. Untested hypothesis, estimate only: Zen 4/5 desktop CCDs read at 32 bytes per FCLK cycle, 64 GB/s each at 2000 MHz (Chips and Cheese), and our measured 54-56 GB/s read / ~27 GB/s write sits just under one CCD's limits. If the CPU has two CCDs and threads were packing onto one, spreading them could lift a 55 GB/s read to about 70, worth about +14% at UD-Q5 (34 to 39 tok/s ceiling, layer split, no MTP), +16% at UD-Q6 and +10% at UD-Q4. Zero if the CPU has one CCD.", "setup": { "gpus": "2x Radeon AI PRO R9700", "model": "Qwen3.8-Flash-Next GGUFs", "engine": "llama.cpp master df9beac (source read)", "flags": "-ncmoe, --no-host, --repack, -lm mlock, -t / -C / --cpu-strict" }, "metrics": [ { "label": "x86 repack-eligible expert bytes, AP-Q4_K_XL", "value": 100, "unit": "%" }, { "label": "x86 repack-eligible expert bytes, UD-Q4_K_XL", "value": 57.6, "unit": "%" }, { "label": "x86 repack-eligible expert bytes, AP-Q5_K_M", "value": 29.0, "unit": "%" }, { "label": "x86 repack-eligible expert bytes, UD-Q5_K_XL / UD-Q6_K_XL", "value": 0, "unit": "%" }, { "label": "Zen 4/5 desktop per-CCD read limit at FCLK 2000", "value": 64, "unit": "GB/s" }, { "label": "Est. UD-Q5 2x R9700 ceiling if DDR5 read goes 55 to 70 GB/s (estimate)", "value": "34 to 39", "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-cpu/repack.cpp", "https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/src/llama-model.cpp", "https://github.com/ggml-org/llama.cpp/blob/df9beac7153166550571c70cc667f270b745da49/ggml/src/ggml-backend.cpp", "https://github.com/ggml-org/llama.cpp/pull/19738", "https://github.com/ggml-org/llama.cpp/pull/19706", "https://github.com/ggml-org/llama.cpp/issues/26110", "https://github.com/ggml-org/llama.cpp/pull/26135", "https://github.com/ggml-org/llama.cpp/pull/20596", "https://chipsandcheese.com/p/amds-ryzen-9950x-zen-5-on-desktop" ], "next": "Check the CPU's CCD count, measure read bandwidth with threads on one CCD vs both, then a llama-bench thread and CPU-mask ladder and -lm none vs mlock at UD-Q5." }, { "id": "2026-10-24-vulkan-vs-rocm-flash", "date": "2026-10-24T06:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Vulkan beats ROCm for MoE decode on R9700s; in hybrid Flash runs the gap shrinks to ~16-25%", "summary": "Every recent R9700 comparison we found shows llama.cpp's Vulkan backend 1.5-1.8x faster than ROCm/HIP for MoE decode. Fresh 2x R9700 numbers posted today on the Vulkan tensor-parallel PR: Qwen3.6-35B-A3B Q4_K_M decodes at 165.2 tok/s on one card with Vulkan vs 91.5 with ROCm, and 132.6 vs 87.6 with layer split. Dense 27B is a tie in layer split. For Flash-class models with experts in DDR5, the CPU part doesn't depend on the backend, so we estimate ROCm would cost about 16% at UD-Q6, 18% at UD-Q5 and 21-25% at Q4. Our pick for the dual-R9700 Flash lane: Vulkan with layer split.", "details": "Prefill is the exception: on one R9700, HIP leads at long prompts (pp2048 4635 vs 3675, pp4096 4325 vs 3386, Qwen3-30B-A3B UD-Q4_K_XL), but offloaded-expert prefill is PCIe-bound, so that lead should shrink in hybrid runs (to be measured). Vulkan tensor split (llama.cpp PR #25051) is still open: on 2x R9700 it reaches 45.0 tok/s on dense Qwen3.8-27B Q4_K_M (ROCm tensor 40.2, layer split 27), but for a MoE that fits one card it is slower than a single GPU (144.8 vs 165.2). Whether tensor split works with CPU-resident experts for Flash is untested. A real multi-agent workload on one R9700 also found HIP 1.3-1.4x slower at every concurrency level. The hybrid percentages are estimates from our bandwidth model (GPU share of each decode step 37-64%), not measurements.", "setup": { "model": "Qwen3.6-35B-A3B / Qwen3.8-27B (published); Qwen3.8-Flash-Next (estimate)", "quant": "Q4_K_M (published); Q4-Q6 (estimate)", "engine": "llama.cpp", "engine_version": "master + PR #25051 (published)", "gpus": "1-2x Radeon AI PRO R9700", "context": "tg128 / pp512-4096", "flags": "-fa 1" }, "metrics": [ { "label": "35B-A3B Q4_K_M decode, 1 GPU, Vulkan", "value": 165.2, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode, 1 GPU, ROCm", "value": 91.5, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode, 2 GPU layer, Vulkan", "value": 132.6, "unit": "tok/s" }, { "label": "35B-A3B Q4_K_M decode, 2 GPU layer, ROCm", "value": 87.6, "unit": "tok/s" }, { "label": "27B Q4_K_M decode, 2 GPU tensor, Vulkan", "value": 45.0, "unit": "tok/s" }, { "label": "27B Q4_K_M decode, 2 GPU tensor, ROCm", "value": 40.2, "unit": "tok/s" }, { "label": "Est. ROCm decode loss vs Vulkan, Flash UD-Q5 hybrid", "value": 18, "unit": "%" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/25051", "https://github.com/ggml-org/llama.cpp/discussions/22463", "https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html", "https://finaltype.github.io/quant-blog/en/devlog/2026-09/2026-09-26_moe35b-r9700-speed-tuning/" ], "next": "Same-commit Vulkan vs HIP A/B on Flash-Next Q5 with experts in DDR5 on the local desktop; test Vulkan tensor split with CPU-resident experts once the PR merges." }, { "id": "2026-10-24-parallel-slots-offload-cliff", "date": "2026-10-24T05:10:00-05:00", "project": "dual-r9700", "kind": "progress", "title": "Parallel slots with experts in DDR5: a 32-token decode cliff in llama.cpp, and what batching buys on 2x R9700", "summary": "In llama.cpp, a MoE expert op whose weights sit in system RAM is moved to a GPU once a ubatch holds 32 or more tokens (GGML_OP_OFFLOAD_MIN_BATCH, default 32, in both the HIP/CUDA and Vulkan backends). In decode, tokens per step = active slots x (1 + draft tokens), so 32 parallel slots, or just 8 slots with 3 MTP drafts each, switch the DDR5-resident layers from CPU reads to PCIe uploads of the used experts. Our bandwidth estimate says that roughly halves aggregate decode at that point; raising the threshold to about 256 avoids it while large prefill batches still offload.", "details": "Source reading (llama.cpp master 3642238, 2026-10-24): for MUL_MAT_ID the offload batch size is op->ne[2], the tokens in the ubatch; sched_copy_experts() then uploads only the experts the router used (MoE-cache hits come from VRAM), after a synchronous readback of expert ids per layer. Estimates below are bandwidth-only, assume uniform routing (the least overlap between slots, so real sharing should be better), use our measured 55 GB/s DDR5 read and ~23 GB/s achieved on one PCIe 5.0 x8 link, and ignore CPU compute limits, which may cap the CPU path past about 8-16 tokens per step. Qwen3.8-Flash-Next UD-Q5_K_XL, layer split, 23 expert layers in RAM: 34 tok/s at 1 slot, 47 total at 4 slots, 52 at 8, 67 at 31, then about 31 at 32 tokens per step with the default threshold. AP-Q4_K_XL: 60, 97 and 111 at 1, 4 and 8 slots. So parallel slots add only about 1.4-1.5x total throughput at 4-8 slots in this worst-case routing model, and per-agent speed drops fast. Not yet measured on hardware.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q5_K_XL, AP-Q4_K_XL (estimates)", "engine": "llama.cpp (source reading + bandwidth model)", "engine_version": "master 3642238", "gpus": "2x Radeon AI PRO R9700", "context": "262K f16 KV pool, 8K fill per slot", "flags": "-sm layer, -ncmoe per quant, -np N" }, "metrics": [ { "label": "Default offload threshold (tokens per ubatch)", "value": 32, "unit": "tokens" }, { "label": "Est. UD-Q5 aggregate decode, 8 slots", "value": 52, "unit": "tok/s" }, { "label": "Est. UD-Q5 aggregate decode, 31 slots", "value": 67, "unit": "tok/s" }, { "label": "Est. UD-Q5 aggregate decode, 32 tokens/step at default threshold", "value": 31, "unit": "tok/s" }, { "label": "Est. 8 slots x 4 MTP tokens verify step, default vs threshold 256", "value": "1.02 vs 0.46", "unit": "s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-cuda/ggml-cuda.cu", "https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/ggml/src/ggml-vulkan/ggml-vulkan.cpp", "https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/src/llama-context.cpp", "https://github.com/ggml-org/llama.cpp/blob/3642238cc12050c8ab70afdf4b3121ea365c7579/tools/batched-bench/README.md" ], "next": "Measure with llama-batched-bench at 1-48 parallel sequences, default threshold vs GGML_OP_OFFLOAD_MIN_BATCH=256, Q5 first." }, { "id": "2026-10-24-dual-r9700-platform-lanes-ddr5", "date": "2026-10-24T04:10:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Dual R9700 platform: M.2 slots that steal GPU 2's lanes, and what DDR5 speed and channels are worth", "summary": "From manufacturer specs: on the ASUS ProArt X870E-Creator, populating M.2_2 drops the second GPU slot to x4; on the Crosshair X870E Apex, M.2_3 gives x8/x4 and M.2_2 plus M.2_3 disables slot 2; on the Crosshair X870E Dark Hero, slot 2 runs PCIe 3.0 x4 until x8/x8 is set in the BIOS. Our calibrated estimate says AM5 RAM tuning (5200 to 6000 MT/s) is worth about +8% decode at UD-Q5, slipping to AMD's stock DDR5-3600 for four DIMMs costs about 20%, and a 4-channel Threadripper TRX50 platform would be worth about +52%.", "details": "Why it matters: with Flash-class experts in DDR5, prefill is PCIe-bound and Q5/Q6 decode is DDR5-bound, so a silently halved link or a BIOS that fell back to 3600 MT/s costs real speed. The R9700 is a two-slot blower (about 267 x 111 x 39-40 mm), so slot spacing is a non-issue on any board with two CPU-wired x16 slots. Ryzen 9000 has 28 PCIe 5.0 lanes with 4 to the chipset, so x8/x8 is the AM5 maximum; boards like the Strix X870E-E have only one CPU x16 slot. The Apex is 2-DIMM (128 GB max), too small for 147-158 GiB Q5/Q6 Flash GGUFs. TRX50 with Threadripper 9000 runs four channels of DDR5-6400 RDIMM (204.8 GB/s theoretical) and x16/x16 Gen5. Estimated 2x R9700 decode ceilings (layer split, no MTP, 262K f16 KV, 8K fill), all from a bandwidth model calibrated to a measured 55 GB/s read on 4x64 GB DDR5-5200 and scaled at the same efficiency, not measured on the other platforms: UD-Q5_K_XL 27 / 34 / 37 / 52 / 63 tok/s and AP-Q4_K_XL 52 / 60 / 64 / 77 / 85 tok/s for AM5 at 3600 / 5200 / 6000, TRX50 and WRX90. Check with lspci -vv under load that both cards show Width x8 at 32GT/s.", "setup": { "gpus": "2x Radeon AI PRO R9700", "model": "Qwen3.8-Flash-Next GGUFs", "engine": "bandwidth model (estimate) + manufacturer specs", "context": "262K" }, "metrics": [ { "label": "Est. UD-Q5 decode, AM5 4x64 GB @3600", "value": 27.2, "unit": "tok/s" }, { "label": "Est. UD-Q5 decode, AM5 4x64 GB @5200 (calibrated)", "value": 34.1, "unit": "tok/s" }, { "label": "Est. UD-Q5 decode, AM5 4x64 GB @6000", "value": 37.0, "unit": "tok/s" }, { "label": "Est. UD-Q5 decode, TRX50 4ch @6400", "value": 51.7, "unit": "tok/s" } ], "sources": [ "https://www.asus.com/us/motherboards-components/motherboards/proart/proart-x870e-creator-wifi/techspec/", "https://dlcdnets.asus.com/pub/ASUS/mb/SocketAM5/ROG_CROSSHAIR_X870E_APEX/E25780_ROG_CROSSHAIR_X870E_APEX_EM_WEB.pdf", "https://rog.asus.com/ea/motherboards/rog-crosshair/rog-crosshair-x870e-dark-hero/spec/", "https://rog.asus.com/us/motherboards/rog-strix/rog-strix-x870e-e-gaming-wifi7-neo/wtb/", "https://www.sapphiretech.com/en/commercial/radeon-ai-pro-r9700", "https://en.wikipedia.org/wiki/Template:AMD_Ryzen_9000_Series", "https://www.tweaktown.com/news/105825/msi-releases-amd-agesa-1-2-0-3e-bios-officially-supports-4-64gb-sticks-up-to-256gb-ram/index.html", "https://www.kingston.com/en/memory/server-memory/memory-population-rules-wrx90-trx50" ], "next": "Verify both cards train at x8 Gen5 under load and DDR5 runs at its set speed on our dual-R9700 machine." }, { "id": "2026-10-24-quality-per-gib-q4plus", "date": "2026-10-24T03:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Qwen3.8-Flash-Next at Q4 or higher: where the extra GiB in Q5 and Q6 actually go", "summary": "We read the tensor headers of seven Flash-Next GGUFs at 4-bit or more without downloading the weights. Most of the 43.7 GiB that UD-Q5_K_XL adds over UD-Q4_K_XL never touches the GPU: 23.9 GiB is the n-gram table moving from IQ4_NL to Q8_0, which stays in system RAM and is read only a few rows per token. Another 10.5 GiB is the expert down projections moving to Q8_0, and only 9.4 GiB is the gate and up experts moving to Q5_K. On agentionai's published held-out test, their AP-Q5_K_M scores the same as UD-Q4_K_XL (mean KL 0.0771 vs 0.0778) with the same number of GPU-side bytes, and every tensor in it is 4-bit or more.", "details": "Bytes that matter for decode (everything except the n-gram table): UD-Q4_K_XL 76.8 GiB, AP-Q5_K_M 76.7, UD-Q4_K_XL with IQ4_NL downs (bitlamas DN4) 68.6, AP-Q4_K_XL 67.3, UD-Q5_K_XL 96.7, UD-Q6_K_XL 106.8. Two R9700s hold 64 GiB, so every option spills some experts to DDR5. On Unsloth's own KL table, Q4 to Q5 cuts mean KL by 35% (0.0469 to 0.0304), and Q5 to Q6 cuts it by only 11% more (0.0271). AP-Q5_K_M keeps its trunk at Q6_K instead of Q8_0, so its always-read dense part is about 0.9 GiB smaller. Our bandwidth model puts its decode ceiling on 2x R9700 at about 50 tok/s, against 45 for UD-Q4_K_XL, 34 for UD-Q5_K_XL and 30 for UD-Q6_K_XL. These are estimates only: layer split, no MTP, no expert cache, 55 GB/s measured DDR5. agentionai warns that this model's n-gram table has memorised wikitext, which pushes top-1 agreement 6.9 to 8.6 points higher than on held-out text. That means wikitext KL figures understate how much a quant hurts. Agentionai's lower tiers (AP-Q4_K_M and below) use IQ3 or IQ2 experts and fall below a 4-bit floor. Quality numbers from different publishers use different references and can't be compared with each other.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4/Q5/Q6_K_XL, AP-Q4_K_XL, AP-Q5_K_M, AP-Q5_K_XL, Q4_K_XL-DN4", "engine": "llama.cpp (GGUF header audit + bandwidth model)", "gpus": "2x Radeon AI PRO R9700 (modeled)", "context": "262K f16 KV (modeled)", "flags": "layer split, n-gram table in system RAM" }, "metrics": [ { "label": "UD-Q5_K_XL extra bytes over UD-Q4_K_XL that are n-gram table", "value": 23.9, "unit": "GiB" }, { "label": "GPU-side bytes, AP-Q5_K_M vs UD-Q4_K_XL", "value": "76.7 vs 76.8", "unit": "GiB" }, { "label": "Held-out mean KL, AP-Q5_K_M vs UD-Q4_K_XL (agentionai)", "value": "0.0771 vs 0.0778", "unit": "" }, { "label": "Estimated 2x R9700 decode ceiling, AP-Q5_K_M / UD-Q4 / UD-Q5 / UD-Q6", "value": "50 / 45 / 34 / 30", "unit": "tok/s (estimate)" } ], "sources": [ "https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF", "https://unsloth.ai/docs/models/qwen3.8-next", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://huggingface.co/bitlamas/Qwen3.8-Flash-Next-Q4_K_XL-DN4" ], "next": "Measure held-out KL and decode speed for AP-Q5_K_M, AP-Q4_K_XL, UD-Q4_K_XL and UD-Q5_K_XL on 2x R9700 with identical flags." }, { "id": "2026-10-24-native-4bit-gate0", "date": "2026-10-24T02:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Which native 4-bit Qwen3.8-Flash-Next packs for 2x R9700 are really 4-bit or more", "summary": "We read the tensor headers of tcclaviger's MXFP4-FP8-GPTQ pack for vLLM without downloading the weights. Every tensor class is 4-bit or more: routed experts 4.25 bits per weight (MXFP4), n-gram table 6.53 (int6), attention FP8, MTP head BF16, 110.6 GiB total. Radiance's stock Flash-Next pack keeps 4-bit codebook experts and an int8 trunk, but its recipe stores the MTP draft head at 2 bits (u2), so it fails a strict no-sub-4-bit rule.", "details": "Per Radiance's docs the 2-bit draft head is only a coarse drafting pass: candidates are re-scored against bf16 weights and the target verifies every token, so output should not change. Turning speculation off doesn't help, because the 2-bit tensor is still in the file. The publisher's 2x R9700 TP2 recipe for the vLLM pack claims 100 tok/s single-request decode and 323 tok/s at 16 concurrent, using 82 GiB of system RAM, with experts that don't fit VRAM held in RAM and the n-gram table streamed from NVMe. Radiance's card implies about 160 tok/s (15 ms per step at about 2.4 accepted tokens). These speeds come from the publishers and we haven't checked them. The quality figures use different references and can't be compared directly: the vLLM pack scores mean KL 0.055 and 95.2% top-1 against BF16, and the Radiance pack 0.079 overall (0.024 on text turns) and 92.9% top-1.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "MXFP4 experts / int6 n-gram / FP8 attention (vLLM); w4 codebook + int8 (Radiance)", "engine": "vLLM (tcclaviger image), Radiance", "engine_version": "HF revisions 6fe3e74 and 8267a67", "gpus": "2x Radeon AI PRO R9700", "context": "262K / 200K", "flags": "header audit only, no inference run" }, "metrics": [ { "label": "vLLM pack routed experts", "value": 4.25, "unit": "bits/weight" }, { "label": "vLLM pack n-gram table", "value": 6.53, "unit": "bits/weight" }, { "label": "vLLM pack total size", "value": 110.6, "unit": "GiB" }, { "label": "Radiance pack MTP draft head", "value": 2, "unit": "bits" } ], "sources": [ "https://huggingface.co/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ", "https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe", "https://hub.docker.com/r/stilldeadcode/vllm-radiance/", "https://codeberg.org/StillDeadcode/radiance" ], "next": "Run the vLLM pack's TP2 recipe on two R9700s and measure decode, prefill and memory against our GGUF Q5/Q6 results." }, { "id": "2026-10-24-llamacpp-hip-rdna4-llvm23-fix", "date": "2026-10-24T01:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "llama.cpp HIP builds for R9700 (gfx1201) on LLVM 23 / new ROCm need b11847 or later", "summary": "Until llama.cpp PR #30885 merged on 2026-10-23 (in release b11847), HIP builds for RDNA4 (gfx1200/gfx1201) and RDNA3 APUs (gfx1150/gfx1151) failed to compile with LLVM 23 and the newest ROCm, because the bf16 WMMA operands were typed __bf16 instead of short. The fix is bit-identical and needs no version gate.", "details": "Found by a new release-radar script that scans llama.cpp, vLLM, ROCm, Lemonade and related repos daily for RDNA4/HIP/ROCm changes. If you build llama.cpp with HIP for an R9700 against a fresh toolchain, update to b11847 or newer first. Prebuilt Vulkan builds are not affected by this compile error.", "setup": { "engine": "llama.cpp (HIP backend)", "engine_version": "b11847", "gpus": "Radeon AI PRO R9700 (gfx1201)" }, "metrics": [], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/30885", "https://github.com/ggml-org/llama.cpp/issues/30839", "https://github.com/ggml-org/llama.cpp/releases/tag/b11847" ], "next": "Use b11847 or later for the dual-R9700 HIP test lane." }, { "id": "2026-10-24-offload-prefill-pcie", "date": "2026-10-24T00:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Why llama.cpp prefill is 10x slower than Strata with Flash experts in DDR5: it's PCIe-bound at the default ubatch", "summary": "In llama.cpp, any prompt chunk of 32+ tokens copies every expert kept in system RAM over PCIe to a GPU, so prefill speed is roughly ubatch size divided by the time to copy those experts. Our measured baseline (Vulkan, Q5_K_M, --n-cpu-moe 28, 205 tok/s at 36K) works out to about 23 GB/s at the default -ub 512, which is 73% of a PCIe 5.0 x8 link: it was PCIe-bound. Estimated ceiling for UD-Q5_K_XL on two R9700s rises from about 230 tok/s at -ub 512 to about 880-960 at -ub 2048 and 1,700-1,840 at -ub 4096, before GPU compute limits.", "details": "From llama.cpp master (commit 8e5a58df87, 2026-10-24): the scheduler offloads big-batch ops on CPU-resident weights to the first GPU that accepts them, so with -sm layer on two cards all expert copies use one card's PCIe link while the other link idles (source reading, not yet measured). Under mmap the loader replaces pinned host buffers with plain CPU memory; PR #28223 (keep explicit host overrides) was closed unmerged and PR #26659 is still open, so on stock master only --load-mode none pins the experts. That costs RAM for the whole file (UD-Q4/Q5/Q6_K_XL are 103.7 / 147.4 / 157.5 GiB, including the 27-51 GiB n-gram table), which fits on a 256 GB DDR5 machine. Published evidence: on an RX 9070 XT (gfx1201, Vulkan) with Qwen3.6-35B-A3B Q4_K_M offloaded, going from -b 1024 -ub 512 to -b 8192 -ub 2048 took 48.5K prefill from 705 to 1,566 tok/s, while -ub 4096 was slower (808); on CUDA, pinned experts took Flash-Next UD-Q6_K_XL prefill from 166 to 330-379 tok/s at 26K with decode flat; on a 3090, gpt-oss-120b went from 380 to 2,091 tok/s between -ub 512 and 8192 with decode down 7%. Short prompts pay the full copy too (about 2 s for ~46 GiB of Q5 experts), and raising GGML_OP_OFFLOAD_MIN_BATCH to 2048 cut one 300-token prompt's time to first token from 29 s to 7.7 s on a box with 135 GB of experts. Our ceilings are estimates from measured GGUF byte counts, using UD-Q5_K_XL expert sizes as a stand-in for Q5_K_M.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (estimates); baseline Q5_K_M (measured)", "engine": "llama.cpp master source + llama.cpp Vulkan baseline", "gpus": "2x Radeon AI PRO R9700 (PCIe 5.0 x8 each)", "context": "262K planned, 36K baseline", "flags": "-sm layer, --n-cpu-moe 24-28, -ub 512/2048/4096" }, "metrics": [ { "label": "Baseline prefill (measured, -ub 512 assumed)", "value": 205, "unit": "tok/s" }, { "label": "Implied PCIe copy speed at baseline", "value": 23.0, "unit": "GB/s" }, { "label": "Q5 ceiling at -ub 2048 (estimate)", "value": "880-960", "unit": "tok/s" }, { "label": "Q5 ceiling at -ub 4096 (estimate)", "value": "1700-1840", "unit": "tok/s" }, { "label": "gfx1201 Vulkan, ub 512 to 2048 (published)", "value": "705 to 1566", "unit": "tok/s" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/issues/25859", "https://inovello.dev/writeups/pinned-host-experts-under-mmap/", "https://github.com/ggml-org/llama.cpp/pull/28223", "https://github.com/ggml-org/llama.cpp/pull/26659", "https://github.com/ggml-org/llama.cpp/pull/19976", "https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend.cpp", "https://huggingface.co/blog/Doctor-Shotgun/llamacpp-moe-offload-guide", "https://github.com/gamozolabs/llama.cpp", "https://bittide.aicompass.dev/article/c9df2d89-8522-46ec-aa3f-fc1aed5b17d3", "https://deepindeepsea.github.io/r9700-vllm-blogs/part12-r9700-the-full-sweep.html" ], "next": "Measure on our two R9700s, Q5 first: -ub 512 vs 2048 vs 4096 with -b 8192, then --load-mode none, RADV's transfer queue, HIP vs Vulkan, a higher offload threshold for short prompts, and whether only one card's link carries the copies." }, { "id": "2026-10-23-hybrid-flash-prompt-cache", "date": "2026-10-23T23:20:00-05:00", "project": "idea-lab", "kind": "result", "title": "Agents on Qwen3.8-Flash-Next: without context checkpoints every turn re-reads the whole prompt", "summary": "Flash-Next is a hybrid DeltaNet model, so llama.cpp's --cache-reuse does nothing for it; context checkpoints are its only prompt cache. Published measurements: --ctx-checkpoints 0 gives a 0% cache hit and 20.5 s per follow-up turn, versus 99.5% and 4.2 s with a single checkpoint. One checkpoint hits as often as the default 32 while using far less RAM.", "details": "Each Flash-Next checkpoint costs 112.6 MiB, plus 2,072 bytes per prefix token when MTP is on (about 555 MiB at 224K). Current llama.cpp flags are -ctxcp/--ctx-checkpoints (default 32), -cms/--checkpoint-min-step (default 8192) and -cram/--cache-ram (default 8192 MiB); --checkpoint-every-n-tokens no longer exists. Hybrid checkpoints can still be invalidated by upstream issue #24055; the fix, PR #25592, was unmerged at last check, and users who cherry-picked it report 35-41 s re-prefills dropping to 0.1-1.3 s. On the client side, anything that changes the start of the prompt (a per-minute timestamp, a mid-session model switch) forces a full re-read; Hermes Agent made its timestamp date-only for this reason. Strata has a separate host-RAM conversation cache: two alternating 57K-token chats went from 9.2 s to 2.1 s prefill on re-entry, at about 1.5 GB of RAM per conversation. These are cited numbers from other rigs; our own dual-R9700 measurement is still to come.", "setup": { "model": "Qwen3.8-Flash-Next", "engine": "llama.cpp server / Strata (published reports)", "flags": "--ctx-checkpoints, --cache-ram, --conversation-cache-mib" }, "metrics": [ { "label": "Follow-up turn, 0 checkpoints", "value": 20.5, "unit": "s" }, { "label": "Follow-up turn, 1 checkpoint", "value": 4.2, "unit": "s" }, { "label": "Cache hit, 1 checkpoint", "value": 99.5, "unit": "%" }, { "label": "Checkpoint size without MTP", "value": 112.6, "unit": "MiB" } ], "sources": [ "https://kksoftwareag.github.io/strix-halo-qwen-3.8-flash-next/memory.html", "https://github.com/ggml-org/llama.cpp/issues/18497", "https://github.com/ggml-org/llama.cpp/pull/25592", "https://particula.tech/blog/prompt-reprocessing-swa-hybrid-models-kv-cache", "https://github.com/NousResearch/hermes-agent/pull/27675", "https://thomaswildetech.com/software-development/ai-agents/strata/" ], "next": "Measure per-turn cache hit and turn latency for Flash on our dual R9700s, Strata versus llama.cpp with and without checkpoint flags." }, { "id": "2026-10-23-3090-gap-q4plus", "date": "2026-10-23T21:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "The fair dual-3090 target for Qwen3.8-Flash-Next at 4-bit and up: 37-41 tok/s", "summary": "Most viral dual-RTX-3090 Flash-Next speeds (111-160 tok/s) are 3-bit or lower, or don't state a quant. At 4-bit and above, the best published 2x 3090 llama.cpp result is 37-41 tok/s (49 on code output) with UD-Q4_K_XL at full 261K f16 KV, using an expert VRAM cache plus MTP. That box started at 17 tok/s without the cache, close to our own 16.9-20.3 llama.cpp baseline, so the gap is software, not hardware.", "details": "Plain banded expert offload at UD-IQ4_XS on 2x 3090 gives 35-38 tok/s. Excluded from the comparison: Strata IQ3_XXS 111 tok/s, tqLLM 77 tok/s at 128K (IQ3_S, one 3090, CUDA only, no multi-GPU yet), EXL3 2.5 bpw 38.6 tok/s, and 150-160 tok/s posts with no quant named. On the 3090 box the cache hit rate depended on slot count, not quant (Q4 and Q6 both 84-85% at 135 slots); Q4 won only by fitting more slots, and -ub 512 freed about 50 slots of VRAM versus 2048. Two R9700s have 64 GB of VRAM against 48 GB; our estimate is about 302 / 236 / 213 cache slots per layer at Q4 / Q5 / Q6 with MTP, against 150 on the 3090 pair (estimate, R9700 compute buffers unmeasured). Reading the merged llama.cpp expert-cache code (PR #29887): every selected expert gets its own real slot, batches up to 32 tokens use the cache, and the draft model's context has it off, so the duplicate-id bug reported on the older draft cache PR (#27861) should not apply to MTP + cache on master. Also from the 3090 write-up: q8_0 KV was slower (-4.8% short, -18% at 131K), MTP acceptance collapsed to ~20% at 131K depth where an n-gram draft won, and DIMM thermal throttling halved decode after a few minutes until a fan was added.", "setup": { "gpus": "2x RTX 3090 (comparison, published)", "engine": "llama.cpp with expert cache + MTP", "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL", "context": "261888, f16 KV", "flags": "experts in host RAM, -ub 512, MTP draft n-max 3" }, "metrics": [ { "label": "Best >=4-bit 2x 3090 llama.cpp decode (thinking prompt)", "value": "37-41", "unit": "tok/s" }, { "label": "Same, code emission", "value": 49, "unit": "tok/s" }, { "label": "Same box before expert cache (UD-Q6_K_XL)", "value": 17, "unit": "tok/s" }, { "label": "2x 3090 banded offload, UD-IQ4_XS", "value": "35-38", "unit": "tok/s" }, { "label": "Our 2x R9700 llama.cpp baseline (Q5_K_M, no cache)", "value": "16.9-20.3", "unit": "tok/s" }, { "label": "Est. R9700 pair cache slots per layer, Q5 + MTP", "value": 236, "unit": "slots (estimate)" }, { "label": "3090 pair cache slots per layer, Q4 + MTP", "value": 150, "unit": "slots" } ], "sources": [ "https://inovello.dev/writeups/qwen3-flash-next-2x3090-q4-mtp/", "https://www.reddit.com/r/LocalLLM/comments/1w069fh/qwen38flashnext_on_2x_rtx_3090s_3343_toks/", "https://x.com/geldeki/status/2095560770163458426", "https://x.com/coffeecup2020/status/2113265219283660816", "https://github.com/r0b0tlab/qwen38-flashnext-exl3", "https://github.com/ggml-org/llama.cpp/pull/29887", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-moe-cache.cpp" ], "next": "Run the same recipe on two R9700s with llama.cpp master (expert cache sized per card, -ub 512, MTP on and off), Q5 first. The target is beating 41 tok/s at full 262K context with Q5." }, { "id": "2026-10-23-dual-r9700-known-traps", "date": "2026-10-23T20:00:00-05:00", "project": "dual-r9700", "kind": "progress", "title": "Dual R9700 multi-GPU traps: what hangs and the fixes that work", "summary": "Cited roundup for two or more R9700s. vLLM tensor parallel (tp=2) deadlocks with both cards at 100% on ROCm 7.2.x RCCL; NCCL_PROTO=Simple is the fix (an AMD engineer notes the proposed RCCL patch is equivalent), while --enforce-eager, NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce do not help. vLLM v0.21.0 adds a shared-memory broadcast hang on big models; v0.20.2 works. llama.cpp's ROCm build can hold an idle card at 100% and about 95 W; GPU_MAX_HW_QUEUES=1 brought it to 22 W.", "details": "Also seen: a Ryzen iGPU visible to ROCm makes llama.cpp split onto it and segfault (hide it or pin the device); an open ROCm 7.14 report shows a llama.cpp generation hang that ends in an MES queue failure and a GPU reset failure until reboot, while Vulkan passes on the same machine; a missing TensileLibrary_lazy_gfx1201.dat means an incomplete rocBLAS install. For context, Level1Techs measured Qwen3.6-35B-A3B FP8 decode at 17.4 tok/s with tp=2 and 15.7 with tp=4 on R9700s (vLLM 0.20.2, NCCL_PROTO=Simple), so more cards did not speed up a small-active MoE at batch 1. Whether ROCm 7.14's newer RCCL avoids the deadlock without the env var is not confirmed, so we keep NCCL_PROTO=Simple pinned. We wrote a read-only pre-flight script that checks a machine for all of these; our own A/B of GPU_MAX_HW_QUEUES on decode speed is still to come.", "setup": { "gpus": "2-4x Radeon AI PRO R9700 (gfx1201), published reports", "engine": "vLLM ROCm images, llama.cpp HIP and Vulkan", "engine_version": "per source" }, "metrics": [ { "label": "Idle draw, llama.cpp HIP, GPU_MAX_HW_QUEUES unset", "value": 95, "unit": "W" }, { "label": "Idle draw, GPU_MAX_HW_QUEUES=1", "value": 22, "unit": "W" }, { "label": "35B-A3B FP8 decode, vLLM tp=2 (Level1Techs)", "value": 17.4, "unit": "tok/s" }, { "label": "35B-A3B FP8 decode, vLLM tp=4 (Level1Techs)", "value": 15.7, "unit": "tok/s" } ], "sources": [ "https://github.com/vllm-project/vllm/issues/40980", "https://github.com/ROCm/rocm-systems/issues/5480", "https://forum.level1techs.com/t/amd-radeon-r9700-linux-tp2-and-tp4-the-definitive-guide/250504", "https://github.com/ROCm/ROCm/issues/5706", "https://github.com/lemonade-sdk/llamacpp-rocm/issues/96", "https://github.com/ROCm/legacy-rocm-build/issues/6630", "https://www.reddit.com/r/LocalLLaMA/comments/1vjtokl/dual_radeon_ai_pro_r9700_server_much_slower_than/" ], "next": "Measure GPU_MAX_HW_QUEUES=1 against unset on our dual-R9700 Flash config for decode speed, and run a small tp=2 vLLM smoke test with NCCL_PROTO=Simple before the vLLM lane." }, { "id": "2026-10-23-flash-decode-roofline", "date": "2026-10-23T19:05:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Flash decode ceiling on dual R9700 + DDR5: about 34 tok/s at Q5 without MTP, so cutting DDR5 reads per token is the main lever", "summary": "A bandwidth-only ceiling (estimate) built from measured GGUF byte counts, the R9700's 640 GB/s spec and our measured 55 GB/s DDR5 read speed puts Qwen3.8 Flash on two R9700s (layer split, experts that don't fit in VRAM kept in DDR5, no MTP) at about 45 / 34 / 30 tok/s for UD-Q4 / Q5 / Q6_K_XL, or 63 / 44 / 37 with ideal tensor split. Our llama.cpp baseline (16.9-20.3 tok/s) runs at 55-66% of its ceiling. Strata's measured 52-65 tok/s at Q5 beats the no-MTP ceiling, so it can be reading at most 0.15-0.34 GiB of experts from DDR5 per generated token instead of 0.86 GiB, which points to MTP plus experts served from VRAM.", "details": "Per token, time = bytes read from VRAM / GPU bandwidth + expert bytes read from DDR5 / DDR5 bandwidth. Dense weights read every token (attention and DeltaNet, lm_head, router, shared expert): about 4.46 GiB at UD-Q5_K_XL; routed experts: 10 of 512 per layer. With experts in DDR5, DDR5 is 57-77% of each token's time at Q5/Q6, so ideal tensor split helps Q4 by about 39% but Q5/Q6 only by 23-27%. Q4 to Q5 costs about 24% of ceiling, Q5 to Q6 about 13%. If every Q5 expert fit in VRAM (four cards), the ceiling would be about 69 tok/s on R9700 vs about 101 on RTX 3090: the 3090's bandwidth edge only shows once DDR5 is out of the loop. Assumptions: GPU decode at 75% of peak bandwidth, 2.5 GiB compute reserve per card, 262K f16 KV read to an 8K fill, tensor split modeled with no all-reduce cost. It ignores CPU compute limits, kernel launch overhead and PCIe, so real numbers land below these ceilings.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "model (bandwidth roofline), checked against measured llama.cpp Vulkan and Strata 0.2.3 runs", "gpus": "1-2x Radeon AI PRO R9700", "context": "262K f16 KV, 8K fill", "flags": "DDR5 4x64 GB at 5200, measured 53.8-56.2 GB/s read" }, "metrics": [ { "label": "Q4 ceiling, 2 GPUs layer split, no MTP (estimate)", "value": 45.1, "unit": "tok/s" }, { "label": "Q5 ceiling, 2 GPUs layer split, no MTP (estimate)", "value": 34.2, "unit": "tok/s" }, { "label": "Q6 ceiling, 2 GPUs layer split, no MTP (estimate)", "value": 29.8, "unit": "tok/s" }, { "label": "Q5 ceiling, 2 GPUs ideal tensor split (estimate)", "value": 43.5, "unit": "tok/s" }, { "label": "llama.cpp baseline vs its ceiling", "value": "55-66", "unit": "%" }, { "label": "Max DDR5 expert reads per token implied by Strata Q5", "value": "0.15-0.34", "unit": "GiB" } ], "sources": [ "https://www.amd.com/en/products/graphics/workstations/radeon-ai-pro/ai-9000-series/amd-radeon-ai-pro-r9700.html", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622" ], "next": "Check the model's slope with a llama.cpp --n-cpu-moe and thread sweep, and run Strata Q5 with MTP on vs off to split MTP gains from expert-cache gains." }, { "id": "2026-10-23-35b-one-card-vram-budget", "date": "2026-10-23T18:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.6-35B-A3B Q5 at full 262K context on one R9700 leaves under 2.75 GB for compute buffers", "summary": "Exact tensor sizes read from the GGUF headers: UD-Q5_K_XL has 22.26 GB of routed experts and 1.99 GB of other GPU weights, and its f16 KV cache at 262,144 tokens is 5.00 GB (10 of 40 layers use full attention). That is 29.25 GB on a 32 GB card, and this config runs fully on one R9700 with nothing offloaded, so llama.cpp's compute buffers fit in at most 2.75 GB. Fit calculators that assume 3 GB or more per card will wrongly say this model needs offload.", "details": "The 0.50 GB token embedding table stays in system RAM (llama.cpp's default for input-layer tensors), so it doesn't count against VRAM. KV size: 10 full-attention layers x 2 KV heads x (256 + 256) dims x 2 bytes x 262,144 tokens = 5.00 GB. GB here means GiB. Also useful for anyone building tools: Hugging Face allows browser CORS Range requests on GGUF files (the resolve URL permits the Range header and its CDN returns 206 with an open CORS header), so a web page can read these headers without downloading weights or running a proxy.", "setup": { "model": "Qwen3.6-35B-A3B", "quant": "Unsloth UD-Q5_K_XL", "engine": "llama.cpp (Vulkan)", "gpus": "1x Radeon AI PRO R9700 (32 GB)", "context": "262144", "flags": "f16 KV, all layers on GPU" }, "metrics": [ { "label": "Routed experts", "value": 22.26, "unit": "GB" }, { "label": "Other GPU weights", "value": 1.99, "unit": "GB" }, { "label": "f16 KV at 262K", "value": 5.0, "unit": "GB" }, { "label": "Max left for compute buffers", "value": 2.75, "unit": "GB" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF" ], "next": "Measure actual VRAM in use on the card to pin the compute-buffer number exactly, and use it as the default in amdai.net's fit calculator." }, { "id": "2026-10-23-measured-flash-q5-vs-q6-strata", "date": "2026-10-23T17:12:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Measured on our dual R9700s: Strata UD-Q5_K_XL runs Qwen3.8 Flash at about 52-65 tok/s, 3x llama.cpp and faster than Q6", "summary": "On the local desktop (two R9700s plus DDR5 expert offload, 256K context), Strata 0.2.3 with Unsloth UD-Q5_K_XL, MTP and int8 KV decoded about 52 tok/s on a short story and about 65 tok/s on code (range 48-68), with 2043-2514 tok/s prefill at 36K. That is about 3x the decode and about 10x the prefill of our llama.cpp Vulkan Q5_K_M baseline (16.9-20.3 tok/s decode, 205 prefill). UD-Q6_K_XL on Strata was slower: median about 40 tok/s warm decode. Q5 stays the sweet spot; Q4 is not yet measured on our machine.", "details": "Baseline: llama.cpp v0.6.0 Vulkan, Q5_K_M, --n-cpu-moe 28, about 23 GB VRAM per card; short-context decode 16.9-20.3 tok/s, at 36K decode 20.0 / prefill 205. Strata 0.2.3.1 (engine 0.1.48 HIP), UD-Q6_K_XL, MTP, int8 KV, 256K: warm decode 31-57 (median ~40); at 36K decode 22.8-51.1 / prefill 1204-1833; at 131K decode 35.0-47.4 / prefill 1242-1864; VRAM 29.8/29.6 GB, RAM 126.9 GB. Strata 0.2.3, UD-Q5_K_XL, MTP, int8 KV, 256K: story ~52, code ~65 (48-68); at 36K decode 43.9-52.4 / prefill 2043-2514; at 131K decode 47.6-54.3 / prefill 2519-2882; VRAM 29.8/29.6 GB, RAM ~112 GB, MTP acceptance 70-75%, swap flat. --tune quick kept the defaults. Q5 also used about 15 GB less system RAM than Q6 (about 112 vs 126.9 GB). Command: strata launch --model ud-q5_k_xl --family unsloth --context 256k --kv int8 --mtp on --gpu 0,1 --parallel 1 --set STRATA_ROUTE_TAIL_SKIP=0", "setup": { "gpus": "2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write)", "engine": "Strata 0.2.3 / 0.2.3.1 (engine 0.1.48 HIP); baseline llama.cpp v0.6.0 Vulkan", "model": "Qwen3.8 Flash", "quant": "UD-Q5_K_XL, UD-Q6_K_XL (baseline Q5_K_M)", "context": "256K (Strata)" }, "metrics": [ { "label": "Strata UD-Q5_K_XL decode, code", "value": 65, "unit": "tok/s" }, { "label": "Strata UD-Q5_K_XL decode, short story", "value": 52, "unit": "tok/s" }, { "label": "Strata UD-Q5_K_XL decode at 131K", "value": "47.6-54.3", "unit": "tok/s" }, { "label": "Strata UD-Q5_K_XL prefill at 131K", "value": "2519-2882", "unit": "tok/s" }, { "label": "Strata UD-Q6_K_XL warm decode (median)", "value": 40, "unit": "tok/s" }, { "label": "llama.cpp Vulkan Q5_K_M baseline decode", "value": "16.9-20.3", "unit": "tok/s" }, { "label": "llama.cpp Vulkan baseline prefill at 36K", "value": 205, "unit": "tok/s" }, { "label": "MTP acceptance (Q5)", "value": "70-75", "unit": "%" } ], "sources": [], "next": "Measure Q4 (UD-Q4_K_XL) on the same Strata setup to confirm the 2-7% Q5 penalty we expect from published numbers." }, { "id": "2026-10-23-measured-rdna4-fork-vs-strata", "date": "2026-10-23T17:11:00-05:00", "project": "dual-r9700", "kind": "result", "title": "RDNA4 llama.cpp fork on dual R9700s: 1.3-1.6x baseline decode with MTP, but half of Strata and 10-15x slower prefill", "summary": "The RDNA4 llama.cpp fork (commit 185252d, ROCm 7.2.0, gfx1201, -sm layer -ts 1,1, q8 KV, flash attention) reached 22-26 tok/s on a story and 32 tok/s on code with MTP (shared Q8_0 head, n-max 3, -t16). That beats our llama.cpp Vulkan baseline by 1.3-1.6x but is about half of Strata UD-Q5_K_XL, and its prefill (144-175 tok/s) is 10-15x slower.", "details": "Thread count matters: without MTP, -t32 gave only 8.9-9.3 tok/s because the CPU was oversubscribed; -t16 gave 20.3-20.5. With MTP at -t16: story 22-26, code 32, VRAM 29.9/28.2 GB. At 140K context: 36K decode 22.8 / prefill 175; 131K decode 16.2 / prefill 144. Strata at the same 131K depth: decode 47.6-54.3 / prefill 2519-2882.", "setup": { "gpus": "2x Radeon AI PRO R9700 (PCIe Gen5 x8 each, ReBAR 32 GiB), DDR5 4x64 GB at 5200 (53.8-56.2 GB/s read, ~27 GB/s write)", "engine": "RDNA4 llama.cpp fork, commit 185252d, ROCm 7.2.0 (gfx1201)", "model": "Qwen3.8 Flash", "flags": "-sm layer -ts 1,1, q8 KV, FA, MTP shared Q8_0 head n-max 3, -t16", "context": "140K" }, "metrics": [ { "label": "Fork + MTP decode, code", "value": 32, "unit": "tok/s" }, { "label": "Fork + MTP decode, story", "value": "22-26", "unit": "tok/s" }, { "label": "Fork no MTP, -t16", "value": "20.3-20.5", "unit": "tok/s" }, { "label": "Fork no MTP, -t32 (oversubscribed)", "value": "8.9-9.3", "unit": "tok/s" }, { "label": "Fork decode / prefill at 131K", "value": "16.2 / 144", "unit": "tok/s" } ], "sources": [], "next": "Use -t16 (physical cores minus headroom), not all threads, for hybrid CPU-expert runs." }, { "id": "2026-10-23-hsa-override-segfault", "date": "2026-10-23T17:10:00-05:00", "project": "dual-r9700", "kind": "failure", "title": "A system-wide HSA_OVERRIDE_GFX_VERSION=12.0.0 segfaults llama.cpp in op_repeat on R9700; set 12.0.1 per process", "summary": "With HSA_OVERRIDE_GFX_VERSION=12.0.0 set system-wide, the RDNA4 llama.cpp fork (ROCm 7.2.0) segfaulted in op_repeat on our dual R9700s. Setting HSA_OVERRIDE_GFX_VERSION=12.0.1 (matching the R9700's gfx1201) for just the llama.cpp process fixed it.", "details": "The R9700 is gfx1201, so the 12.0.0 value does not match the card. Remove any global override (for example in /etc/environment or a shell profile) and pass HSA_OVERRIDE_GFX_VERSION=12.0.1 only to the process that needs it, so other ROCm apps are not affected.", "setup": { "gpus": "2x Radeon AI PRO R9700 (gfx1201)", "engine": "RDNA4 llama.cpp fork, commit 185252d, ROCm 7.2.0" }, "metrics": [], "sources": [], "next": "Fixed; no further action." }, { "id": "2026-10-23-r9700-card-tuning-decode", "date": "2026-10-23T17:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "R9700 card tuning for decode: power caps barely matter, memory overclock backfires", "summary": "From published R9700 measurements: decode is memory-bound and draws about 180-190 W under a 300 W cap, so raising the cap doesn't add tok/s. A -120 mV undervolt gave +7.7-8.9% prefill but only +1.3% decode. On amdgpu DKMS 6.19.4 any memory overclock above the 1258 MHz top step is accepted, then pins memory lower and cost 6.6% decode in a clean A/B.", "details": "Undervolt pays only when the card is actually power-limited (sustained prefill near 300 W). -150 mV was stable at 300 W but crashed with a page fault at 330 W on a 32K prompt after a 2K prompt passed, so short benchmarks don't prove stability. Verify memory clocks with pp_dpm_mclk, not OD_MCLK, which reads back the requested value even when it is ignored. The common 210 W cap with about -75 mV is reported as ~30% lower peak power for a few percent speed loss (community self-reports). The same pattern shows on datacenter GPUs: on H200, decode never hit a power cap and clock locking saved up to 32% decode energy at minimal throughput loss. PCIe ASPM performance mode was reported +10.8% decode on a dense 27B and ~0% on MoE. Not yet measured on our dual-R9700 hybrid-offload setup.", "setup": { "gpus": "Radeon AI PRO R9700 (published single-card tests)", "engine": "llama.cpp (published)", "model": "Qwen3.6-27B Q4_K_M (undervolt and memory-clock tests)" }, "metrics": [ { "label": "Decode power draw under 300 W cap", "value": "180-190", "unit": "W" }, { "label": "-120 mV undervolt, prefill pp32768", "value": 8.9, "unit": "%" }, { "label": "-120 mV undervolt, decode tg32", "value": 1.3, "unit": "%" }, { "label": "Memory OC above 1258 MHz, decode tg32", "value": -6.6, "unit": "%" }, { "label": "H200 decode energy saved by clock locking (up to)", "value": 32, "unit": "%" } ], "sources": [ "https://github.com/daimonionnn/amd-r9700-vllm-and-tuning-toolkit/blob/main/docs/r9700-oc-uv-findings.md", "https://www.reddit.com/r/ROCm/comments/1vuu58x/", "https://www.reddit.com/r/ROCm/comments/1vr5j5e/", "https://github.com/ggml-org/llama.cpp/discussions/21043", "https://arxiv.org/abs/2605.11999" ], "next": "A/B test power caps, a locked memory clock, COMPUTE profile and PCIe ASPM on dual R9700s with Flash experts in DDR5, where idle gaps between expert steps may make clock ramp-up matter more." }, { "id": "2026-10-23-x-community-4bit-native", "date": "2026-10-23T16:20:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Community reports: the fastest 4-bit Flash numbers on R9700s come from native engines, not GGUF", "summary": "A scan of public R9700 posts on X (378 in the last 7 days) found the biggest Qwen3.8-Flash-Next numbers at 4-bit or above come from non-GGUF engines: about 228 tok/s single stream on two R9700s with Radiance's 4-bit experts, and about 200 tok/s on four R9700s with vLLM + r9700-stack MXFP4, versus 67 on llama.cpp on the same four cards. All figures are self-reported by their posters and unverified here.", "details": "Headline dual-R9700 Flash results of 216-272 tok/s (vLLM + Paiton W3A8) and 130 tok/s (SGLang, EXL3 3.05 bpw) are 3-bit, so they are excluded from our comparisons, which only cover Q4 and up. Other useful reports: splitting a model that fits on one card across two R9700s cost about 7% decode in llama.cpp (75.4 to 70.3 tok/s, Qwen3.6-35B-A3B Q4_K_M), and a chipset PCIe 4.0 x4 slot cost about 12% (66.2 tok/s). For reference, our own measurement on one R9700 plus DDR5 is 32.6 tok/s for Qwen3.8 Flash UD-Q6_K_XL on Strata versus 16 on llama.cpp. We plan to add a 4-bit native lane (vLLM MXFP4 at TP2 and Radiance W4) to the dual-R9700 sweep after confirming every expert tensor is at least 4-bit.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "W4 / MXFP4-FP8 (4-bit class)", "engine": "Radiance 1.4.0; vLLM + r9700-stack (as reported)", "gpus": "2x and 4x Radeon AI PRO R9700 (as reported)", "context": "up to 256K (as reported)" }, "metrics": [ { "label": "Radiance W4, 2x R9700, single stream (reported)", "value": 228, "unit": "tok/s" }, { "label": "Radiance W4, 2x R9700, 16 concurrent (reported)", "value": 945, "unit": "tok/s" }, { "label": "vLLM + r9700-stack MXFP4 TP4, 4x R9700, single stream (reported)", "value": 200, "unit": "tok/s" }, { "label": "llama.cpp, same 4x R9700 (reported)", "value": 67, "unit": "tok/s" }, { "label": "Public R9700 posts on X, last 7 days", "value": 378, "unit": "posts" } ], "sources": [ "https://x.com/ECander84852/status/2110182801639408108", "https://x.com/dad__vibes/status/2111137765396210053", "https://x.com/dad__vibes/status/2111573866996977673", "https://x.com/dad__vibes/status/2111908353802998061", "https://x.com/sincoice/status/2113225429679210835" ], "next": "Confirm Radiance and r9700-stack expert formats are at least 4-bit, then measure both at TP2 against the best GGUF Q5 result." }, { "id": "2026-10-23-llamacpp-moe-expert-cache", "date": "2026-10-23T15:06:00-05:00", "project": "dual-r9700", "kind": "result", "title": "llama.cpp now caches offloaded MoE experts in VRAM, and Flash can use tensor split", "summary": "llama.cpp master added --moe-cache-mib (merged Oct 7, 2026; per-GPU multi-GPU support Oct 8), an LRU cache in VRAM for experts kept in system RAM. On Qwen3.8-Flash-Next Q4_0 with 2x RTX 4090, the author measured decode going from 34.78 to 64.39 tok/s (1.85x, 95.8% cache hits), with prompt speed at 0.73x in that all-cache setup and 0.91x with a smaller cache. A single-R9700 test on Windows showed the cache only paying off near 94% hits. The current source also allows --split-mode tensor for the Flash architecture (qwen4exp) and Qwen3.5/3.6 MoE files, so 'tensor mode does not work on MoE' is too broad: it is blocked for specific architectures such as DeepSeek2, GLM-DSA and MiniMax.", "details": "Estimate, not measured: two R9700s at 262K context with f16 KV leave about 46.8 GiB for experts (assuming 3 GiB per card for compute buffers). An offline LRU trace study posted on the PR suggests that cached share would give roughly 97.6% / 95.2% / 94.1% hits for UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL, all in the range where the cache paid off on an R9700. On ROCm, tensor mode's all-reduce is host-staged; a community direct-P2P patch measured +7.5% to +11.2% decode on 2x R9700 with dense Q8_0 models. Linux R9700 pairs report about 28 GB/s GPU-to-GPU and 21.5 GB/s CPU-to-GPU copy bandwidth.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL (planned)", "engine": "llama.cpp master (HIP)", "gpus": "2x Radeon AI PRO R9700 (planned target)" }, "metrics": [ { "label": "Cited: 2x RTX 4090 decode, master --fit", "value": 34.78, "unit": "tok/s" }, { "label": "Cited: 2x RTX 4090 decode, -cmoe + 15000 MiB cache", "value": 64.39, "unit": "tok/s" }, { "label": "Cited: cache hit rate at that setting", "value": 95.8, "unit": "%" }, { "label": "Estimated expert VRAM budget, 2x R9700 at 262K f16", "value": 46.8, "unit": "GiB" } ], "sources": [ "https://github.com/ggml-org/llama.cpp/pull/29887", "https://github.com/ggml-org/llama.cpp/pull/30112", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp", "https://github.com/ggml-org/llama.cpp/pull/25028", "https://github.com/JohnTDI-cpu/llama-hip-p2p-allreduce", "https://forum.level1techs.com/t/does-the-lrlink-pcie-bridge-fix-rccls-broken-p2p-on-dual-r9700-update-yes-it-does/249623" ], "next": "Measure UD-Q5_K_XL at 262K on two R9700s: the current -ncmoe baseline vs --fit with a partial cache vs all experts cached, with real hit rates, then Q4 and Q6 and a tensor-split check." }, { "id": "2026-10-23-ngram-table-vram-budget", "date": "2026-10-23T14:05:00-05:00", "project": "idea-lab", "kind": "result", "title": "Qwen3.8-Flash-Next: the n-gram table is 27-51 GiB and belongs in DDR5", "summary": "Reading the GGUF headers shows the model's n-gram embedding table (per_layer_token_embd) is a single lookup tensor of 26.85 GiB at UD-Q4_K_XL (IQ4_NL) and 50.70 GiB at both UD-Q5_K_XL and UD-Q6_K_XL (Q8_0). Routed experts are 71.73, 91.61 and 101.75 GiB. Since only a few table rows are read per token, the VRAM question on two R9700s (64 GiB) is really only about experts. Q5 and Q6 differ by just ~10 GiB of experts.", "details": "llama.cpp master tags per_layer_token_embd as an input-layer get-rows tensor, so it stays on the CPU like token_embd. Where Strata puts it is not yet checked. Estimated -ncmoe starting points at 262K context with f16 KV: 17 for Q4, 24 for Q5, 26 for Q6 (assumes 3 GiB compute reserve per GPU and KV from the model card's 12 sparse-attention layers, about 6.75 GiB at 262K). These start points are estimates until measured.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "GGUF header analysis (no inference)", "gpus": "2x Radeon AI PRO R9700 (planned target)" }, "metrics": [ { "label": "N-gram table, UD-Q4_K_XL", "value": 26.85, "unit": "GiB" }, { "label": "N-gram table, UD-Q5_K_XL / Q6_K_XL", "value": 50.7, "unit": "GiB" }, { "label": "Routed experts, UD-Q4_K_XL", "value": 71.73, "unit": "GiB" }, { "label": "Routed experts, UD-Q5_K_XL", "value": 91.61, "unit": "GiB" }, { "label": "Routed experts, UD-Q6_K_XL", "value": 101.75, "unit": "GiB" } ], "sources": [ "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.cpp" ], "next": "Check n-gram table placement in Strata and run an -ncmoe sweep around the estimated start points on the local desktop." }, { "id": "2026-10-23-ddr5-offload-failure", "date": "2026-10-23T13:57:00-05:00", "project": "dual-r9700", "kind": "failure", "title": "Earlier dual ROCm + DDR5 offload run judged not viable", "summary": "On the local desktop, Qwen Flash UD-Q6_K_XL on llama.cpp with DDR5 expert offload ran at about 16 tok/s, versus about 32.6 tok/s on Strata v0.1.42 at 262K context. Back then, dual ROCm plus DDR5 offload in llama.cpp was judged not viable.", "details": "Likely causes from research: Q6 instead of Q5, no VRAM expert cache or MTP in stock llama.cpp for HIP, and an older Strata version (newer releases report 1.34-1.40x decode from keeping every expert in RAM).", "setup": { "model": "Qwen3.8-Flash-Next (Qwen Flash)", "quant": "UD-Q6_K_XL", "engine": "llama.cpp (ROCm) and Strata", "engine_version": "Strata v0.1.42", "gpus": "2x Radeon AI PRO R9700", "context": "262K", "flags": "DDR5 expert offload" }, "metrics": [ { "label": "llama.cpp decode", "value": 16, "unit": "tok/s" }, { "label": "Strata v0.1.42 decode", "value": 32.6, "unit": "tok/s" } ], "sources": [ "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md" ], "next": "Retest with UD-Q5_K_XL, Strata 0.2.3 and the RDNA4 llama.cpp fork with MTP." }, { "id": "2026-10-23-3090-vs-r9700", "date": "2026-10-23T13:56:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Cited comparison: dual 3090 vs dual R9700 at Q4", "summary": "Same model and quant (Qwen3.8-Flash-Next UD-Q4_K_XL): dual RTX 3090 on the llama.cpp moe-cache fork posts 39.28 tok/s decode; dual R9700 on Strata posts 64 tok/s (different host rigs). The gap on our rig is software and version, not hardware. A 3090 has more memory bandwidth (936 vs 640 GB/s), yet R9700s win when the engine keeps experts in RAM well.", "details": "Stock llama.cpp on one R9700 with --n-cpu-moe manages only 18-20 tok/s at Q4 on the same model (published). llama.cpp lacks a VRAM expert cache on HIP; the moe-cache work is CUDA-only and unmerged.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL", "engine": "llamacpp-club3090 moe-cache (3090) vs Strata (R9700)", "engine_version": "per sources", "gpus": "2x RTX 3090 vs 2x Radeon AI PRO R9700", "context": "204,800 (3090) / 40K (R9700)", "flags": "per sources" }, "metrics": [ { "label": "2x RTX 3090 decode", "value": 39.28, "unit": "tok/s" }, { "label": "2x R9700 decode (Strata)", "value": 64, "unit": "tok/s" }, { "label": "1x R9700 decode (stock llama.cpp)", "value": "18-20", "unit": "tok/s" } ], "sources": [ "https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md", "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md", "https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622" ], "next": "Reproduce on the local desktop with Strata 0.2.3 and the RDNA4 Flash-Next llama.cpp fork." }, { "id": "2026-10-23-sweet-spot-q5", "date": "2026-10-23T13:55:00-05:00", "project": "dual-r9700", "kind": "result", "title": "Sweet-spot quant for Flash models on dual R9700: UD-Q5_K_XL", "summary": "From published R9700 measurements: Q5_K_XL is almost as good as Q6_K_XL (mean KL vs Q8_0 0.038 vs 0.033, same top token 94.55% vs 94.61%) and only 2-7% slower than Q4_K_XL on two R9700s (60 vs 64 tok/s on short story, 94 vs 96 on code). Q4 is the speed fallback; Q6 is dominated.", "details": "Qwen3.8-Flash-Next file sizes: UD-Q4_K_XL 103.7 GiB, UD-Q5_K_XL 147.4 GiB, UD-Q6_K_XL 157.5 GiB. All fit in 64 GiB VRAM plus DDR5 with experts spilled to RAM. GLM-5.3-Flash Q6_K_XL (271.8 GiB) is too large for comfort, so Q5 is its ceiling. DeepSeek-V4-Flash has no Q5/Q6 GGUF; UD-Q4_K_XL is the pick. Published numbers come from an EPYC 8-channel DDR4 host, not this rig.", "setup": { "model": "Qwen3.8-Flash-Next", "quant": "UD-Q4_K_XL / UD-Q5_K_XL / UD-Q6_K_XL", "engine": "Strata (published)", "engine_version": "per source", "gpus": "2x Radeon AI PRO R9700", "context": "40K", "flags": "layer split, 8-bit KV, MTP, every expert in RAM" }, "metrics": [ { "label": "Q5_K_XL decode, 2x R9700 (story)", "value": 60, "unit": "tok/s" }, { "label": "Q4_K_XL decode, 2x R9700 (story)", "value": 64, "unit": "tok/s" }, { "label": "Q5_K_XL mean KL vs Q8_0", "value": 0.038, "unit": "" }, { "label": "Q6_K_XL mean KL vs Q8_0", "value": 0.033, "unit": "" } ], "sources": [ "https://github.com/Niko1221/Strata/blob/main/docs/QUANTS.md", "https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF", "https://unsloth.ai/docs/models/qwen3.8-next" ], "next": "Run the ordered Q5 benchmark sequence on the local desktop, with Q4 and Q6 as secondary runs." } ] }