https://amdai.net/lab-log/ Public AMDai results log of measured and cited findings, plus informative failures. Dual R9700 speed and the hourly Idea lab. No invented tok/s. en-US https://amdai.net/lab-log/#2026-10-28-cpu-cooler-for-dual-r9700 2026-10-28-cpu-cooler-for-dual-r9700 Wed, 28 Oct 2026 12:00:00 GMT AMD ships the Ryzen 5 9600X, Ryzen 9 9900X and Ryzen 9 9950X without a cooler; AMD recommends a premium air cooler for the 65 W 9600X and a liquid cooler for the 120 W 9900X and 170 W 9950X for best boost. for a local-AI box with blower R9700s, where the radiator goes matters: the cards pull their air from the case's front and bottom intake, so a 360 radiator should go on top as exhaust, and a case that only takes one in front or bottom is better paired with a large dual-tower air cooler. https://amdai.net/lab-log/#2026-10-28-case-for-two-r9700s 2026-10-28-case-for-two-r9700s Wed, 28 Oct 2026 12:00:00 GMT Every R9700 board-partner card is a 2-slot blower no longer than 271 mm. on the ASRock X870 Taichi Creator and Gigabyte B850 AI TOP, owners report 4-slot spacing between the two CPU x8/x8 slots, so the bottom card ends by expansion slot 7 and leaves a 2-slot gap above it. that means any 7-slot ATX case with at least 271 mm of card length fits two cards; no 8-slot case is needed unless you plan a third card or a riser. blowers push their heat out the back, so the case's job is fresh intake air for the card area. https://amdai.net/lab-log/#2026-10-28-nvme-for-local-models 2026-10-28-nvme-for-local-models Wed, 28 Oct 2026 12:00:00 GMT Every Q4 to Q6 quant of Qwen3.6-35B-A3B (20.6 to 29.7 GiB), Qwen3.8-Flash-Next (103.7 to 157.6 GiB) and GLM-5.3-Flash (186.0 to 271.8 GiB) adds up to about 1,587 GiB once you keep one extra file of room for the next download and 150 GiB for the OS. That fits a 2 TB drive (1,863 GiB) with 10% left free; a fourth Flash-class family such as MiniMax-M3.1-Flash (UD-Q4 232 GiB) needs 4 TB. On llama.cpp's default memory-mapped loader, loading is usually limited by page faults and one CPU core rather than the drive, so a Gen5 SSD helps less than its doubled rating suggests. https://amdai.net/lab-log/#2026-10-27-sglang-tp2-vs-llamacpp-r9700 2026-10-27-sglang-tp2-vs-llamacpp-r9700 Tue, 27 Oct 2026 12:00:00 GMT The most complete public 2x R9700 SGLang rig (v0.5.15 plus RDNA4 patches, TP=2) measured Qwen3.6-35B-A3B AWQ int4 at 67.0 tok/s short and 22.0 tok/s at 197K input. llama.cpp on a single R9700 does 165.2 tok/s (Vulkan, Q4_K_M, tg128) and our own Q5 run does 100.8 short and about 50 at 256K. Upstream SGLang still blocks gfx1201 builds, and its only CPU-expert path (KTransformers) is built around Intel AMX, so Flash-class models stay on llama.cpp or Strata. https://amdai.net/lab-log/#2026-10-27-rocm-smi-gone-in-rocm-10 2026-10-27-rocm-smi-gone-in-rocm-10 Tue, 27 Oct 2026 12:00:00 GMT AMD's ROCm 10.1 transition guide lists ROCm SMI as removed from the Core SDK (replaced by AMD SMI, package amdrocm-amdsmi); it was dropped from TheRock builds in 10.0. On a fresh 10.x install, 'rocm-smi' prints command not found even though ROCm works. 'amd-smi --rocm-smi' prints the familiar concise table, and 'amd-smi monitor --gfx --vram-usage --watch 1' replaces 'watch -n1 rocm-smi'. Our own scan of amdai.net found 23 command lines across 15 pages still calling rocm-smi with no fallback, including the verify tutorial's pass/fail checklist; we are fixing those. https://amdai.net/lab-log/#2026-10-27-kv-estimate-audit-llamacpp 2026-10-27-kv-estimate-audit-llamacpp Tue, 27 Oct 2026 12:00:00 GMT We checked the KV estimate for all 19 model presets in the amdai.net calculator against the model configs and llama.cpp master's memory code (Oct 27). 13 match within 10%. GLM-5.3-Flash was the big miss: only 11 of its 45 layers keep a growing cache (34 are linear-attention layers with a fixed state), so 262K context needs about 4.95 GiB at f16, not the 11.25 GiB we showed. Qwen3.8-Flash-Next needs about 7.6 GiB at 262K, not 6.0, because its sparse-attention indexer keys also live in the cache. Gemma 4's sliding-window layers were under-counted by 0.4-0.8 GiB. https://amdai.net/lab-log/#2026-10-27-r9700-street-price-noon 2026-10-27-r9700-street-price-noon Tue, 27 Oct 2026 12:00:00 GMT As of Oct 27 at noon CT, every new in-stock Radeon AI PRO R9700 32 GB we found sold by Micro Center costs $1,949.99 to $1,999.99, and Newegg and B&H list $1,999.99 (ASUS Turbo). That is up $250-300 from yesterday's $1,699.99-$1,799.99. The cheapest is the Gigabyte AI TOP at $1,949.99 (Micro Center, in-store pickup only), 50% above AMD's $1,299 MSRP. Two cards (64 GB VRAM) now cost $3,899.98 before tax. https://amdai.net/lab-log/#2026-10-27-llama-cpp-removed-load-flags 2026-10-27-llama-cpp-removed-load-flags Tue, 27 Oct 2026 12:00:00 GMT Since release b10875 (2026-09-09, PR #28334) llama-server and llama-cli exit with 'invalid argument' on --no-mmap, --mmap, --mlock and --direct-io, and llama-bench rejects -mmp and -dio. Use --load-mode none (old --no-mmap), mmap+mlock (old --mlock), mlock (old --no-mmap --mlock) or dio; in llama-bench, -lm none. The env vars LLAMA_ARG_NO_MMAP, LLAMA_ARG_MMAP and LLAMA_ARG_MLOCK are no longer read at all, so Docker or systemd setups that used them quietly fall back to the default load mode (auto, which memory-maps). https://amdai.net/lab-log/#2026-10-27-vllm-int4-r9700-gating 2026-10-27-vllm-int4-r9700-gating Tue, 27 Oct 2026 12:00:00 GMT Reading vLLM main (e7fa26d5c8, 2026-10-27): the Oct 13 W4A16 decode tuning (+9-21% GEMM, up to -66% on speculative shapes) is gated to Strix Halo (gfx115x); the new int4 MoE decode kernel (+44% on a 35B-A3B, Strix Halo) and MXFP4 W4A8 GEMV are gfx11-only; upstream has no tuned int4 MoE configs for the R9700. On R9700, GPTQ checkpoints crash at startup unless you pass --linear-backend TRITON, and the default ROCM_ATTN backend gave wrong outputs or GPU faults outside head 128 / KV block 16, while TRITON_ATTN was clean. https://amdai.net/lab-log/#2026-10-27-llama-server-metrics-gauges 2026-10-27-llama-server-metrics-gauges Tue, 27 Oct 2026 12:00:00 GMT In current llama.cpp, the /metrics gauges predicted_tokens_seconds and prompt_tokens_seconds are reset by every scrape and only updated when a request finishes, so dashboards show 0 while a long generation runs and two scrapers steal each other's windows. Rates from the cumulative counters are reliable. Strata's /metrics already reports a live 2-second tok/s plus per-request speeds. https://amdai.net/lab-log/#2026-10-27-vram-per-dollar 2026-10-27-vram-per-dollar Tue, 27 Oct 2026 12:00:00 GMT With the cheapest new R9700 at $1,949.99 (up $250 since yesterday) it costs $60.9 per GB of VRAM, against $58.9-63.7 per GB for a used RTX 3090 at eBay sold medians of $1,413-$1,528. Per watt it wins clearly (9.4 W/GB vs 14.6), and 64 GB takes two R9700s at 600 W instead of three 3090s at 1,050 W. https://amdai.net/lab-log/#2026-10-27-gfx1201-upstream-bugs 2026-10-27-gfx1201-upstream-bugs Tue, 27 Oct 2026 12:00:00 GMT We searched 10 GitHub repos Radeon local-AI users rely on for issues naming gfx1201 or R9700. 15 RDNA4-titled issues were closed as fixed in the last 30 days, including two Tensile GEMM page faults and a ~12% Vulkan prefill regression in llama.cpp. Still open: a ROCm-only llama.cpp generation hang that resets the GPU (Vulkan passes), a cross-stream wait penalty whose size depends on MES firmware, a GFXOFF-related idle power cost, a vLLM TP=2 deadlock, and a vLLM 0.28 FP8 kernel choice that slows RDNA4 decode. https://amdai.net/lab-log/#2026-10-27-new-model-licenses 2026-10-27-new-model-licenses Tue, 27 Oct 2026 12:00:00 GMT Of 88 open-weight models released in the last 90 days, 20 were tagged only 'other' or nothing on Hugging Face. After reading each LICENSE: 62 open, 7 open with conditions, 5 commercial only under a revenue cap, 9 non-commercial or research only, 1 not for EU use, 4 still unclear. Personal and home use is fine for every Flash-class model we run. https://amdai.net/lab-log/#2026-10-27-coding-cli-prompt-tax 2026-10-27-coding-cli-prompt-tax Tue, 27 Oct 2026 12:00:00 GMT We captured what five coding CLIs send to a local endpoint. First-turn prompts: Claude Code 19,384 tokens, Qwen Code 15,091 (plus a 7,084-token memory call every turn), Codex CLI 8,343, OpenCode 7,158, Aider 749. Replayed on a real one-slot llama-server, Qwen Code's memory call evicted the session when the RAM prompt cache was off: turn 2 re-read 15,602 tokens instead of 21. https://amdai.net/lab-log/#2026-10-26-hip-build-flags-rccl 2026-10-26-hip-build-flags-rccl Mon, 26 Oct 2026 12:00:00 GMT Since llama.cpp PR #27825 (merged 2026-09-15), a HIP build with GGML_HIP_RCCL=OFF (the default) uses a built-in two-GPU AllReduce for --split-mode tensor. On 2x R9700 it beat the old fallback by +17.5% prefill and +8.3% decode, and in published RCCL-on vs off tests decode was equal or slightly faster without RCCL while RCCL only helped long prefills. https://amdai.net/lab-log/#2026-10-26-ddr5-dimm-heat-hybrid-decode 2026-10-26-ddr5-dimm-heat-hybrid-decode Mon, 26 Oct 2026 12:00:00 GMT When a big MoE model keeps its experts in DDR5 next to two R9700s, the DIMMs work hard for hours. DDR5 doubles its refresh rate above 85 C (JEDEC refresh interval 3.9 us drops to 1.95 us). For 64 GB sticks built from 32 Gb chips, that takes the time a rank spends refreshing from about 10.5% to about 21%, so bandwidth-bound work can slow by up to ~13%. For hybrid decode we estimate up to ~7-11% fewer tokens per second once sticks pass 85 C, depending on how much of each token's time is spent reading DDR5 (an upper-bound estimate). Linux can read each stick's temperature with the spd5118 driver, so this is easy to check. https://amdai.net/lab-log/#2026-10-26-four-r9700-two-desktops-power-budget 2026-10-26-four-r9700-two-desktops-power-budget Mon, 26 Oct 2026 12:00:00 GMT Before buying a second desktop for two more R9700s, we built a power budget from published measurements. Under full compute an R9700 has been measured at about 390-400 W at the card (spec TBP 300 W), while decode-bound llama.cpp is much lower: about 340-353 W for two cards together in layer split. With all four cards at 300 W, the two desktops draw an estimated ~1,780 W at the wall; even at the lowest allowed cap (210 W) it is ~1,640 W. A 15 A, 120 V circuit may carry only 1,440 W continuously (the 80% rule for loads over 3 hours), so the two boxes need separate circuits, or one 20 A circuit with cards capped at 260 W or less. https://amdai.net/lab-log/#2026-10-26-llama-bench-noise-aa-test 2026-10-26-llama-bench-noise-aa-test Mon, 26 Oct 2026 12:00:00 GMT Before trusting a 3-5% win between two configs, we ran the same llama-bench command 6 times in a row. Inside one run, repetitions varied about 5%, but the medians of whole runs varied 6-7%, and the first run after loading was 12-16% slower than the rest. Comparing any two of the identical runs, a 95% interval built from one run's repetitions wrongly flagged a difference in 9 of 15 pairs. llama-bench's own tools don't guard against this: the generation warmup is a single token, repetitions run back-to-back by default, and compare-llama-bench.py prints a plain ratio of averages with no spread. https://amdai.net/lab-log/#2026-10-26-openwebui-background-tasks-prompt-cache 2026-10-26-openwebui-background-tasks-prompt-cache Mon, 26 Oct 2026 12:00:00 GMT Open WebUI 0.12.4 sends extra model calls per chat turn (follow-up suggestions every turn, plus title and tags on the first), each a fresh one-message prompt that shares no prefix with the conversation. On a one-slot llama-server they replace the slot's state. With the default host prompt cache (--cache-ram 8192 MiB) the conversation is saved and restored and each later turn re-processes only about 70 tokens; with --cache-ram 0, or a cache smaller than one conversation's saved state, every turn re-processed the whole 11K-token chat (about 40x the prefill time in this test). https://amdai.net/lab-log/#2026-10-26-glm-flash-chat-template-render 2026-10-26-glm-flash-chat-template-render Mon, 26 Oct 2026 12:00:00 GMT The chat template embedded in the current Unsloth GLM-5.3-Flash GGUF matches the upstream 2026-08-27 version and misses the 2026-10-15 upstream rewrite of tool-result reordering. Rendered with llama.cpp's own Jinja engine, one block of 100 parallel tool results takes 1.14 s with the embedded template vs 31 ms with the new one (400 results: 17.0 s vs 0.19 s), with byte-identical output. Fix without re-downloading: pass the upstream template with --chat-template-file. https://amdai.net/lab-log/#2026-10-26-r9700-street-price 2026-10-26-r9700-street-price Mon, 26 Oct 2026 12:00:00 GMT Across the six US R9700 32 GB models, the cheapest new card in stock is $1,699.99 (Gigabyte AI TOP and XFX at Micro Center, in-store pickup), 31% above AMD's $1,299 MSRP. The lowest price seen since Sep 29 is $1,599.99. Amazon's in-stock new offers run $2,399.99 to $2,680.81. https://amdai.net/lab-log/#2026-10-26-strata-mtp-q2-only-second-card 2026-10-26-strata-mtp-q2-only-second-card Mon, 26 Oct 2026 12:00:00 GMT In Strata main (2026-10-25), the runtime drafter sizes its 512 experts as kernels::cpu::BLOB, which is the Q2_0 layout (1,382,400 bytes per expert, 707,788,800 for the file). mtp_pack.py can write Q4_0 or Q8_0 draft experts, but nothing loads them (open issue #3728), and --mtp-q4 only requantizes the dense head, not the experts. Anyone who keeps every file at 4-bit or more therefore runs Strata with no draft layer. On a layer split that also turns off pipeline windows, the RDNA4 default that gave +21% code decode on 2x R9700 (103 to 125 tok/s) in the maintainer's table. https://amdai.net/lab-log/#2026-10-26-vllm-tp2-allreduce-rccl-only 2026-10-26-vllm-tp2-allreduce-rccl-only Mon, 26 Oct 2026 12:00:00 GMT In current vLLM source, the custom all-reduce, QuickReduce and AITER all-reduce paths are enabled only on MI300-class GPUs; on gfx1201 vLLM forces disable_custom_all_reduce on, so every tensor-parallel all-reduce is an RCCL call (this is why --disable-custom-all-reduce 'has no effect' on R9700). With two all-reduces per layer, Qwen3.6-35B-A3B makes 80 per token (4 KiB each) and Qwen3.8-Flash-Next 96 (5 KiB). The only published R9700 rccl-tests table (4 cards, P2P off) is flat at 31-36 us up to 32 KiB, which is about 2.9 and 3.4 ms of communication per token, before any compute. https://amdai.net/lab-log/#2026-10-26-gguf-reupload-drift 2026-10-26-gguf-reupload-drift Mon, 26 Oct 2026 12:00:00 GMT On 2026-10-05 Unsloth re-uploaded files in its GLM-5.3-Flash and Qwen3.8-Flash-Next GGUF repos 'due to upstream changes'. We diffed the old and new files from the Hugging Face history. For GLM-5.3-Flash only shard 1 of each quant changed, and it holds metadata only (9.0 MiB, no tensors): the architecture name went from glm5next to glm5-next, the name mainline llama.cpp merged on 2026-09-30, and one key was added (attention.indexer.index_share_mtp = true). A copy downloaded before then can be fixed by replacing that one shard instead of re-downloading 186-272 GiB. For Qwen3.8-Flash-Next the UD-Q4/Q5/Q6 model shards did not change; the three non-shared MTP head files did, by one value: the MTP layer's attention compress ratio went from 0 to 4. https://amdai.net/lab-log/#2026-10-26-35b-a3b-mtp-vs-draft-r9700 2026-10-26-35b-a3b-mtp-vs-draft-r9700 Mon, 26 Oct 2026 12:00:00 GMT Published tests show a separate small draft model makes Qwen3.6-35B-A3B slower (one RTX 3090: 135.7 tok/s baseline vs 115.9-121.3 mean with drafts), while the model's own MTP head helps one user at a time: on an R9700 (Q4_K_M, Vulkan) one slot went from about 108 to about 133 tok/s, but MTP made a 3-slot job 1.58x slower. We read the headers of Unsloth's MTP GGUFs: the extra MTP block is Q8_0 attention with Q5_K/Q6_K experts in UD-Q5_K_XL, nothing below Q4. https://amdai.net/lab-log/#2026-10-26-multi-gpu-silent-corruption 2026-10-26-multi-gpu-silent-corruption Mon, 26 Oct 2026 12:00:00 GMT Several 2026 Radeon multi-GPU setups on ROCm/HIP produced fluent gibberish or blank output at normal speed with no error, and llama-bench still reported healthy numbers. The Windows layer-split case was fixed in ROCm's runtime (PR #10553, merged 2026-09-02); Windows -sm tensor on ROCm is still open; Linux cards with no peer-to-peer path need a no-peer-copy build. Always check text, not just tok/s, after a stack change. https://amdai.net/lab-log/#2026-10-26-slot-restore-hybrid-checkpoints 2026-10-26-slot-restore-hybrid-checkpoints Mon, 26 Oct 2026 12:00:00 GMT Before llama.cpp b11507, restoring a saved slot on hybrid (Gated DeltaNet) models like Qwen3.5/3.6/3.8 reported success but re-processed the whole prompt, because context checkpoints were not saved. PR #26004 now appends them to the slot file. A published AMD Vulkan test (Radeon 8050S, Qwen3.5-35B-A3B UD-Q4_K_XL, 58K tokens) cut the first request after restore from 181.9 s to 4.7 s. https://amdai.net/lab-log/#2026-10-26-thinking-budget-latency 2026-10-26-thinking-budget-latency Mon, 26 Oct 2026 12:00:00 GMT At about 33 tok/s (a Flash-class MoE on 2x R9700 with experts in DDR5), every 1,000 thinking tokens is roughly 31 seconds before the first visible word. llama.cpp can cap this per request with thinking_budget_tokens plus reasoning_budget_message, and since June 2026 a client can end thinking mid-stream with POST /v1/chat/completions/control, which allows a wall-clock deadline instead of a token count. A bare cap without a stop message hurts: HumanEval on a Qwen3.5 9B fell to ~79% vs ~89% with a message (~93% uncapped), and on Qwen3.6-35B-A3B a bare 4,096-token cap ran 170 s as reasoning spilled into the answer, vs 78 s with Qwen's own stop sentence. https://amdai.net/lab-log/#2026-10-26-r9700-hang-recovery-classes 2026-10-26-r9700-hang-recovery-classes Mon, 26 Oct 2026 12:00:00 GMT Public Radeon AI PRO R9700 (gfx1201) fault reports split into three recovery classes. (1) The server process dies and the driver's reset succeeds: a systemd Restart=on-failure is enough. (2) A silent wedge: the server stays up at 100% GPU busy and 0% memory busy but produces no tokens, or blocks at idle power with no error, so nothing restarts it and a /health check still answers 200. (3) A dead GPU: the reset ends with 'PSP resume failed' / ret -62 or a reset storm with a CPU soft lockup, and only a reboot helps; repeated auto-retries in that state have hard-hung an R9700. https://amdai.net/lab-log/#2026-10-26-endpoint-security 2026-10-26-endpoint-security Mon, 26 Oct 2026 12:00:00 GMT llama-server binds 127.0.0.1 by default, and --api-key protects every route except /health and the web-UI files. Without a key, /props, /slots and /v1/models are open, and the default CORS setting echoes any Origin. Builds from b8227 up to but not including b11393 have a 9.2-rated use-after-free on POST /completion (CVE-2026-107183), so update to b11393 or newer. The RPC backend used for multi-machine splits has no authentication and had a 9.8-rated remote code execution bug this year (fixed in b8492), so it belongs on a dedicated point-to-point link only. https://amdai.net/lab-log/#2026-10-26-r9700-tts-llama-server 2026-10-26-r9700-tts-llama-server Mon, 26 Oct 2026 12:00:00 GMT llama.cpp merged a native POST /tts endpoint for llama-server on 2026-10-20 (release b11781), with streaming, speaker-reference upload and router-mode autoload, so a TTS model can sit behind the same server as the LLM. Published R9700 results for Qwen3-TTS: the PyTorch reference runs at RTF 1.90 (slower than real time), while a custom HIP port reaches RTF 0.55 for the 1.7B model and 0.41-0.54 for the 0.6B. No llama.cpp R9700 TTS number has been published yet. https://amdai.net/lab-log/#2026-10-26-r9700-image-video-gen 2026-10-26-r9700-image-video-gen Mon, 26 Oct 2026 12:00:00 GMT From published R9700 ComfyUI runs: SDXL 1024x1024 runs at 4.49-4.6 it/s. 4-step distillation LoRAs cut Qwen-Image-2512 from 85.9 s to 12.4 s and Wan 2.2 A14B 480p from 341.7 s to 56.6 s, far more than any runtime knob. In a same-box A/B, TORCH_BLAS_PREFER_HIPBLASLT=1 plus COMFYUI_ENABLE_MIOPEN=1 made all four video workflows 5.3-7.3% slower and left Qwen-Image within 2%. https://amdai.net/lab-log/#2026-10-26-r9700-lora-pagefault 2026-10-26-r9700-lora-pagefault Mon, 26 Oct 2026 12:00:00 GMT The R9700 (gfx1201) is officially supported for LoRA/QLoRA training (Unsloth lists RDNA4 as Full; TheRock lists gfx1201 Release Ready), but stock hipBLASLt GEMMs can fault in the LoRA backward pass at a fixed step. On ROCm 7.2.4 a reporter saw bf16 LoRA on Qwen3-8B fault at step 85 and QLoRA NF4 at step 799. The fix (rocm-libraries PR 8909, merged 2026-08-14) ran clean in independent validations at about 1.5% extra step time. https://amdai.net/lab-log/#2026-10-26-embed-rerank-pick 2026-10-26-embed-rerank-pick Mon, 26 Oct 2026 12:00:00 GMT From official GGUF sizes and vendor scores: Qwen3-Embedding-4B Q5_K_M (2.69 GiB) scores 69.45 on MTEB multilingual, +5.1 over the 0.6B Q8_0 (0.60 GiB, 64.33), while the 8B Q5_K_M (5.05 GiB) adds only +1.1 more. Qwen3-Reranker-0.6B (0.60 GiB at Q8_0) lifts retrieval to 65.80 MTEB-R from 61.82, well ahead of bge-reranker-v2-m3 at 57.03. https://amdai.net/lab-log/#2026-10-25-strix-halo-vs-r9700 2026-10-25-strix-halo-vs-r9700 Sun, 25 Oct 2026 12:00:00 GMT Joining kyuz0's two public llama-bench grids (same scripts, best backend per row, 8 overlapping Q4+ models), one or two R9700s lead a 128 GB Ryzen AI Max+ 395 by a median ~2.0x on decode (tg128) and ~2.9x on prompt processing (pp512), widening to ~3.6-4.1x prefill at 32K-64K depth. Strix Halo's edge is capacity: it runs 80-90 GiB Q4/Q5 MoE models entirely in unified memory. https://amdai.net/lab-log/#2026-10-25-model-swap-router 2026-10-25-model-swap-router Sun, 25 Oct 2026 12:00:00 GMT llama-server's router mode (start it with no model and a --models-preset INI) loads models on demand, but it decides what to unload only by counting: --models-max defaults to 4 and the scheduler never checks VRAM or RAM. On a dual-R9700 box where one Flash-class model fills both cards and DDR5, run it with --models-max 1, or a request naming a second big model will try to load it alongside the first. llama-swap's capacity router lets you declare per-card and RAM budgets instead, and can also host non-llama.cpp servers. https://amdai.net/lab-log/#2026-10-25-llamacpp-grammar-sampling-cost 2026-10-25-llamacpp-grammar-sampling-cost Sun, 25 Oct 2026 12:00:00 GMT In current llama.cpp, a request with tools and tool_choice "auto" uses a lazy grammar that stays idle until the model starts a tool call, while tool_choice "required" or a response_format JSON schema constrains every token from the start. Sampling checks only the chosen token against the grammar and computes the full-vocabulary mask only when that token is rejected. Experimental backend (GPU) sampling, -bs, is turned off for any request that carries a grammar, lazy tool grammars included, so agent requests always sample on the CPU. https://amdai.net/lab-log/#2026-10-25-whisper-stt-rdna4 2026-10-25-whisper-stt-rdna4 Sun, 25 Oct 2026 12:00:00 GMT For voice input on RDNA4 cards, whisper.cpp (HIP or Vulkan) is the working path: a test on an R9700 ran the 11 s JFK sample with tiny.en in 206 ms, and an RX 9070 XT on Vulkan transcribed a 19.3-minute talk with large-v3-turbo in 48 s (about 24x real time, 1.5% WER). One RDNA4 tester reports the CTranslate2 ROCm forks behind faster-whisper produce wrong transcripts, so avoid that stack. The turbo model is 0.81 GiB at q8_0. whisper-server only serves /inference unless you pass --inference-path /v1/audio/transcriptions, which OpenAI-style agent clients expect. https://amdai.net/lab-log/#2026-10-25-flash-next-vision-cost 2026-10-25-flash-next-vision-cost Sun, 25 Oct 2026 12:00:00 GMT Flash-Next reads images through a separate 0.84 GiB projector (27-layer ViT, patch 16, merge 2, so one image token per 32x32 pixels). On an R9700 the GPU encoder takes 57 ms for a 260-token picture and 0.40 s for 972 tokens (3.13 s and 14.1 s on a CPU encoder). With llama.cpp's default cap of 4,096 tokens per image, a 1920x1080 screenshot becomes 2,040 tokens, so on a hybrid setup that prefills around 200 tok/s the image tokens, not the encoder, dominate time to first token. Capping images at 1,024 tokens halves that. https://amdai.net/lab-log/#2026-10-25-r9700-scoreboard-band 2026-10-25-r9700-scoreboard-band Sun, 25 Oct 2026 12:00:00 GMT Published single-card R9700 results for the standard llama-bench test: Vulkan tg128 126-146 t/s (FA off) and 131-153 (FA on), pp512 5,000-5,900; ROCm/HIP 112.7 / 121.6 tg128 at build 8ea2902. A card well under this band points to a driver, power or PCIe problem, not the hardware. The ROCm scoreboard's R9700 row is stale (93.84 tg128 at bd4ef13); newer builds are about 20% faster.