Cited, not measured on the local desktop
Local vision on Radeon
Local vision here means you pass a picture to a local LLM and get text back — caption, OCR, or a question about the image — without a hosted API. On AMDai the first path is llama.cpp multimodal: a text GGUF plus a matching multimodal projector (mmproj) GGUF, served by llama-mtmd-cli or llama-server. That is a different job from ComfyUI image generation. Speeds stay not yet measured by AMDai.
How we test — Measured cells cite official amdai-bench/run@1 cards. Estimates stay labeled. Q3 is estimate-only. Q1/Q2 stay off.
What local vision is
llama.cpp encodes the image with the mmproj file and feeds embeddings into the language model. The mtmd README says you typically need two GGUF files: the text model and a corresponding mmproj. libmtmd replaces the older llava.cpp path. The multimodal doc names llama-mtmd-cli and llama-server (OpenAI-compatible /chat/completions) as the tools. The house 35B daily driver on /gpus/r9700 is a text Qwen — do not invent an mmproj for that file.
Linux path — two files, then a picture
-
Step 1
Get a matching text GGUF and mmproj
The mtmd README: multimodal projector files are specific to each architecture. Do not pair a random mmproj with a random text file. ggml-org publishes ready multimodal GGUFs; their collection page is the cited shopper list. Most of those ready files ship Q4_K_M by default — that is their listing, not a house ranking. Q3 stays a labeled estimate. Q1/Q2 stay off. Start from that collection, not a retired Qwen vision family.
-
Step 2
Build llama-mtmd-cli on Ubuntu
Ubuntu 24.04+ is the measured AMDai OS. The Gemma 3 vision notes in llama.cpp say: cmake -B build, then cmake --build build --target llama-mtmd-cli. Vulkan llama.cpp stays first for tokens. Confirm the GPU with rocminfo before you download a pair of files. Windows is a cited second path, not measured on the local desktop.
-
Step 3
Run llama-mtmd-cli with --mmproj and --image
Cited from the llama.cpp multimodal doc and Gemma 3 notes: llama-mtmd-cli -m model.gguf --mmproj mmproj.gguf --image your_image.jpg. Or llama-server -m model.gguf --mmproj mmproj.gguf. -hf can pull a supported ggml-org pair; --no-mmproj disables multimodal on that load; --mmproj local_file.gguf overrides the projector.
-
Step 4
Know where the projector sits
The multimodal doc: the projector is offloaded to GPU by default. --no-mmproj-offload keeps it off the GPU. That is a RAM / VRAM trade, not a measured local desktop split. Minimum system RAM needed and RAM needed at full context for any text sidecar stay on the calculator. Do not invent an mmproj VRAM megabyte.
-
Step 5
Do not confuse this with image gen
Vision here is image in, text out. Local image generation — Stable Diffusion, SDXL, Flux-class — is ComfyUI on HIP/ROCm PyTorch at /image-gen. That path does not speak llama-server /v1. llama.cpp stays first for LLMs and for this mmproj job.
GPU-class fit
Every row is a labeled estimate. Confirm both GGUFs. Speed stays not yet measured by AMDai. Q3 is a labeled estimate.
-
8–16 GB class
Labeled estimate: start from a ggml-org Q4_K_M pair if that listing fits. Q3 is a labeled estimate. Confirm both GGUFs. Speed not yet measured by AMDai.
Speed · not yet measured by AMDai
-
20–24 GB class
Labeled estimate: more room for a larger text GGUF plus mmproj. Still two files. Q3 stays a labeled estimate. Speed not yet measured by AMDai.
Speed · not yet measured by AMDai
-
32 GB R9700
The measured house card is a text 35B. Vision is a separate cited pair — do not invent an mmproj for UD-Q5_K_XL. Speed not yet measured by AMDai.
Speed · not yet measured by AMDai
-
Strix Halo / shared memory
Labeled estimate on shared system memory. Size both GGUFs on the calculator. Speed not yet measured by AMDai.
Speed · not yet measured by AMDai
FAQ
-
What is an mmproj file?
The multimodal projector. llama.cpp encodes the image into embeddings with that GGUF, then the text model reads them. The mtmd README: you typically need the text GGUF and a matching mmproj. Pairing is architecture-specific.
-
Which vision model should I start with?
Cite the ggml-org multimodal GGUF collection and the llama.cpp multimodal doc. Their ready examples include Gemma 3 4B / 12B / 27B via -hf (the 1B variant has no vision). Most ready files list Q4_K_M. Q3 is a labeled estimate. Start from that collection, not a retired Qwen family. We have no local desktop vision tok/s.
-
Can I add vision to the house 35B daily driver?
Not as a file we publish. The R9700 daily driver is Qwen3.6-35B-A3B UD-Q5_K_XL, a text GGUF. We do not invent an mmproj for that card. Use a cited multimodal pair instead.