Local RAG on Radeon

What local RAG is

Chunk, embed, retrieve, answer. The embedding model is a small GGUF. Context is the retrieved window plus the question. LLM weights are the chat sidecar — that is where VRAM and system RAM move. Official nomic Q6_K is 108 MiB and F16 is 262 MiB; official UD-Q5_K_XL 35B on one R9700 is the measured daily driver. Adding those two on one card is an estimate until a first-party card lands. Empty cells stay not measured yet.

nomic-embed fit bands

Official nomic-embed-text-v1.5 GGUF sizes. House comparison is Q6_K / Q5_K_M / Q4_K_M. Q3_K_M is a labeled estimate — not a local desktop run. Runtime VRAM stays not measured yet. We do not invent a disk size.

Also cited, not in the Q6/Q5/Q4 set: Q8_0 140 MiB, F16 262 MiB, BGE small 33.4M params, BGE base 0.1B params. File listings without a published GGUF size stay not measured yet.

Pick by VRAM — embed plus the chat sidecar

Every row is a labeled estimate. Pick the nomic file from official sizes. The embed GGUF is small; the chat sidecar is the VRAM question. Speed stays not yet measured by AMDai.

• Radeon AI PRO R9700
• RX 9070 XT
• RX 9070
• RX 9060 XT
• RX 7900 GRE
• RX 7800 XT
• RX 7600 XT
• RX 6800 XT
• RX 6950 XT / 6900 XT
• RX 7700 XT
• RX 9070 GRE
• RX 6700 XT
• RX 7600

Linux path — index a folder, ask a question

• Ubuntu 24.04+ first

  Bare-metal Ubuntu 24.04+ is the AMDai measured path for the daily-driver LLM. RAG follows that OS even though we have no first-party retrieval card. See the GPU in rocminfo before you pick a HIP gfx pin. Windows is a cited second path, not the first try.

  • First-run setup
  • R9700 measured page
  • GPU directory

• Serve a dedicated embedding GGUF with llama.cpp

  The llama.cpp server README documents --embeddings (also --embedding): restrict the process to embedding use, and use it only with a dedicated embedding model. The OpenAI-shaped route is POST /v1/embeddings. That is a second llama-server, not the chat /v1 on :8080. Vulkan is the consumer default, same as the daily driver. House comparison is nomic-embed-text-v1.5 Q6_K (108 MiB official), Q5_K_M (95 MiB), and Q4_K_M (81 MiB). Q3_K_M (65 MiB) is a labeled estimate. Q8_0 (140 MiB) and F16 (262 MiB) stay cited larger files. The nomic card requires a task prefix such as search_query: on questions.

  • llama.cpp server README
  • nomic-embed GGUF
  • llama.cpp

• Index a folder of docs

  Chunk the files, POST each chunk to /v1/embeddings, store the vectors next to the text. That loop is the index. We do not rank a vector store or a Python library — pick one you already run, or use a harness that already does docs-plus-chat. Keep the index on fast local disk. A huge corpus on a slow USB stick will stall before VRAM does. Retrieval quality stays not yet measured by AMDai.

  • AnythingLLM harness
  • Open WebUI harness
  • Harnesses

• Ask a question against the daily-driver LLM

  Embed the question the same way you embedded the chunks. Retrieve the nearest chunks. Send those chunks plus the question to the measured llama.cpp Vulkan chat server — house daily driver is Qwen3.6-35B-A3B, -c 262144, f16 KV. Size -c for the retrieved window; a stuffed prompt can OOM the same way a long chat does. Long RAG prompts are prefill-bound — see /prefill for cited -b / -ub. Do not invent a recall score. The long-context gate means the stated window ran, not that the model found a fact in the middle.

  • 35B model page
  • Long context
  • Prefill
  • Calculator

• HIP only when rocminfo lists the gfx

  ROCm/HIP is the second llama.cpp compile when you need it. Pin -DAMDGPU_TARGETS to the string rocminfo prints. Do not mix a gfx110x pin onto a gfx1201 card. Vulkan stays the consumer default for both the embed server and the chat sidecar.

  • Vulkan vs HIP
  • Wrong gfx
  • llama.cpp

Windows — cited pointer

• Windows follows /setup/windows

  llama.cpp Vulkan is the honest Windows path AMDai already documents for tokens. The same Vulkan build can serve --embeddings. HIP/ROCm userspace stays a Linux product. Cited, not measured on the local desktop. Ubuntu 24.04+ on bare metal remains the measured default. We do not invent a Windows retrieval score.

  • Windows setup
  • Ubuntu setup
  • Windows / WSL2 preview

Next to the daily driver

RAG sits beside the measured llama.cpp Vulkan daily driver, not instead of it. The embed server is a second process (--embeddings). Speech and image gen are different jobs: whisper.cpp / Piper, and ComfyUI HIP/ROCm PyTorch. Harnesses that already say docs-plus-chat on this site are AnythingLLM and Open WebUI — that is a catalog note, not a ranking. There is no house Stack Lab recipe for RAG. Minimum system RAM needed and RAM needed at full context stay the shopper RAM lines when the chat sidecar offloads. A 64 GB kit estimate is the consumer shopper default — labeled estimate, not a measured RAG peak. We have not measured embed-plus-LLM on the local desktop.

Harnesses Stack Lab

When it breaks

• OOM while embedding or next to the chat sidecar

  Official nomic Q6_K is 108 MiB and F16 is 262 MiB. That can still starve a card that already holds a 35B GGUF if you load both into the same VRAM pool. Size the LLM on the calculator first. Run the embed server as its own process. When the sidecar offloads, shopper lines stay Minimum system RAM needed and RAM needed at full context. A second card is a /multi-gpu question — we have not measured RAG-plus-LLM on the local desktop.

  • OOM at long context
  • System RAM too small
  • Calculator
  • Dual / multi-GPU

• Huge corpus on a slow disk

  The index is files plus vectors. A large folder on a spinning drive or a USB stick stalls before the GPU does. Keep the corpus and the vector store on fast local NVMe. We do not invent an embeddings-per-second number for a slow disk.

  • Slow
  • Hardware

• Wrong context window on the chat sidecar

  Retrieved chunks plus the question consume the same -c budget as a long chat. If you stuff more tokens than the server window, you OOM or the model drops the start. Size -c on the calculator. House daily driver is 262144. Read Long context before you paste a whole book into one prompt.

  • OOM at long context
  • Long context
  • Calculator

FAQ

• What is local RAG on a Radeon?

  Chunk your files, embed the chunks with a dedicated llama.cpp embedding GGUF, retrieve the nearest ones, and ask the local LLM to answer from that context. The embed server uses --embeddings and /v1/embeddings. The chat sidecar stays llama.cpp Vulkan. Speeds and retrieval quality are not yet measured by AMDai.

  • llama.cpp
  • Calculator

• Which embedding file should I start with?

  Cite the nomic-embed-text-v1.5 GGUF table: Q6_K is 108 MiB, Q5_K_M is 95 MiB, Q4_K_M is 81 MiB. Q3_K_M (65 MiB) is a labeled estimate — not a local desktop run. Q8_0 (140 MiB) and F16 (262 MiB) stay cited larger files. BGE small is 33.4M params and BGE base is 0.1B params on the official cards — we do not invent those GGUF disk sizes. Q1 and Q2 stay off.

  • nomic-embed GGUF
  • Quants

• Is there an Open in Stack Lab recipe for RAG?

  No. There is no house Stack Lab recipe id for local RAG. Do not invent one. The gap is listed on /wanted. Use /stack-lab for the measured daily-driver and Flash recipes only.

  • Stack Lab
  • Wanted

Sources

• llama.cpp server README (--embeddings, /v1/embeddings)
• nomic-embed-text-v1.5 GGUF (official size table)
• nomic-embed-text-v1.5 model card (task prefixes)
• BAAI/bge-small-en-v1.5 model card (33.4M params)
• BAAI/bge-base-en-v1.5 model card (0.1B params)