Make your own GGUF quants for a Radeon GPU with llama.cpp

Disk and RAM for a 35B MoE

Worked example is Qwen3.6-35B-A3B — official card 35B total / 3B active. Source sizes come from Hugging Face listings. Computed sums stay labeled estimate. Hugging Face listings for Qwen/Qwen3.6-35B-A3B model-00001-of-00026.safetensors through model-00026-of-00026.safetensors (sum 71.9 GB, estimate). Hugging Face listings for unsloth/Qwen3.6-35B-A3B-GGUF BF16/Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf + 00002-of-00002.gguf (sum 69.4 GB, estimate).

Minimum system RAM needed for convert is a 96 GB kit kit estimate: 71.9 GB HF weights (Hugging Face listings for Qwen/Qwen3.6-35B-A3B model-00001-of-00026.safetensors through model-00026-of-00026.safetensors (sum 71.9 GB, estimate)) + 8 GB OS baseline + 16 GB headroom, rounded to a real DIMM. llama.cpp: memory and disk requirements are the same while converting. --use-temp-file is the cited escape if the process is killed. RAM needed at full context is the inference KV line at the model’s full context — not the convert. GB here is Linux GiB printed as GB.

Quality and speed (Q6 / Q5 / Q4, Q3 estimate)

Decode cells are official local desktop Qwen3.6-35B-A3B llama.cpp Vulkan cards on one R9700 — same numbers as /compare/q4-q5-q6. UD-Q3_K_XL is a labeled estimate (Hugging Face listing for Qwen3.6-35B-A3B-UD-Q3_K_XL.gguf (16.8 GB)). PPL / KLD stay not measured on the local desktop. Prefer Q5 over Q4 when the file still fits.

How-to

Linux first. Build llama.cpp, then stay in that tree for convert, imatrix, quantize, and perplexity. Minimum system RAM needed and RAM needed at full context print above.

1. Step 1

   Stage the work on NVMe, not a slow bulk drive

   A 35B MoE convert writes the Hugging Face folder and a high-precision GGUF next to it. Hugging Face listings for Qwen/Qwen3.6-35B-A3B model-00001-of-00026.safetensors through model-00026-of-00026.safetensors (sum 71.9 GB, estimate). Hugging Face listings for unsloth/Qwen3.6-35B-A3B-GGUF BF16/Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf + 00002-of-00002.gguf (sum 69.4 GB, estimate). llama.cpp’s quantize README: the model is fully loaded, and “memory and disk requirements are the same.” Leave leftover space for one Q6 / Q5 / Q4 output plus, if you run KLD, a large logit file. Put that on NVMe. A USB or archive HDD will stall the convert.

   • NVMe
   • Qwen3.6-35B-A3B
   • Calculator

2. Step 2

   Convert the HF folder to an f16 or bf16 GGUF

   Install llama.cpp Python requirements, then run convert_hf_to_gguf.py on the local Qwen/Qwen3.6-35B-A3B folder. Official --outtype values include f16, bf16, and auto (highest-fidelity 16-bit). The quantize README example uses --outtype bf16; --outtype auto (or omitting --outtype) is the usual pick when the card is already 16-bit. --remote is experimental and still downloads config and tokenizer. After a LoRA merge, the same convert step lives on /fine-tuning — this page does not retell adapters.

   # Ubuntu 24.04+ · llama.cpp first. Stage on NVMe, not a slow bulk drive.
   # convert_hf_to_gguf.py --outtype choices (official): f32, f16, bf16, q8_0, auto.
   # --outtype auto (or omitting --outtype) is the 16-bit pick when the HF folder is already 16-bit.
   # --remote is marked experimental; prefer a local folder after you download the card.
   python convert_hf_to_gguf.py ./Qwen3.6-35B-A3B \
     --outfile ./Qwen3.6-35B-A3B-bf16.gguf \
     --outtype bf16

   • Fine-tuning (merge, then convert)
   • llama.cpp
   • First-run setup

3. Step 3

   Optional: llama-imatrix with your calibration text

   llama.cpp’s imatrix README: compute an importance matrix from a model and a text file, then pass it to llama-quantize to reduce quality loss. Mandatory flags are -m / --model and -f / --file. Default output is imatrix.gguf. GPU offload uses -ngl. This is your calibration text, not Unsloth’s unpublished mix — a stock imatrix Q5_K_M is still not a UD-Q5_K_XL file.

   # Optional. -m and -f are mandatory on the imatrix README.
   # Use your own calibration text. Do not claim this matches Unsloth UD.
   ./build/bin/llama-imatrix \
     -m ./Qwen3.6-35B-A3B-bf16.gguf \
     -f calibration.txt \
     -o imatrix.gguf \
     -ngl 99

   • imatrix
   • Quants
   • Unsloth UD

4. Step 4

   llama-quantize to Q6_K, Q5_K_M, or Q4_K_M

   The official two-phase path is convert, then llama-quantize. House-rig outputs on this page are Q6_K, Q5_K_M, and Q4_K_M. Pass --imatrix imatrix.gguf when you built one. Do not --allow-requantize from an already-quantized GGUF unless you accept the README warning that it can severely reduce quality versus 16-bit. UD-Q3_K_XL is a labeled estimate in the table only — do not produce a Q3 file on the house rig. Q1/Q2 stay off.

   # House-rig outputs: Q6_K, Q5_K_M, Q4_K_M. Do not produce a Q3 file on this box.
   ./build/bin/llama-quantize --imatrix imatrix.gguf \
     ./Qwen3.6-35B-A3B-bf16.gguf ./Qwen3.6-35B-A3B-Q6_K.gguf Q6_K
   ./build/bin/llama-quantize --imatrix imatrix.gguf \
     ./Qwen3.6-35B-A3B-bf16.gguf ./Qwen3.6-35B-A3B-Q5_K_M.gguf Q5_K_M
   ./build/bin/llama-quantize --imatrix imatrix.gguf \
     ./Qwen3.6-35B-A3B-bf16.gguf ./Qwen3.6-35B-A3B-Q4_K_M.gguf Q4_K_M

   • Quants
   • Q6 vs Q5 vs Q4
   • Stacks

5. Step 5

   Verify with llama-perplexity (and optional KLD)

   llama.cpp measures quant accuracy loss with perplexity and/or KL-divergence versus the f16 / bf16 baseline. The perplexity README uses Wikitext-2 (scripts/get-wikitext-2.sh) unless noted, and says PPL is not comparable across models. Record baseline logits with --kl-divergence-base, then rerun the quant with --kl-divergence-base plus --kl-divergence. We have no house PPL or KLD score — run the tool, do not copy a screenshot.

   # llama.cpp perplexity README: Wikitext-2 via scripts/get-wikitext-2.sh unless noted.
   # Numbers are not comparable across models or tokenizers. We do not invent a house PPL.
   ./build/bin/llama-perplexity -m ./Qwen3.6-35B-A3B-Q5_K_M.gguf -f wiki.test.raw
   
   # Record f16/bf16 logits, then compare the quant. Cite llama.cpp perplexity README.
   # The logit file is very large (README: 11 GiB for LLaMA 2, 37 GiB for LLaMA 3 on Wikitext-2).
   # 35B size is not listed — stage leftover on NVMe. We do not invent a 35B .kld size.
   ./build/bin/llama-perplexity -m ./Qwen3.6-35B-A3B-bf16.gguf \
     -f wiki.test.raw --kl-divergence-base ./baseline.kld
   ./build/bin/llama-perplexity -m ./Qwen3.6-35B-A3B-Q5_K_M.gguf \
     -f wiki.test.raw --kl-divergence-base ./baseline.kld --kl-divergence

   • KL-divergence
   • amdai-bench
   • How we test

6. Step 6

   Then record an amdai-bench run

   PPL / KLD is the quality check. Shopper speed and VRAM still come from an amdai-bench/run@1 card — same method as /benchmarks/methodology. Launch llama-server with the new GGUF, then record decode, prefill, peak VRAM, and both RAM lines. Link the card from /results. Do not invent a tok/s.

   • amdai-bench
   • Methodology
   • Run cards
   • Stack Lab

FAQ

• When should I make my own GGUF instead of downloading Unsloth UD?

  When the model has no Q5 / Q6 GGUF yet, or you want an imatrix built on your own calibration text. The house default is still a downloaded Unsloth UD-*_XL file. Stock llama-quantize does not reproduce UD dynamic mixes. After a LoRA merge, convert then quantize — /fine-tuning covers the merge.

  • Quants
  • Fine-tuning
  • Stacks

• Which quant should I produce on a Radeon box?

  Q6_K, Q5_K_M, or Q4_K_M. Prefer Q5 over Q4 when the file still fits. Q3 is a labeled estimate on /compare/q4-q5-q6 — not a house-rig quantize command. Q1/Q2 stay off.

  • Q6 vs Q5 vs Q4
  • Quants
  • Calculator

• How much disk and RAM does a 35B convert take?

  Hugging Face listings for Qwen/Qwen3.6-35B-A3B model-00001-of-00026.safetensors through model-00026-of-00026.safetensors (sum 71.9 GB, estimate). Hugging Face listings for unsloth/Qwen3.6-35B-A3B-GGUF BF16/Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf + 00002-of-00002.gguf (sum 69.4 GB, estimate). llama.cpp says convert memory and disk match because the model is fully loaded. Minimum system RAM needed for convert is a 96 GB kit estimate (71.9 GB weights + 8 GB OS baseline + 16 GB headroom, rounded to a real DIMM). RAM needed at full context is the inference line for the produced file at the model’s full context — measured on the house UD-Q5 when a card recorded it. Stage on NVMe.

  • Qwen3.6-35B-A3B
  • Calculator
  • NVMe

• How do I know the quant is any good?

  llama-perplexity versus the f16 / bf16 GGUF, or KL-divergence with --kl-divergence-base then --kl-divergence (llama.cpp perplexity README). Then an amdai-bench run for decode / VRAM. We do not invent a house PPL or KLD number.

  • amdai-bench
  • How we test
  • KL-divergence

Sources

• llama.cpp convert_hf_to_gguf.py
• llama.cpp tools/quantize README
• llama.cpp tools/imatrix README
• llama.cpp tools/perplexity README
• llama.cpp repository
• Unsloth — Dynamic 3.0 GGUFs
• Qwen — Qwen3.6-35B-A3B model card
• Unsloth — Qwen3.6-35B-A3B-GGUF
• Unsloth — Qwen3.6-35B-A3B-GGUF file tree