Weights
Quants, without folklore
A quant is a stored-weight encoding. Lower bits shrink the file and usually cost quality. This site’s floor is Q4. Thecalculatorestimates VRAM from these encodings. Numbers there are labeled estimates, not benches.
Q4 through Q8 (llama.cpp family)
Nominal bits-per-weight used by the calculator (llama.cpp-family labels, not measured entropy):
| Quant | Nominal bpw | House note |
|---|---|---|
| Q4_K_M | 4.85 | Floor. Prefer Q5 when it fits. |
| Q5_K_M / UD-Q5_K_XL | 5.69 | Daily-driver family. Prefer Q5 over Q4. |
| Q6_K / UD-Q6_K_XL | 6.56 | Flash agent family on our rig. |
| Q8_0 | 8.5 | Near-lossless GGUF; heavy. |
| BF16 | 16 | Unquantized. Rarely a home fit. |
UD-Q5_K_XL / UD-Q6_K_XL share the Q5 / Q6 buckets unless a Hugging Face listing supplies a publisher file size (decimal GB → GiB). Daily-driver listing:unsloth/Qwen3.6-35B-A3B-GGUF(UD-Q5_K_XL 26.6 GB). Flash:unsloth/Qwen3.8-Flash-Next-GGUF(UD-Q6_K_XL folder ~169 GB).
Unsloth UD dynamic quants
Unsloth’s UD (dynamic) GGUFs mix higher-precision tensors into a named bucket so important layers stay closer to the original. We do not invent a quality delta. When the publisher lists a file size, the calculator uses that size instead of params × bpw. Prefer the UD-Q5_K_XL / UD-Q6_K_XL files we name on /stacks.
KV cache types
Weights are one allocation. Context is another. The calculator’s KV estimate is2 × kv_layers × kv_heads × head_dim × context × bytes. Hybrid Qwen cards (3.6 / 3.8 Flash) only grow KV on full-attention layers — that count comes from the public model card, not from our tape measure.
- f16 — 2 bytes. Daily driver on our rig.
- q8_0 — 1 byte. Smaller KV; quality trade is not measured here.
- q4_0 — 0.5 bytes. Last KV lever before you cut context. Still not a weight quant below Q4.
Why nothing below Q4
Q1, Q2, Q3, and IQ1–IQ3 (including draft / MTP heads) stay off this site. They exist in the GGUF ecosystem. They are how people “fit” a model that does not fit, then wonder why the agent is drunk. The house rule is: prefer Q5 over Q4; never go below Q4; use the calculator and a smaller context or a smaller named model instead.