Weights: parameters × bits per weight
weight size, GB ≈ parameters in billions × bits per weight / 8
8B at Q4_K_M: 8 × 4.85 / 8 ≈ 4.8 GB
A quant is how many bits are stored per weight. Fewer bits means less memory and more speed, but lower precision.
| Quant | Bits per weight | What you lose |
|---|---|---|
FP16 / BF16 |
16 | nothing — that’s how models are released |
Q8_0 |
8.5 | indistinguishable from FP16 by eye |
Q6_K |
6.6 | nearly indistinguishable |
Q5_K_M |
5.7 | small things on hard tasks |
Q4_K_M |
4.85 | noticeable on code and exact facts, but the main choice for your own hardware |
Q3, Q2 |
3–2 | the model starts to get confused; better a smaller model at Q4 |
A bigger model at Q4 usually beats a smaller one at Q8 in the same memory: 14B at Q4 (8.5 GB) versus 8B at Q8 (8.5 GB).
Weights by model size, GB
| Parameters | Q4_K_M | Q5_K_M | Q6_K | Q8_0 | FP16 |
|---|---|---|---|---|---|
| 4B | 2.4 | 2.8 | 3.3 | 4.3 | 8 |
| 8B | 4.8 | 5.7 | 6.6 | 8.5 | 16 |
| 14B | 8.5 | 10 | 11.5 | 14.9 | 28 |
| 27B | 16.4 | 19.2 | 22.1 | 28.7 | 54 |
| 32B | 19.4 | 22.8 | 26.2 | 34 | 64 |
| 70B | 42.4 | 49.8 | 57.4 | 74.4 | 140 |
A real GGUF file differs by a few percent: some layers are quantized differently.
Context: the cache grows with length
For every token in the context the model stores attention keys and values (the KV cache):
bytes per token = 2 × layers × KV heads × head size × 2 bytes (FP16)
Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 = 128 KiB per token
Llama 3.1 70B: 2 × 80 × 8 × 128 × 2 = 320 KiB per token
| Context | 8B | 70B |
|---|---|---|
| 8K | 1 GiB | 2.5 GiB |
| 16K | 2 GiB | 5 GiB |
| 32K | 4 GiB | 10 GiB |
| 128K | 16 GiB | 40 GiB |
An 8B model “supports 128K”, but with that window it won’t fit on a 12 GB card: 4.8 GB of weights + 16 GiB of cache.
That’s why Ollama sets the window below the maximum, and you raise it for the task (A model on your own hardware). Quantizing the
cache (OLLAMA_KV_CACHE_TYPE=q8_0) halves it. It works with flash attention on — Ollama turns that on itself where the
backend and GPU support it — and applies to every model at once. Models with many attention heads per key–value pair
(high GQA), such as Qwen2, lose the most precision: on those, compare answers before and after.
The total for a GPU
needed ≈ weights + KV cache for the chosen context + 1–2 GB of overhead
14B Q4 with a 16K context: 8.5 + ~2.5 + 1.5 ≈ 12.5 GB — tight on 12 GB, comfortable on 16 GB
The per-token cache size of a 14B model depends on its architecture; for an estimate take the middle between 8B and 70B,
or look at ollama show, which lists the number of layers and heads.
MoE: memory by all parameters, speed by active ones
A mixture-of-experts (MoE) model stores many subnetworks and turns on a few for each token. The name shows it:
30B-A3B means 30 billion in total, 3 billion active.
| Memory | Speed | |
|---|---|---|
| dense 30B | like 30B: ~18 GB at Q4 | like 30B |
| MoE 30B-A3B | like 30B: ~18 GB at Q4 | like 3B — several times faster |
That’s why MoE models suit setups with lots of memory but low bandwidth: a Mac with unified memory, a server with a lot of regular RAM.
Speed: memory bandwidth / weight size
Every new token reads all active weights from memory, so the generation ceiling is:
tokens per second ≤ bandwidth, GB/s / size of active weights, GB
In practice you get 60–80% of the ceiling. Reading a long prompt is faster: it’s processed in a batch.
| Hardware | Memory | GB/s | 8B Q4 | 32B Q4 | 70B Q4 |
|---|---|---|---|---|---|
| PC, DDR5-5600, dual channel | system | 90 | ≤ 18 | ≤ 5 | ≤ 2 |
| RTX 3060 | 12 GB | 360 | ≤ 74 | doesn’t fit | doesn’t fit |
| RTX 4060 Ti | 16 GB | 288 | ≤ 59 | doesn’t fit | doesn’t fit |
| RTX 3090 / 4090 | 24 GB | 936 / 1008 | ≤ 190–210 | ≤ 48–52 | doesn’t fit |
| RTX 5090 | 32 GB | 1792 | ≤ 370 | ≤ 92 | doesn’t fit |
| Mac M4 Pro | unified, up to 64 GB | 273 | ≤ 56 | ≤ 14 | ≤ 6 |
| Mac M4 Max | unified, up to 128 GB | 546 | ≤ 113 | ≤ 28 | ≤ 13 |
This shows why a 16 GB 4060 Ti is slower than a 3060: more memory, but a narrower bus. And why a 128 GB Mac runs 70B, but slowly.
What it costs in the cloud
For comparison: Haiku 5.5 over the API is $0.10 per million input tokens (Models as of October 2026). Your own GPU pays off when the data can’t leave, when you go through tens of millions of tokens a month, or when you already have the card. You won’t get flagship level on your own hardware: open models that fit in 24 GB are closer to the small cloud tiers, and the gap shows on code and long tasks. How big it is, only your own tasks will tell: run a dozen typical ones on both models and compare the answers.