basforge

Search titles, text and the “If it doesn’t work” cases.

    RU
    Cheat sheetAILocal models

    How much memory a model needs: parameters, quants, context and speed

    Memory for a model is the weights (parameters × bits per weight) plus the context cache, which grows with the conversation, plus 1–2 GB of overhead. Generation speed is bound by memory bandwidth: every token reads all the weights. Formulas and tables to estimate before downloading 40 GB.

    Weights: parameters × bits per weight

    weight size, GB ≈ parameters in billions × bits per weight / 8
    8B at Q4_K_M:  8 × 4.85 / 8 ≈ 4.8 GB

    A quant is how many bits are stored per weight. Fewer bits means less memory and more speed, but lower precision.

    Quant Bits per weight What you lose
    FP16 / BF16 16 nothing — that’s how models are released
    Q8_0 8.5 indistinguishable from FP16 by eye
    Q6_K 6.6 nearly indistinguishable
    Q5_K_M 5.7 small things on hard tasks
    Q4_K_M 4.85 noticeable on code and exact facts, but the main choice for your own hardware
    Q3, Q2 3–2 the model starts to get confused; better a smaller model at Q4

    A bigger model at Q4 usually beats a smaller one at Q8 in the same memory: 14B at Q4 (8.5 GB) versus 8B at Q8 (8.5 GB).

    Weights by model size, GB

    Parameters Q4_K_M Q5_K_M Q6_K Q8_0 FP16
    4B 2.4 2.8 3.3 4.3 8
    8B 4.8 5.7 6.6 8.5 16
    14B 8.5 10 11.5 14.9 28
    27B 16.4 19.2 22.1 28.7 54
    32B 19.4 22.8 26.2 34 64
    70B 42.4 49.8 57.4 74.4 140

    A real GGUF file differs by a few percent: some layers are quantized differently.

    Context: the cache grows with length

    For every token in the context the model stores attention keys and values (the KV cache):

    bytes per token = 2 × layers × KV heads × head size × 2 bytes (FP16)
    Llama 3.1 8B:   2 × 32 × 8 × 128 × 2 = 128 KiB per token
    Llama 3.1 70B:  2 × 80 × 8 × 128 × 2 = 320 KiB per token
    Context 8B 70B
    8K 1 GiB 2.5 GiB
    16K 2 GiB 5 GiB
    32K 4 GiB 10 GiB
    128K 16 GiB 40 GiB

    An 8B model “supports 128K”, but with that window it won’t fit on a 12 GB card: 4.8 GB of weights + 16 GiB of cache. That’s why Ollama sets the window below the maximum, and you raise it for the task (A model on your own hardware). Quantizing the cache (OLLAMA_KV_CACHE_TYPE=q8_0) halves it. It works with flash attention on — Ollama turns that on itself where the backend and GPU support it — and applies to every model at once. Models with many attention heads per key–value pair (high GQA), such as Qwen2, lose the most precision: on those, compare answers before and after.

    The total for a GPU

    needed ≈ weights + KV cache for the chosen context + 1–2 GB of overhead
    14B Q4 with a 16K context:  8.5 + ~2.5 + 1.5 ≈ 12.5 GB — tight on 12 GB, comfortable on 16 GB

    The per-token cache size of a 14B model depends on its architecture; for an estimate take the middle between 8B and 70B, or look at ollama show, which lists the number of layers and heads.

    MoE: memory by all parameters, speed by active ones

    A mixture-of-experts (MoE) model stores many subnetworks and turns on a few for each token. The name shows it: 30B-A3B means 30 billion in total, 3 billion active.

    Memory Speed
    dense 30B like 30B: ~18 GB at Q4 like 30B
    MoE 30B-A3B like 30B: ~18 GB at Q4 like 3B — several times faster

    That’s why MoE models suit setups with lots of memory but low bandwidth: a Mac with unified memory, a server with a lot of regular RAM.

    Speed: memory bandwidth / weight size

    Every new token reads all active weights from memory, so the generation ceiling is:

    tokens per second ≤ bandwidth, GB/s / size of active weights, GB

    In practice you get 60–80% of the ceiling. Reading a long prompt is faster: it’s processed in a batch.

    Hardware Memory GB/s 8B Q4 32B Q4 70B Q4
    PC, DDR5-5600, dual channel system 90 ≤ 18 ≤ 5 ≤ 2
    RTX 3060 12 GB 360 ≤ 74 doesn’t fit doesn’t fit
    RTX 4060 Ti 16 GB 288 ≤ 59 doesn’t fit doesn’t fit
    RTX 3090 / 4090 24 GB 936 / 1008 ≤ 190–210 ≤ 48–52 doesn’t fit
    RTX 5090 32 GB 1792 ≤ 370 ≤ 92 doesn’t fit
    Mac M4 Pro unified, up to 64 GB 273 ≤ 56 ≤ 14 ≤ 6
    Mac M4 Max unified, up to 128 GB 546 ≤ 113 ≤ 28 ≤ 13

    This shows why a 16 GB 4060 Ti is slower than a 3060: more memory, but a narrower bus. And why a 128 GB Mac runs 70B, but slowly.

    What it costs in the cloud

    For comparison: Haiku 5.5 over the API is $0.10 per million input tokens (Models as of October 2026). Your own GPU pays off when the data can’t leave, when you go through tens of millions of tokens a month, or when you already have the card. You won’t get flagship level on your own hardware: open models that fit in 24 GB are closer to the small cloud tiers, and the gap shows on code and long tasks. How big it is, only your own tasks will tell: run a dozen typical ones on both models and compare the answers.