LLM Quantization: Weights vs KV Cache Explained

Quantization makes a language model smaller by storing its numbers with fewer bits, so each value takes less memory but becomes slightly approximate. Weight-only quantization compresses the model's fixed weights, which are the numbers learned in training and reused for every request. It does not shrink the KV cache, the memory a conversation creates as it grows, so a long chat can still run out of device memory. To fit longer conversations or more users, you also quantize the KV cache, and when someone says "a 4-bit model," you should ask whether the 4 bits describe the weights or the cache.

This guide explains both, with a worked example, a memory calculation, real code and the trade-offs, so you can find which store limits your device and target it.

#Key takeaways

• Quantization maps precise numbers to fewer allowed values. That needs fewer bits and loses some precision.
• Weights are fixed. They are learned once, shared by every request, and sit in memory the whole time.
• The KV cache grows. For each new token, every layer adds one key and one value to a saved sequence, and each active request has its own cache.
• Weight-only quantization does not touch the KV cache. KV cache quantization does the opposite: it leaves the weights unchanged.
• Both can reduce quality, because fewer allowed values mean less precision. Test on your own tasks.
• A "4-bit model" label is incomplete. Ask what is 4-bit: weights, KV cache, or both.

#What is quantization?

A language model is mostly numbers. Quantization replaces each precise number with the nearest value from a smaller set of allowed values, so it can be stored in fewer bits.

#A tiny worked example

Suppose four weights are [0.12, -0.53, 0.98, -0.08]. A simple symmetric 4-bit scheme uses integers from -8 to 7:

1. Find the largest absolute value: 0.98.
2. Choose a scale: 0.98 ÷ 7 = 0.14.
3. Round each value divided by the scale: [1, -4, 7, -1]. These are the stored 4-bit integers.
4. To use them, multiply back by the scale: [0.14, -0.56, 0.98, -0.14].

The restored numbers are close but not identical to the originals (for example 0.12 became 0.14, and -0.08 became -0.14). That small error is the cost of quantization. Real methods use smarter scales per small group of values and protect important weights, but the principle is the same.

#What are model weights?

Weights are the numbers the model learned during training. They are fixed, reused for every request, and shared by every user of that model copy. Because they are a shared, constant store, compressing them cuts the memory the model needs just to be loaded.

Rough arithmetic for a 7-billion-parameter model, ignoring overhead from scales and other tensors:

That is why 4-bit weight quantization lets large models fit on consumer GPUs and laptops. Two well-known weight-only methods are GPTQ and AWQ. The AWQ paper describes a hardware-friendly low-bit weight-only approach and reports that protecting just 1% of salient weights can greatly reduce quantization error [1][2].

#What is the KV cache?

When a model generates text, each new token must attend to the tokens before it. Recomputing all earlier keys and values at every step would be wasteful, so the model saves them and reuses them. That saved store is the key-value (KV) cache [3].

How it grows:

• For every new token, every layer adds one key vector and one value vector.
• The cache is per request. Each active conversation has its own.
• It grows with the length of the conversation and the number of simultaneous users.

The Hugging Face documentation notes that the KV cache can occupy a significant portion of memory and become a bottleneck for long-context generation [4].

#Weights vs KV cache at a glance

#How much memory does the KV cache use?

Per token, the cache stores a key and a value for each layer:

bytes per token = 2 × layers × KV heads × head dimension × bytes per value

Take a 7B-class model with 32 layers, 32 KV heads of size 128 (a hidden size of 4096) and FP16 values (2 bytes). This is the shape of the original Llama 2 7B, used here as an illustration:

• Per token: 2 × 32 × 32 × 128 × 2 = 524,288 bytes, which is 0.5 MiB.
• A 4,096-token conversation: 2 GiB.
• A 32,768-token conversation: 16 GiB.
• Eight simultaneous 4,096-token conversations: 16 GiB.

Compare that with the 4-bit weights, about 3.5 GB. At long context or with many users, the KV cache can be larger than the model itself. This is why a model that loads fine can still run out of memory in a long chat.

Many newer models use grouped-query attention, which shares keys and values across groups of heads and gives far fewer KV heads, so their caches are much smaller than this example. Always compute with your own model's configuration.

#What KV cache quantization saves

Storing the cache in 8 bits instead of 16 roughly halves it, and 4 bits roughly quarters it, before small overheads for scales:

#So why is "4-bit" an incomplete label?

Because quantization can target different stores:

• Weights only (often written W4A16: 4-bit weights, 16-bit activations). The model file is smaller, but the KV cache is unchanged.
• KV cache only (for example an FP8 KV cache). The weights stay as they were, and long contexts or more requests fit.
• Both. The most memory saved, with the most places for quality to slip.

When you read "4-bit model," ask: are the 4 bits on the weights or the KV cache? Then target whichever store is limiting your device.

#How to quantize the weights

• GPTQ and AWQ are post-training, weight-only methods that produce 4-bit models for common GPUs [1][2].
• GGUF quantization formats are widely used with llama.cpp for running models on CPUs and consumer hardware.
• Reported gains: the AWQ authors say their TinyChat framework offers more than 3x speedup over the Hugging Face FP16 implementation on desktop and mobile GPUs [2].

#How to quantize the KV cache

#With vLLM

vLLM supports storing the KV cache in FP8. From its documentation [5]:

bashvllm serve <model> --kv-cache-dtype fp8

or in Python:

pythonfrom vllm import LLM
llm = LLM(model="...", kv_cache_dtype="fp8")

Notes from the documentation:

• Supported values include fp8, fp8_e4m3 and fp8_e5m2.
• With fp8, all quantization scales default to 1.0. For best accuracy, the docs recommend calibrating scales on a dataset with llm-compressor.
• Some layer types, such as sliding-window attention, are more sensitive, so there is an option to keep chosen layers at native precision (--kv-cache-dtype-skip-layers).
• The smaller footprint lets you store more tokens, improving throughput and supporting longer context windows.

#With Hugging Face Transformers

Transformers has a quantized cache with several backends. This example comes from its documentation [4]:

pythonout = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    cache_implementation="quantized",
    cache_config={
        "nbits": 4,
        "backend": "hqq",
        "axis_key": 1,
        "axis_value": 1
    }
)

The documentation lists hqq, quanto and fp8 backends, and warns that quantizing the cache can hurt latency when the context is short and there is enough GPU memory without it, so find a balance between memory efficiency and latency [4].

#What research reports

The KIVI paper (ICML 2024) proposes a tuning-free 2-bit KV cache quantization. It finds that keys should be quantized per channel and values per token, and reports 2.6x less peak memory including model weights, up to 4x larger batch size, and 2.35x to 3.47x throughput on real inference workloads, with Llama, Falcon and Mistral models keeping almost the same quality [6]. These are the authors' reported results on their tested models, so measure your own.

#Does quantization hurt quality?

Yes, it can. Fewer allowed values mean less precision, whether the target is the weights or the cache. The effect depends on the method, the model, the bit width and the task.

How to check:

1. Measure on your own tasks, not only a general benchmark.
2. Test long contexts if you quantize the KV cache, since errors can accumulate across many tokens.
3. Compare against the full-precision model on the same prompts.
4. Watch sensitive layers. vLLM's documentation notes sliding-window layers can be more sensitive [5].
5. Use an evaluation set and track changes over time. See our LLM evaluation guide and reducing LLM hallucinations.

#Other ways to cut KV cache memory

Quantization is only one lever:

• Grouped-query attention in the model design, which means fewer KV heads.
• Sliding-window attention, where, per the Hugging Face docs, the cache stops growing once those layers reach the window size [4].
• Cache offloading to CPU memory, trading some speed for GPU memory [4].
• Better memory management in serving systems, such as paged KV cache designs.
• Shorter prompts and conversation summaries, so fewer tokens are stored. See LLM tokens explained, which covers why history is resent and how provider prompt caching differs from this GPU-side cache.

#Step-by-step: find your memory bottleneck

1. Compute weight memory: parameters × bytes per weight.
2. Compute KV cache per token from your model's config, using the formula above.
3. Multiply by your maximum context length and the number of concurrent requests.
4. Compare the weights and cache against your available GPU memory.
5. Pick the lever that targets the larger store (table below).
6. Test quality with and without the change.
7. Monitor in production for out-of-memory errors, latency and answer quality.

#Pros and cons

#Common mistakes

1. Assuming "4-bit" shrinks everything. It may only shrink the weights.
2. Ignoring concurrency. The cache is per request, so ten users means ten caches.
3. Skipping evaluation. Benchmarks on short prompts will not show long-context damage.
4. Quantizing the cache when you are not memory-limited, adding quality risk and sometimes latency for nothing.
5. Using one number for all models. Cache size depends on layers, KV heads and head size.

For the hardware background, see how fast the CPU, cache, RAM and SSD are. For how models generate text, see how large language models work. To cut inference cost more broadly, read LLM cost optimization, and for fast inference infrastructure, see why Rust is growing in AI.

#Need to run an LLM on your own hardware?

We help teams choose models, quantization and serving setups that fit their memory and latency budgets, and evaluate quality before launch. Get a scope estimate for your self-hosted LLM project, or read our AI agent development service.

#FAQ

#How does quantization make a language model smaller?

It stores the model's numbers with fewer bits, mapping precise values to a smaller set of allowed values. A 16-bit weight becomes a 4-bit one, which uses a quarter of the memory but is slightly approximate.

#Why does my model run out of memory in a long conversation after quantization?

Weight-only quantization shrinks the fixed weights, not the KV cache that grows with every token. A long conversation, or many simultaneous ones, can fill the memory with cache.

#What is the KV cache?

The saved keys and values for tokens the model has already processed. Each layer adds one key and one value per token, and the model reuses them instead of recomputing the past when choosing the next token.

#What is the difference between weight quantization and KV cache quantization?

Weight quantization compresses the model's learned parameters, so the model fits on the device. KV cache quantization compresses the per-request memory created during generation, so longer conversations and more requests fit.

#Does a 4-bit model have a 4-bit KV cache?

Not necessarily. A 4-bit label often refers only to the weights. Check whether the cache is also quantized, and to what precision.

#Does quantization reduce answer quality?

It can, because fewer allowed values mean less precision. The impact varies by method, model and task, so evaluate on your own use cases.

#Which should I quantize first, weights or the KV cache?

Whichever store limits your memory. If the model does not fit, quantize the weights. If it fits but long contexts or many users run out of memory, target the KV cache.

#How much memory does the KV cache use?

It equals 2 × layers × KV heads × head dimension × bytes per value × tokens × requests. For a 7B-class model with full multi-head attention in FP16, that is about 0.5 MiB per token.

#References

1. Frantar, E., et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." arXiv:2210.17323. arxiv.org/abs/2210.17323
2. Lin, J., et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. arxiv.org/abs/2306.00978
3. Hugging Face Transformers documentation. "Caching" explanation. huggingface.co/docs/transformers
4. Hugging Face Transformers documentation. "Cache strategies" (quantized cache, offloading). huggingface.co/docs/transformers/main/en/kv_cache
5. vLLM documentation. "Quantized KV Cache." docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html
6. Liu, Z., et al. "KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache." ICML 2024. arxiv.org/abs/2402.02750

Method and limits: concepts and reported figures come from the sources above, read in October 2026. Memory numbers are calculations for an illustrative 7B-class configuration, and performance and quality results are the authors' reports on their own tested models. Verify against your own model and hardware.