Quantization and serving
Running a model is a memory and speed problem. Quantization and smart batching make it cheap enough to use.
Words on this page (13)
- quantization
- : Storing weights with fewer bits (for example 4 instead of 16) to save memory at a small quality cost.
- parameter
- : One learned number inside a model. Modern models have billions of them. Training adjusts them.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- GPU
- : Graphics processing unit. A chip with thousands of cores, ideal for the matrix math in AI.
- KV cache
- : Stored keys and values of earlier tokens, reused during generation so each new token is cheap to compute.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- inference
- : Running a trained model to produce output, as opposed to training it.
- prefill
- : The first stage of generation, where the model reads the whole prompt at once.
- prompt
- : The input text you give a model, including instructions, examples and data.
- decode
- : The second stage of generation, where the model writes one token at a time.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- batch
- : A group of examples processed together in one step.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
Memory math
A model with N stored in 16-bit format needs about 2N bytes. A 70 billion parameter model needs about 140 GB for , more than one holds. In 8-bit it needs 70 GB, in 4-bit about 35 GB. Add memory for the , which grows with context length and the number of simultaneous users.
Quantization
It stores weights with fewer bits, mapping ranges of values to a small set of levels. Quality at 8-bit is usually negligible and at 4-bit is small for large models, larger for small ones. Formats like GGUF are for local with llama.cpp. Methods like AWQ and GPTQ target GPU serving.
Serving
Generation has two phases: reads the whole in parallel, then produces one at a time. Decode is limited by memory bandwidth. Servers such as vLLM use continuous batching, which adds new requests into a running , and paged , which manages the KV cache like virtual memory, to raise throughput. Speculative decoding has a small model draft several tokens that the large model verifies in one pass.
A worked example: how a number is squeezed
Weights between -1 and 1 stored as 4-bit integers (16 levels). A weight of 0.37 is mapped to the nearest level, say 0.33. The error is small for any one weight, and across billions the errors mostly average out. Large models tolerate this well. Small ones lose more.
Why generation is slow in a particular way
The first stage (prefill) reads the whole prompt at once, fast. The second (decode) produces one token at a time and must read all the weights from memory each time, so memory speed, not raw compute, limits it. Servers batch many users together to share that reading cost.
How the squeezing works, step by step
Take weights between -1 and 1 and 4 bits, which give 16 levels. The levels are spaced 2 / 15 = 0.133 apart: -1, -0.867, and so on. To store 0.37:
- Shift and scale it: (0.37 + 1) / 0.133 = 10.3.
- Round to the nearest whole level, 10.
- Store the integer 10 plus the shared scale.
- When running, convert back: -1 + 10 × 0.133 = 0.333.
No weight is ever off by more than half a step, 0.067.
What goes wrong: one outlier
Suppose one weight in a row is 8 and the rest are near 0.3. The range is now -8 to 8, so the step grows to 1.07. The nearest levels to 0.3 are -0.53 and 0.53, so every small weight snaps to 0.53 and the layer breaks. The symptom is a model that works at 8-bit and produces garbage at 4-bit. Practical methods give each small group of weights its own scale, so one outlier damages only its group. To confirm, compare perplexity or your own test answers before and after.
What a serving budget looks like
Memory holds weights plus the KV cache. Invented numbers for a 70B model with grouped-query attention: 80 layers, 8 key-value heads, 128 numbers per head, 2 bytes each. That is 327,680 bytes per token, so one 8,000 token conversation takes 2.6 GB. Ten users at once need 26 GB on top of 35 GB of 4-bit weights.
Why batching pays
Say reading the 35 GB of weights takes 35 ms on a card that moves 1,000 GB per second. One user then gets about 29 tokens a second. Eight users in one batch share that same read, so the step still takes about 35 ms but produces 8 tokens: about 229 tokens a second in total. This ignores the cache reads, which grow with the batch.
Common mistake
Quantizing a small model to 2 bits and being surprised it got worse.
A model that needs 140 GB of memory will not fit on any single consumer card. Squeeze every number from 16 bits to 4 and it needs 35 GB. Squeeze further or shrink the model and it fits on a laptop. Quantization is the trade of a little accuracy for a lot of room, like saving a photo as a smaller JPEG.
Check yourself
How much memory do the weights of a 70B model need at 4-bit?
Not quite. 70 billion parameters at half a byte each is about 35 GB.
Which generation phase is limited by memory bandwidth?
Not quite. Producing one token at a time repeatedly reads the weights and KV cache.
A 4-bit quantized small model performs noticeably worse than its 16-bit version. Why?
Not quite. Large models tolerate low precision better than small ones.
You got 0 of 3 right.
Go deeper
- Repo
vLLM project · Throughput-focused serving with paged attention.
- Repo
ggml · Local inference and the GGUF format.
- Article
Kipply · The memory and compute math of serving.
- Paper
Kwon et al. · The vLLM paper.
Spotted a mistake or something unclear? Report it on GitHub.