Inside a modern LLM
Current models are still transformers. The changes are efficiency tricks that make them cheaper to train and run.
Words on this page (14)
- transformer
- : The neural network architecture behind modern language models. It uses attention to relate every token to the others.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- RoPE
- : Rotary position embedding: a way to tell the model where each token sits by rotating its query and key vectors.
- embedding
- : A list of numbers that represents the meaning of a piece of text or an image, so similar things sit close together.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- KV cache
- : Stored keys and values of earlier tokens, reused during generation so each new token is cheap to compute.
- ReLU
- : An activation function that outputs the input if positive, otherwise 0.
- mixture of experts
- : A design with many specialist sub-networks where a router uses only a few per token, so compute is far less than the total size.
- parameter
- : One learned number inside a model. Modern models have billions of them. Training adjusts them.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
- quantization
- : Storing weights with fewer bits (for example 4 instead of 16) to save memory at a small quality cost.
Changes from the 2017 design
The from 2017 is still the core of today's models. Newer models add a few tricks that make them cheaper, faster or able to read more. Here they are in plain words.
RMSNorm keeps the numbers inside the network at a steady size so stays stable. It is a simpler and faster version of an older step called layer normalization.
, short for rotary position , tells the model where each sits in the text. It does this by rotating the query and key by an angle that depends on position. It copes better with long texts than older methods.
Grouped-query , or GQA, lets several attention heads share one set of keys and values. The model stores less in its , so it needs less memory for long inputs.
SwiGLU is a small change to the feed-forward step. It uses two projections, one of which acts as a gate. It gives slightly better results than the older .
, or MoE, replaces one big feed-forward block with many smaller "expert" blocks. A router picks only a few of them for each token. The model can hold a lot of knowledge but only uses a fraction of it per token.
Why this matters to you
These choices decide price and speed. Fewer active mean cheaper tokens. A smaller KV cache means longer inputs fit in the same memory. When a model card says "671B total, 37B active", it means the model stores 671 billion parameters but only uses about 37 billion for each token.
A worked example: reading a model card
Imagine the card says: "128K context, GQA with 8 KV heads, 64 experts with 2 active, RoPE." In plain words: it can read about 100,000 words at once. It shares key and value memory across groups of heads. It has 64 specialist blocks and uses 2 for each token. It encodes position by rotation. From this you can guess that it is large to store but fairly cheap to run.
How grouped-query attention saves memory
The KV cache stores one key and one value vector per token, per layer, per key-value head. Use invented numbers for illustration: 32 layers, 128 numbers per head, 2 bytes per number. With 32 key-value heads each token costs 2 × 32 × 32 × 128 × 2 = 524,288 bytes, half a megabyte. An 8,000 token conversation then needs about 4.2 GB of cache. Let groups of heads share keys and values so only 8 are stored, and the same conversation needs about 1.0 GB. The quality drop is usually small, and the memory saving is exactly 32 divided by 8, a factor of four.
How a router picks experts
Take four experts. For one token the router, a small layer that scores each expert, outputs 2.0, 1.0, 0.5 and -1.0. The model keeps the top two scores, 2.0 and 1.0, and turns them into with : 0.73 and 0.27. The token's output is 0.73 times expert one's answer plus 0.27 times expert two's. The other two experts do no work for this token. A different token may score a different pair.
What active parameters do not save
Every expert must still sit in memory, because the next token might need any of them. A model with 671 billion parameters stored in 16-bit form needs about 1,342 GB just to hold them, even though only 37 billion work per token. Active parameters cut the compute per token. They do not cut the storage, so a big mixture model still needs a large cluster or heavy to run at all.
Common mistake
Comparing models by total parameters alone. Compare active parameters, context length and measured quality on your own task.
You read that a new model has "671 billion parameters" and assume it is slow and expensive to run. The model card says "37 billion active". Only a small group of expert sub-networks wake up for each word, so it costs about what a 37B model costs. The headline number and the real cost differ because of one design choice.
Check yourself
A card says "671B total, 37B active". What does that mean?
Not quite. MoE routes each token to a few experts, so compute follows the active parameters.
What does grouped-query attention save?
Not quite. Sharing key and value heads shrinks the cache that grows with context length.
Two models have the same total size, but one is a mixture of experts with fewer active parameters. Which is cheaper to run per token?
Not quite. Cost per token follows active parameters, not total size.
You got 0 of 3 right.
Go deeper
- Article
Sebastian Raschka · Why generation reuses past keys and values.
- Article
Sebastian Raschka · Compares DeepSeek, Llama, Qwen and others side by side.
- Paper
Su et al. · The RoPE paper.
- Paper
Noam Shazeer · Introduces SwiGLU and its siblings.
Spotted a mistake or something unclear? Report it on GitHub.