The transformer launch and the models that followed
In June 2017 a Google paper introduced the transformer. Within three years it had replaced almost every other architecture for language.
Words on this page (14)
- transformer
- : The neural network architecture behind modern language models. It uses attention to relate every token to the others.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- GPU
- : Graphics processing unit. A chip with thousands of cores, ideal for the matrix math in AI.
- parameter
- : One learned number inside a model. Modern models have billions of them. Training adjusts them.
- prompt
- : The input text you give a model, including instructions, examples and data.
- in-context learning
- : A model picking up a task from examples inside the prompt, without any retraining.
- skill
- : A folder of instructions and scripts an agent loads only when a task needs it.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- KV cache
- : Stored keys and values of earlier tokens, reused during generation so each new token is cheap to compute.
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- context window
- : The maximum number of tokens a model can look at in one call: your prompt, the conversation, tool results and the answer together.
- fine-tuning
- : Further training on a smaller, targeted set of examples to change behavior or style.
The launch
" Is All You Need" (Vaswani et al., 2017) dropped recurrence. Earlier models read text one at a time, which was slow to train and forgot early words. The processes all tokens in parallel and lets each one attend to the others. That fit well, so models could grow.
Two branches
In 2018 Google's BERT read text in both directions and became the standard for understanding tasks like search and classification. OpenAI's GPT read left to right and predicted the next token, which is also how it generates text. GPT-2 (2019, 1.5 billion ) wrote surprisingly coherent paragraphs. GPT-3 (2020, 175 billion parameters) could do new tasks from a few examples in the , with no retraining. That behavior is called , and it is why prompting became a .
Scaling laws
In 2020 researchers at OpenAI showed that test falls as a smooth power law as you increase parameters, data and compute. In 2022 DeepMind's Chinchilla work found that many large models were undertrained: for a fixed compute budget, parameters and tokens should grow together, roughly 20 tokens per parameter. This predictability is why labs were willing to spend huge sums on larger runs.
Two ways to learn language
BERT learns by hiding a word in the middle of a sentence and guessing it: "The cat sat on the ___ and purred." Because it sees words on both sides, it is good at understanding. GPT learns by hiding the future: given "The cat sat on the", guess the next word. That is exactly the job of writing text, so GPT-style models became the chatbots.
A worked example: growth in a few years
GPT-1 (2018) had about 117 million parameters. GPT-2 (2019) had 1.5 billion, roughly 13 times more. GPT-3 (2020) had 175 billion, another 117 times. Each jump needed more data and compute, and each brought new abilities nobody had explicitly trained for, like translating a sentence after seeing two examples in the prompt.
Why parallel training changed the size limit
An older recurrent model had to read a 1,000 token text in order, one token after another, because each step needed the one before it. A transformer processes all 1,000 positions in one pass. GPUs are built for exactly this kind of simultaneous work, so the same hardware time covers far more training text.
Trade-offs: what attention costs
Parallel is not free. Every token is compared with every earlier token. For 1,000 tokens that is about 500,000 comparisons (1,000 × 999 ÷ 2). Double the length to 2,000 and it is about 2 million, four times more. At 10,000 tokens it is roughly 50 million.
The saved keys and values (the ) also take memory that grows with length. That is why long contexts cost more and why labs keep working on cheaper attention. The design choice has a price: fast training and strong results, paid for with expensive long inputs.
A second example: choosing an encoder or a generator
Task one: sort a million support tickets into five categories. A BERT-style model that reads both directions fits, and a small one can be cheap to run. Task two: write a reply to each ticket. That needs generation, so a GPT-style model that predicts the next token is the natural fit.
The test is the output. If the answer is a label or a score, understanding-focused models are often enough. If the answer is new text, you need a generator.
In-context learning is not training
GPT-3 learned tasks "from examples in the prompt". The did not change. The examples sit in the , and the model continues the pattern it sees there. When the conversation ends, the weights are unchanged. Some apps add a separate memory feature on top. Real learning from your examples needs , which does change weights. Calling prompt examples "training the model" is a common loose use of the word.
Common mistake
Calling the transformer "the AI". It is an architecture, a way of wiring a network. What made chatbots was that architecture plus huge data, huge compute and later, training to follow instructions.
Imagine reading a long sentence one word at a time and having to remember everything in your head. By the end, the start has faded. That was how older language models worked. The transformer lets the model glance back at every earlier word at once and decide which ones matter. It could also train on all the words in parallel, so it could be made much bigger, much faster.
Check yourself
Why did the transformer scale better than earlier language models?
Not quite. Parallel training made it practical to build much larger models on much more data.
What did GPT-3 show?
Not quite. In-context learning is why prompting became a useful skill.
BERT hides a word in the middle of a sentence and guesses it. GPT guesses the next word. Which is better for writing text?
Not quite. Generating text means predicting what comes next, which is exactly GPT's training task.
You got 0 of 3 right.
Go deeper
- Paper
Vaswani et al. · The transformer paper.
- PaperBERTPaper
Devlin et al. · Bidirectional pretraining for understanding tasks.
- Paper
Brown et al. · The GPT-3 paper that showed in-context learning at scale.
- Paper
Kaplan et al. · Loss as a power law in size, data and compute.
- Paper
Hoffmann et al. · The Chinchilla paper on balancing parameters and tokens.
- ArticleThe Illustrated GPT-2Article
Jay Alammar · A visual walk through a decoder-only transformer.
Spotted a mistake or something unclear? Report it on GitHub.