Build a small GPT from scratch
The best way to demystify a language model is to write a tiny one. It is about 300 lines of Python.
Words on this page (21)
- transformer
- : The neural network architecture behind modern language models. It uses attention to relate every token to the others.
- self-attention
- : Attention where the tokens attend to other tokens in the same sequence.
- attention
- : The mechanism that lets each token weigh all earlier tokens by relevance when computing its next representation.
- gradient
- : The direction and rate at which the loss changes when each parameter changes. Training steps against it.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- overfitting
- : When a model memorizes training data and does worse on new data.
- loss
- : A number measuring how wrong the model's output is. Training tries to reduce it.
- parameter
- : One learned number inside a model. Modern models have billions of them. Training adjusts them.
- batch
- : A group of examples processed together in one step.
- epoch
- : One full pass through the training data.
- validation set
- : Data held out from training and used to tune choices and check that the model generalizes.
- GPU
- : Graphics processing unit. A chip with thousands of cores, ideal for the matrix math in AI.
- logarithm
- : The power you raise a base to in order to get a number. log base 2 of 8 is 3 because 2 to the power 3 is 8.
- backpropagation
- : The method that computes the gradient for every parameter by applying the chain rule backward through the network.
- optimizer
- : The algorithm that turns gradients into parameter updates, such as SGD or Adam.
- trace
- : The full record of one run: inputs, model calls, tool calls, outputs, timing and cost.
- learning rate
- : How big each gradient descent step is.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- fine-tuning
- : Further training on a smaller, targeted set of examples to change behavior or style.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- softmax
- : A function that turns a list of scores into probabilities that add up to 1.
What you build
A character-level GPT. You load a text file, map each character to an integer, and train a to predict the next character. You add pieces in this order: a bigram model (just counts), then , then multi-head , then feed-forward layers. Next come residual connections, which add a layer's input to its output so signals and flow, and layer normalization. Last is dropout, which randomly zeroes values during to reduce . Each step cuts , so you see why each piece exists.
Terms you will meet
: the learned numbers, which training adjusts. : a group of examples processed together. : one pass through the data. Overfitting: memorizing training data and failing on new data, spotted when validation loss rises while training loss falls. : data held out to measure generalization.
Do it
Follow the Karpathy video with the code open. Then change one thing at a time, such as the number of layers or the context length, and see what happens to loss. A helps, but a laptop is enough for the tiny version.
A worked example: the learning path in code
Step 1: count which character follows which (a bigram model). Loss about 2.5. Step 2: add one self-attention layer so a character can see earlier ones. Loss drops to about 2.1. Step 3: add several heads and layers, residual connections and layer norm. Loss about 1.5 and output looks like verse. Each step is a few lines and each step visibly helps.
What to try after
Change the context length and see what happens. Train on your own text. Make the model bigger until it overfits (validation loss rises) and then add dropout. You are doing, at small scale, exactly what the labs do.
How one training step works
Training repeats one loop thousands of times.
- Cut a batch of short snippets from your text file.
- At every position the model outputs a probability for each possible next character.
- The loss at that position is minus the natural of the probability it gave the true character.
- works out how much each parameter contributed to the loss.
- The moves every parameter a small step that lowers the loss.
one position. The text so far is "th" and the true next character is "e". If the model gave "e" a probability of 0.02, the loss is 3.91. After some steps it gives 0.10 and the loss is 2.30. The loss curve you watch is this number averaged over the batch.
How to tell it is working
Before training, the model guesses almost evenly. If your text has 65 distinct characters, each gets a probability near 1/65, so the starting loss is about ln 65, which is 4.17. A first loss far from that points to a bug in the data or code, not to a weak model. After that, log training loss and validation loss together every few hundred steps. Both should fall. If both stay flat, suspect the or a bug before you suspect the model size.
What goes wrong: it memorizes
The symptom is training loss that keeps falling while validation loss turns upward, and samples that repeat long stretches of the training file word for word. A small text file holds far fewer facts than a model of millions of parameters can store, so it copies. To confirm, print the loss on both sets and look for the gap opening. Fixes, in order of effort: more text, dropout, a smaller model, or stopping at the step with the lowest validation loss.
When this is the wrong tool
A model trained from scratch on one file only continues text in that file's style. It cannot answer questions or follow instructions. For real tasks you start from a pretrained model and adapt it, as the lesson shows.
Common mistake
Copying the code without running it and changing things. Experiments teach the lesson.
Before you start
This is easier if you have done the Building Blocks chapter just before it: , , gradients and attention. The code is short, and each piece maps to a lesson there.
A hobbyist trains a tiny GPT on a folder of Shakespeare plays on a laptop in ten minutes. At first it outputs random letters. After a few minutes it produces words, then lines that look like verse, nonsense but plausibly Shakespearean. Watching the loss number fall while gibberish becomes language does more for your understanding than any article.
Check yourself
Validation loss rises while training loss falls. What is happening?
Not quite. The model is memorizing the training set instead of generalizing.
What does a residual connection do?
Not quite. It makes deep networks trainable.
Your tiny GPT outputs repeated nonsense after one minute of training. What is the most likely reason?
Not quite. Early in training the loss is high and the output is gibberish. Watch the loss fall.
You got 0 of 3 right.
Go deeper
- Video
Andrej Karpathy · The standard walkthrough, about two hours.
- Repo
Andrej Karpathy · A minimal, readable GPT training codebase.
- Repo
Sebastian Raschka · The code for the book Build a Large Language Model (From Scratch).
- Course
Andrej Karpathy · The full series from micrograd to GPT and tokenizers.
Spotted a mistake or something unclear? Report it on GitHub.