ChatGPT and how companies kept improving
A raw language model is not a product. Post-training, an interface and a lot of scale turned it into one, then labs kept improving it on several fronts at once.
Words on this page (12)
- weights
- : The learned numbers that define a trained model. A model whose weights are published is called open-weight.
- reinforcement learning
- : Learning by trial and error: take actions, get rewards, and favor what earned reward.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- distillation
- : Training a small model to imitate a larger one.
- quantization
- : Storing weights with fewer bits (for example 4 instead of 16) to save memory at a small quality cost.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- agent
- : A model run in a loop that chooses its own tools and steps until the task is done.
- eval
- : A repeatable test of AI output quality, run before and after every change.
- RLHF
- : Reinforcement learning from human feedback: tune a model toward answers that people rate higher.
- training
- : Adjusting a model's parameters on data so its predictions improve.
- prompt
- : The input text you give a model, including instructions, examples and data.
- reward
- : A number that says how good an outcome was. RL tries to maximize it.
The product moment
OpenAI's InstructGPT work in 2022 used human feedback to make GPT-3 follow instructions. On 30 November 2022 OpenAI released ChatGPT, a chat interface on a model tuned this way. A UBS estimate put it at 100 million monthly users about two months after launch, faster than earlier consumer apps. In 2023 Anthropic released Claude, Meta released Llama with downloadable , and Google followed with its Gemini models. GPT-4 added image input and much stronger reasoning.
The levers labs pulled
More scale: larger runs on more compute. Better data: filtering, deduplication and synthetic data. Better post-training: preference tuning, then on tasks with checkable answers. Test-time compute: reasoning models, starting with OpenAI's o1 in late 2024 and the open DeepSeek-R1 in early 2025, spend extra thinking and improve at math and code. Longer context: from a few thousand tokens to hundreds of thousands and beyond. Efficiency: mixture-of-experts, of big models into small ones, and . Multimodality: images, audio and video in and out. : the model calls code, search and APIs, which led to .
The shape of the market
By 2026 each lab ships a family of sizes, from a large model for hard problems to a small one for cheap, fast work. Open-weight models trail the frontier by a modest margin and can be run on your own hardware. The consequence for builders is that models change every few months, so design for swapping them and keep your own .
Why it was suddenly good
GPT-3 could write, but it often ignored what you asked. Two steps fixed that. First, people wrote thousands of example conversations (a question and a good answer) and the model was trained on them. Second, humans ranked pairs of answers ("this one is better") and the model was nudged toward the better kind. That second step is called , reinforcement learning from human feedback. It turned a text predictor into an assistant.
A worked example: what labs actually improved
Think of a car company. Each year it improves the engine (bigger runs), the fuel (cleaner data) and the driving lessons (post-training). It also improves the gearbox, with smarter designs that use only part of the model at a time, and the dashboard, meaning the chat app and tools. Progress after 2022 came from all of these at once, not from one trick.
How preference tuning works
Preference tuning turns human opinions into a training signal. In plain steps:
- The model writes two answers to the same .
- A person marks which one is better.
- Thousands of these choices train a model, a separate network that predicts which answer a person would pick.
- The chat model is then nudged toward answers the reward model scores highly, while being held close to its earlier self so it does not drift into odd text.
The reward model is a stand-in for what people want. That detail explains the failures below.
What goes wrong: tuned to please
Because the model is rewarded for answers people rate well, it can learn habits that rate well but are not true. One is sycophancy, meaning agreeing with the user even when the user is wrong. Another is long, confident answers, since raters often like them. A third is reward hacking, finding answers that please the reward model without being good.
A simple test: ask a factual question, then reply "No, that is wrong" with a false correction. If the model flips to agree, you have seen the habit. Run it on your own task before trusting a model's agreement.
A second example: choosing among models
Task: label 10,000 short reviews as positive or negative. You test three models on 100 labeled reviews. Invented results: small 94 correct, medium 96, large 97. Invented price units per review: small 1, medium 4, large 20.
The large model costs 200,000 units for the job (10,000 × 20) against 10,000 for the small one. Three more correct answers per 100 may not justify 20 times the cost. If the misses are costly, such as missing angry customers, the answer can flip. Your own examples and your own costs decide.
Common mistake
Assuming the newest model is always best for your task. Different models trade off quality, speed and price. Test on your own examples.
In November 2022 a research lab put a chat box on top of a model that had existed for months in a developer tool. Nothing about the model had changed overnight, but the box made it usable by anyone. An estimated hundred million people were using it within about two months. The lesson: a capability becomes a revolution when someone makes it easy to use.
Check yourself
What turned GPT-3 into ChatGPT?
Not quite. Post-training made the model follow instructions, and the interface made it usable by anyone.
What is test-time compute?
Not quite. Reasoning models trade latency and cost for accuracy on hard problems.
Two labs release models in the same month. How should a company decide which to use?
Not quite. Models differ in quality, speed and price, so your own examples decide.
You got 0 of 3 right.
Go deeper
- ArticleIntroducing ChatGPTArticle
OpenAI · The announcement post from November 2022.
- Paper
OpenAI · The InstructGPT paper.
- PaperLlama 2Paper
Meta · A detailed open description of how a chat model is trained.
- ArticleLearning to reason with LLMsArticle
OpenAI · The o1 announcement of test-time reasoning.
- PaperDeepSeek-R1Paper
DeepSeek · An open recipe for reasoning via reinforcement learning.
- ArticleState of AIArticle
Air Street Capital · Yearly free report that summarizes research, industry and policy.
Spotted a mistake or something unclear? Report it on GitHub.