Choosing a model

What it isCopy Anchor Link

Models differ on quality, speed, cost, context length, tool use, modality and where they can run. Vendors usually ship a family: a large model for hard reasoning and planning, a mid-size model for most work, and a small fast one for high-volume, simple steps. Open-weight models trade peak quality for control, privacy and cost at scale.

Why it mattersCopy Anchor Link

Using the biggest model everywhere wastes money and time. Using the smallest everywhere fails on hard steps. A common pattern is to route by step: the large model plans and judges, the mid-size model does the work, the small model scans and classifies. You can only do that safely if you have a small set of real test cases to compare models on.

Public benchmarks are useful for shortlisting. They are weak predictors of how a model does on your data, and some are contaminated or saturated. Build ten to fifty examples from your own task and compare.

A worked example: a simple comparisonCopy Anchor Link

Take 50 real support emails with the correct category. Run model A (large), B (mid), C (small). Example results (invented for illustration): A 96 percent at $0.020 per email, B 94 percent at $0.004, C 88 percent at $0.0005. If a 2-point difference is not worth five times the price, pick B. For a medical triage tool, you might pick A. The decision is about your costs of being wrong.

Open or closed?Copy Anchor Link

A closed model is used through the lab's API. An open-weight model can be downloaded and run by you, which helps with privacy and cost at scale. The open-weight models lesson covers the trade-offs, and the Model atlas lists both kinds.

How a fair comparison runsCopy Anchor Link

1. Write down the task and what a wrong answer costs you.
2. Collect 20 real inputs and the answer you want for each.
3. Run every candidate model on all 20 with the same prompt.
4. Score each output as right or wrong, and note the cost and the wait for each call.
5. Pick the cheapest model that clears the quality bar you set in step 1.

Invented numbers for illustration: your bar is 90 percent, so at least 18 of 20. The small model gets 17 right (85 percent) and fails the bar. The mid-size model gets 19 (95 percent) and passes. You choose the mid-size model without ever looking at a leaderboard.

A second example: the big model is the wrong pickCopy Anchor Link

You must tag 10,000 product reviews as positive or negative every night. The task is easy, so a small model may score almost the same as a large one. At an invented $0.0005 per review, the small model costs 10,000 x 0.0005 = $5 a night. At an invented $0.02, the large one costs $200 a night. Unless your own test shows a real quality gap, the extra $195 buys nothing.

What goes wrongCopy Anchor Link

A test set of 10 examples is too small. One flipped answer moves a score by 10 points, so rankings change from run to run. To check, run the same models twice. If the order changes, add more examples.

A test set of only clean, easy inputs makes every model look good. The symptom is a model that scores well in your test and fails in production. To confirm, pull 20 fresh inputs from real use and score them too.

Try it yourselfCopy Anchor Link

Write 10 questions from a subject you know well, with the correct answers beside them. Ask two different chat apps or models each one and mark them right or wrong. Expect both to get the easy ones right and to differ on the hardest two or three. That gap is what a larger test set would measure.

Common mistakeCopy Anchor Link

Choosing by popularity or by one benchmark number. Always test on your own data.