Estimating value and rolling AI out
A project succeeds when it saves more than it costs and people actually use it. Both can be estimated before you build.
Words on this page (6)
- ROI
- : Return on investment: what a project gains compared with what it costs.
- shadow mode
- : Running a new system beside people on real cases without acting on its output, so you can compare it with their decisions at no risk.
- baseline
- : A simple method, such as "same as last week", that any model must beat to be worth its cost.
- data drift
- : Real-world inputs slowly changing from what a system was tested on, so quality falls with no code change.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
- benchmark
- : A fixed test with a score used to compare models.
What it is
(return on investment) compares what a project gains with what it costs. For AI the costs include model calls, building time, review time by people, monitoring and upkeep. The gains are time saved, errors avoided or revenue added. Teams that skip the cost side often find the gain disappears.
A rollout goes in stages: test on past cases, run beside the people doing the job (), release to a small group, then widen. At each stage you compare against the .
A worked example (invented numbers)
A support team gets 10,000 tickets a month. Each takes 4 minutes at a cost of $30 an hour, so $2 a ticket, or $20,000 a month. An assistant drafts replies. It handles 30 percent of tickets with no edits, which saves $6,000. Model calls cost $0.05 a ticket, so $500 for all 10,000. A person spot-checks 5 percent of all drafts, which is 500 tickets, and a check takes as long as handling the ticket did, so that is 500 × $2 = $1,000. Net saving: 6,000 − 500 − 1,000 = $4,500 a month. If building cost $30,000, payback is about seven months. If it handled only 10 percent, the saving would be $2,000 − 500 − 1,000 = $500 and payback would take five years.
After launch
Watch quality, cost and usage. means real inputs slowly change from what the system was tested on, so scores slide without any code change. Keep a test set, rerun it monthly, and keep an easy way for users to report a bad answer. A company answers for what its say: in 2024 a Canadian tribunal ordered Air Canada to pay a customer who relied on wrong refund advice from its website chatbot.
Find the break-even before you build
Using the invented numbers above, fixed costs are $500 for calls plus $1,000 for checking, $1,500 a month. Each percent of tickets handled saves $200 (1 percent of $20,000). Break-even is 1,500 / 200 = 7.5 percent. Below that, the assistant loses money every month. The estimate is only as good as the 30 percent figure, so measure it before launch.
Putting error bars on the shadow-mode result
Run the assistant in shadow mode on 200 past tickets and see 60 drafts need no edits, which is 30 percent. With only 200 tickets the result has a margin of about 6 points either way (roughly 2 × sqrt(0.3 × 0.7 / 200), the sampling-error formula from the lesson). The true rate could be 24 or 36 percent. At 24 percent the net saving is $3,300 a month and payback takes 9.1 months. At 36 percent it is $5,700 and payback takes 5.3 months. The monthly result is above break-even across the whole range. Whether a 5 to 9 month payback is acceptable is a separate decision. A range that crosses 7.5 percent would mean testing more tickets.
What goes wrong: costs the estimate misses
The example counts the 30 percent of tickets drafted well. The other drafts cost time too. Suppose 20 percent of all drafts are wrong, and each takes an extra minute to read and discard. That is 10,000 × 0.2 × 1 minute = 2,000 minutes, about $1,000 a month, which cuts the net saving to $3,500. The symptom is that drafts get accepted but tickets handled per hour barely moves. Confirm by measuring total handling time per ticket before and after, not how many drafts were accepted.
Common mistake
Declaring victory after a demo. Adoption, review time and drift decide the real return.
A mid-sized insurer pilots a claims assistant and the demo is a hit. In shadow mode, though, its suggestions match the human decision on only 70 percent of claims, and the claims staff spend more time correcting drafts than writing them. The insurer narrows the tool to one claim type where it matches 95 percent, and that is where it pays off.
Check yourself
Which cost is most often forgotten in an AI project estimate?
Not quite. Review and monitoring time can outweigh model call costs.
What is shadow mode?
Not quite. It lets you compare AI output to real decisions with no risk.
Scores drop six months after launch with no code change. What is a likely cause?
Not quite. Real inputs move away from the data the system was tested on.
Fixed costs are $1,500 a month and each percent of tickets handled saves $200. When does the project break even?
Not quite. 1,500 / 200 = 7.5 percent. Below that it loses money each month.
You got 0 of 4 right.
Go deeper
- ArticleRules of Machine LearningArticle
Google · Launch, monitor and iterate advice from experience.
- Paper
Sculley et al. · Why running a model costs more than building it.
- Article
Eugene Yan · Evals, guardrails and feedback loops in production.
Spotted a mistake or something unclear? Report it on GitHub.