Skip to content
AI Engineering
Production

Cost, latency and caching

Token cost and wait time are product features. A few levers cut both by a large factor.

Chapter 17: ProductionLesson 2 of 6Level: CoreAbout 12 min
Words on this page (7)
token
: A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
agent
: A model run in a loop that chooses its own tools and steps until the task is done.
prompt
: The input text you give a model, including instructions, examples and data.
tool
: A function the model can ask your program to run, described by a name, description and argument schema.
batch
: A group of examples processed together in one step.
latency
: The time you wait for a response.
API
: Application programming interface: a way for one program to call another, usually over HTTP.

The math

Cost per call is input times the input price plus output tokens times the output price, and output tokens usually cost several times more than input. Example: input costs $3 per million tokens and output $15 per million. A call with 20,000 input tokens costs 20,000 × 3 / 1,000,000 = $0.06. Its 1,000 output tokens cost 1,000 × 15 / 1,000,000 = $0.015. The total is $0.075. An making 30 such calls costs about $2.25 per task. Context re-sent each step is the main cost.

cost=nin pin+nout pout106\text{cost} = \frac{n_{\text{in}} \, p_{\text{in}} + n_{\text{out}} \, p_{\text{out}}}{10^6}

Levers

caching: providers reuse the computed state of an unchanged prompt prefix, charging much less for cached input and responding faster. Keep stable content first. Model routing: use a small model for easy steps. Shorter context: trim results. APIs: lower prices for non-urgent work. Streaming: improves perceived speed. Output limits: cap length.

A worked example: where the money goes

A support agent averages 6 model calls per ticket. Each call sends about 8,000 tokens in and 300 out. At $3 and $15 per million: input 6 × 8,000 = 48,000 tokens = $0.144; output 6 × 300 = 1,800 tokens = $0.027. About $0.17 per ticket, and 84 percent of it is input. Caching shrinks that biggest part. Suppose 6,000 of the 8,000 input tokens in each call are the same unchanged prefix, and cached reads cost a tenth of the normal input price. Each call then costs 6,000 × $0.30 / 1,000,000 + 2,000 × $3 / 1,000,000 = $0.0078 for input. Over 6 calls that is $0.047, so a ticket costs about $0.074, roughly 57 percent less.

Latency is not cost

and cost depend on different parts of the call. Cost is dominated by input tokens in this example. Waiting is dominated by output, because the model writes one token at a time.

Invented numbers: reading 20,000 input tokens takes about 2 seconds before the first word appears. Writing 1,000 output tokens at 40 tokens per second takes 25 seconds. So a long answer is slow even though input was the bigger bill. To cut waiting, shorten the answer, stream it so words appear early, or use a smaller model.

What goes wrong: a cache that never hits

Caching works only if the start of the prompt is identical from call to call. Someone puts the current time or the user's name on the first line. Now the prefix differs on every call and nothing is reused. The symptom is that the bill does not move after you switch caching on. Confirm it by reading the usage numbers in the response, which most providers report as cached tokens. If they stay at zero, move everything that changes to the end of the prompt.

When caching or routing is the wrong tool

A one-off call with a unique 500-token prompt has nothing to reuse, and providers often require a minimum prefix length before caching applies at all. Routing has its own condition. It saves money only if you can tell when the small model failed, for example with a format check or a test. If failures are invisible, bad answers pass through quietly.

Common mistake

Optimizing output length when the cost is mostly input.

Check yourself

A call has 20,000 input tokens at $3 per million. What is the input cost?

Where should stable instructions go to benefit from caching?

Your agent costs $0.17 per ticket and 84 percent is input. Which change cuts the most?

Go deeper

Spotted a mistake or something unclear? Report it on GitHub.

Advanced RAG techniques

18 min · Stretch

Deploying an AI app

15 min · Core

On this page

The mathLeversA worked example: where the money goesLatency is not costWhat goes wrong: a cache that never hitsWhen caching or routing is the wrong toolCommon mistake