API features: streaming, limits and batches

StreamingCopy Anchor Link

The server sends tokens as they are generated, so the user sees the answer start in a fraction of a second instead of waiting for the whole thing. It is built on server-sent events. Handle partial tool calls and errors that arrive mid-stream.

LimitsCopy Anchor Link

Each model has a maximum context size and a maximum output size. Counting tokens before a call prevents errors. Rate limits cap requests and tokens per minute: handle them with retries, exponential backoff and queues. Set timeouts, since a long generation can run for minutes.

Batch and cachingCopy Anchor Link

Batch APIs run many non-urgent requests at a discount, with results returned within hours. Prompt caching charges less for an unchanged prefix. Use both for bulk work such as classifying a large dataset.

Two API stylesCopy Anchor Link

Chat-style APIs take a list of messages and return the next message. Newer response-style APIs add built-in tools, state kept on the server and richer output items. Many providers and routers, such as OpenRouter, expose an OpenAI-compatible endpoint so the same client code can call many models by changing the base URL and model name.

A worked example: handling limitsCopy Anchor Link

Your app sends 300 requests a minute and the provider allows 200. The extra 100 get a 429 error. A good client queues them and retries after 1, 2, then 4 seconds with a little random jitter, so a hundred clients do not retry in lockstep. For a non-urgent job, such as labeling 100,000 reviews overnight, use a batch API: results arrive within hours at a lower price.

Streaming gotchasCopy Anchor Link

Text arrives in small pieces, so append, do not replace. The connection can drop halfway: keep what you have and offer a retry. Let the user press Stop.

How streaming worksCopy Anchor Link

The server keeps the connection open and sends small events as tokens are produced. Your code reads each event and appends its text to the screen. Usage counts arrive in the stream's closing events. Some providers need an option switched on to send them.

Two times matter. Time to first token is the wait before anything appears. Total time is how long the whole answer takes. Streaming improves the first and leaves the second unchanged.

Trade-offs and when not to streamCopy Anchor Link

Skip streaming when code, not a person, consumes the answer. A JSON reply cannot be parsed until it is complete. Skip it when you must check the full answer before anyone sees it, such as a safety filter.

Batch jobs give a discount but you wait, so they suit work nobody is waiting on. Prompt caching only helps when the start of the prompt is identical between calls. Anything that changes early, such as a timestamp at the top of the system prompt, breaks the match.

What goes wrongCopy Anchor Link

Errors arrive in waves. Many clients hit the limit together and retry at the same moment. Add random jitter, then log retry times to see the waves spread out.

A worker freezes. No timeout (a maximum wait you set) was configured. Look for requests that have been open for minutes.

Costs do not drop after you enable caching. Check the usage fields your provider returns for cached tokens, and look at what changes at the top of your prompt.

An answer ends halfway. The connection dropped. Keep the partial text and offer a retry button.

Try it yourselfCopy Anchor Link

Stream a request that asks for a long answer. Print the time when the first piece arrives and the time when the stream ends. Typically the first comes within a second or two and the second several seconds later. Then run it without streaming and confirm the total time is about the same.

Common mistakeCopy Anchor Link

Setting no timeout. A stuck request that never returns can freeze a worker forever.

The codeCopy Anchor Link

Streaming prints words as they arrive instead of waiting for the full answer.

with client.messages.stream(
    model="claude-sonnet-5-5",
    max_tokens=300,
    messages=[{"role": "user", "content": "Explain tokens in two sentences."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)