Images, audio and video APIs
Models can read images, transcribe speech and generate pictures and voices. Each has its own API shape and cost.
Words on this page (12)
- API
- : Application programming interface: a way for one program to call another, usually over HTTP.
- token
- : A piece of text the model reads and writes, usually a word or part of a word. About 4 characters of English on average.
- structured output
- : Model output that follows a schema, such as valid JSON with fixed fields.
- prompt
- : The input text you give a model, including instructions, examples and data.
- diffusion model
- : An image generator trained to remove noise step by step. Generation starts from pure noise.
- text-to-speech
- : Turning written text into spoken audio.
- latency
- : The time you wait for a response.
- chunk
- : A small piece of a document, sized for indexing and retrieval.
- JSON
- : A text format for structured data: keys and values in braces, like {"name": "Ada"}.
- vector
- : An ordered list of numbers. Embeddings and model activations are vectors.
- OCR
- : Optical character recognition: turning an image of text into real text.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
Understanding images and video
Send an image as part of a message, as a URL or encoded data, along with a question. Models can describe a photo, read a chart, extract text from a screenshot or compare two designs. Video is usually handled as sampled frames plus audio. Image are priced by size, so downscale large images. Ask for when you need data.
Generating images
Image APIs take a text and return an image. models are controlled with the prompt, size, seed and sometimes a reference image for edits. Describe subject, style, composition and lighting. Always check licenses and provenance rules for what you publish.
Audio
Speech-to-text turns audio into text, with timestamps and speaker labels in some models. Whisper is the open standard. turns text into natural voice. Realtime voice APIs combine both with low for conversation. Audio is often sent in , so plan for streaming.
Cost and quality
Test with your own images and recordings. Check accuracy on small text, noisy audio and accents, which are the usual weak points.
A worked example: three requests
Describe: send a product photo and ask for a one-sentence alt text. Extract: send a receipt and ask for total, date and vendor as . Compare: send two screenshots and ask what changed. Each is one request with an image plus an instruction. Ask for structured output when the result feeds code.
How an image becomes tokens
The provider cuts the image into patches and turns each patch into that the model reads like tokens. A larger image means more patches and a higher bill. The exact rule differs by provider.
Invented rule for illustration: one token per 1,000 pixels. A 1000 by 1000 image is 1,000,000 pixels, so 1,000 tokens. Halve each side to 500 by 500 and you get 250,000 pixels, so 250 tokens. Halving each side cuts the cost to a quarter.
A second example: where vision fails
Counting many small objects, reading tiny or blurry text, handwriting and judging exact positions are common weak spots. Take a receipt photographed at an angle in dim light. The model may return a total that looks right but is wrong. For printed text at scale, a dedicated can be more reliable and cheaper. Use a vision model when you also need understanding, such as what the receipt is for.
What goes wrong
Plausible but wrong numbers. Symptom: totals that look fine and are off by a digit. To confirm, check 20 samples by hand. In code, test that the line items add up to the total.
Audio with noise, overlapping speakers or strong accents produces mistakes and sometimes invented words. Test with recordings like the real ones, not clean demos.
Private data leaves your control. Photos can include faces, addresses or documents in the background. Check where the provider stores them.
Some image models still misspell text inside the picture. Check any text in the output, and add it yourself afterward if it matters.
Try it yourself
Photograph one printed paragraph three times: sharp, from far away, and slightly blurred. Send each to a vision model and ask it to copy the text. Compare the results. Expect it to read the sharp one well and to start making errors as the quality drops. Note the point where it breaks.
Common mistake
Sending private photos or documents to a provider you have not vetted.
A small online shop owner photographs a handwritten price list and sends it to a vision model with the question "turn this into a table". Thirty seconds later the table is in a spreadsheet. A few years ago that took an afternoon of typing.
Check yourself
How is video usually given to a model?
Not quite. Frames are sampled at intervals and sent together with the transcript.
Why downscale large images before sending?
Not quite. Smaller images cost less and are often enough for the task.
You must extract a total, date and vendor from a receipt photo. What should you ask for?
Not quite. Structured output makes the data easy to check and use in code.
You got 0 of 3 right.
Go deeper
- Docs
Anthropic · Sending images and what works well.
- Repo
OpenAI · Open speech recognition you can run yourself.
- Docs
OpenAI · Image input and generation.
- Article
Jay Alammar · How text becomes an image.
- Docs
OpenAI · Sending images to a model and reading the answer.
- CourseMultimodal modelsCourse
Hugging Face · How vision-language models are built.
Spotted a mistake or something unclear? Report it on GitHub.