Speech and multimodal models

SpeechCopy Anchor Link

Speech recognition used to need separate acoustic and language models. Whisper (OpenAI, 2022) trained a transformer on about 680,000 hours of audio from the web and became a robust open speech-to-text model across many languages. Text-to-speech now sounds natural, and real-time voice conversation is a product feature.

MultimodalCopy Anchor Link

A multimodal model maps different kinds of input into a shared representation. Images are cut into patches and turned into vectors the same way tokens are, then processed by the same transformer. This lets a model read a chart, a screenshot or a handwritten note, and describe or reason about it. CLIP (2021) trained image and text encoders so matching pairs land close together, which powers image search and guides image generation.

Why it matters for buildersCopy Anchor Link

Screenshots and documents can go straight in without a separate OCR step, and voice becomes an interface. Cost and latency go up with image and audio input, so check token usage for each type.

A worked example: image as tokensCopy Anchor Link

A 224 by 224 pixel image cut into 16 by 16 pixel patches gives 14 × 14 = 196 patches. Each patch becomes a vector, like a word. A transformer then reads those 196 vectors with attention, just like words in a sentence. Add the text "What color is the car?" and the model can look at the relevant patches while it answers.

Costs to plan forCopy Anchor Link

Images and audio use many more tokens than the text that describes them. A single high-resolution photo can cost as much as a few thousand words. Downscale before sending, and send audio in short segments.

How speech becomes textCopy Anchor Link

Sound is a stream of numbers, for example 16,000 a second (a common rate). The model first turns a short clip into a spectrogram, a picture showing which pitches are loud at each moment. It then reads that picture with a transformer, much as it would read an image, and writes out text tokens one at a time. Many modern speech models, including Whisper, work in roughly this way. One second of audio at that rate is 16,000 numbers, which is why audio is heavy to process.

A second example: invoice, image or text?Copy Anchor Link

You need to pull the total from a scanned invoice. Option one: run OCR (software that turns a picture of text into text) and send only the text. It is cheap, but it loses layout, so a total in a table can be confused with a subtotal. Option two: send the image directly. The model sees the layout and stamps, but you pay for more tokens. Option two is not needed for a clean typed contract where plain text is enough. Try both on ten real documents and compare.

When it goes wrongCopy Anchor Link

With speech, the symptom is confident but wrong words, mostly names, jargon and accents, and sometimes text invented for silent or noisy stretches. With images, the symptom is a description of details that are not there, or small print that became unreadable after downscaling. To confirm, ask the model to quote the exact text it relied on and compare it with the source. For speech, measure the word error rate: wrong words divided by total words, checked against a transcript you wrote by hand.

Try it yourself: measure your own error rateCopy Anchor Link

Record a 30 second clip that includes two names and some background noise. Run it through any speech-to-text tool. Write down what you actually said and count the wrong, missing or extra words. With invented numbers, 6 wrong out of 100 words is 6 ÷ 100 = 6 percent. Names and jargon are often the biggest source of errors, so those are the words to double-check.

Common mistakeCopy Anchor Link

Trusting speech recognition on names, jargon and noisy audio. Always check those parts.