Speech recognition and voice assistants
Voice systems turn sound into text, think about it, and turn text back into sound. Each step adds delay, and delay decides whether it feels like a conversation.
Words on this page (4)
- speech recognition
- : Turning spoken audio into text.
- text-to-speech
- : Turning written text into spoken audio.
- word error rate
- : Speech recognition accuracy: substitutions, deletions and insertions divided by the number of true words.
- tool
- : A function the model can ask your program to run, described by a name, description and argument schema.
What it is
(ASR, or speech-to-text) converts audio to words. (TTS) does the reverse. A voice assistant chains the two around a language model: listen, transcribe, think, write a reply, speak it. Some newer models take audio in and give audio out directly, which cuts steps and keeps tone.
Open models such as Whisper made good transcription cheap for many languages. Accuracy is measured with (WER): the number of substitutions, deletions and insertions divided by the number of words in the true transcript.
A worked example: word error rate
The speaker says "turn the lights off in the kitchen" (7 words). The system writes "turn the light off in kitchen". One word is changed (lights to light), one is missing (the). That is 1 substitution + 1 deletion = 2 errors over 7 words, a WER of 2 / 7, about 29 percent.
A worked example: the delay budget
Transcribing takes 0.3 seconds, the model 0.6 and speech 0.3. Total 1.2 seconds before the user hears anything. People notice gaps in talk quickly, so teams stream everything: start speaking the first sentence while the model writes the second, and stop the voice at once if the user interrupts.
Where it is used
Meeting notes and captions, call-centre assistants, voice control in cars, language practice, and that read text aloud for people with low vision. Accents, background noise, and names are the usual failures, so test with the speakers you will actually serve.
What WER hides
WER counts every word equally. "Take fifteen milligrams twice a day" heard as "take fifty milligrams twice a day" has one error in six words, a WER of 17 percent. The same score applies if the system just drops "a". The first is dangerous and the second is harmless. Report WER on the words that matter, such as names, numbers and drug names, as well as overall.
Averages hide groups
Invented numbers: 80 speakers have a WER of 6 percent and 20 speakers with a regional accent have 18 percent. Overall: 0.8 × 6 + 0.2 × 18 = 8.4 percent. That looks fine, yet one in five users has a poor experience. To confirm, split your test audio by accent, microphone and noise, and compute WER for each group.
The wait before the delay budget
The 1.2 second budget starts after the user stops talking, but the system must first decide they have stopped. It waits for a pause, typically a fraction of a second (say 0.7 seconds). Add that and the total becomes 1.9 seconds. A shorter wait makes the assistant talk over people who pause to think. A longer wait feels slow. Streaming helps the other end: if speech starts when the model has written its first sentence, say 0.2 seconds into its 0.6, the first sound arrives 0.3 + 0.2 + 0.1 = 0.6 seconds after the pause ends, or 1.3 seconds after the user stopped talking. The 0.1 second speaking start-up is invented.
Common mistake
Testing with clean studio audio. Real users talk in cars and kitchens, and the error rate there can be several times higher.
A clinic records doctor visits so notes write themselves. In the quiet demo the transcripts are nearly perfect. In real rooms, with a printer humming and a patient mumbling a drug name, it confuses two medicines that sound alike. The clinic adds a drug-name list and makes the doctor confirm each medication before the note is saved.
Check yourself
What does word error rate measure?
Not quite. WER counts substitutions, deletions and insertions against the true words.
Why do voice assistants stream their output?
Not quite. Streaming overlaps the steps, which cuts the wait the user feels.
A demo works in a quiet room. What should you do next?
Not quite. Real conditions raise error rates, so test in them before launch.
You got 0 of 3 right.
Go deeper
- Repo
OpenAI · Open speech recognition model for many languages.
- Course
Hugging Face · Free course on speech and audio models.
- Paper
OpenAI · The Whisper paper: how a speech model is trained and how its error rate is measured.
Spotted a mistake or something unclear? Report it on GitHub.