A voice agent is four systems in a loop. Audio arrives, something transcribes it, something decides what to say, something turns that into speech, and the audio goes back. The components are commodities. The loop is not.
| Layer | Job | Typical latency |
|---|---|---|
| Telephony | Carry audio to and from the phone network | 20–100ms |
| Speech to text | Transcribe the caller, streaming | 100–300ms |
| Language model | Decide what to say | 300–800ms |
| Text to speech | Synthesise the reply | 80–200ms |
The number that decides whether it sounds human
In natural conversation the gap between one person stopping and the next starting is around 200ms. Push past roughly 800ms and callers start doing what people do on a bad line — repeating themselves, talking over the agent, or assuming the call dropped.
Add the layers up and the budget is tight before anything goes wrong. That is why the category obsesses over latency, and why platforms advertise time-to-first-token rather than mouth-to-ear: the first number is flattering and the second is what the caller experiences.
The actually hard problem
Not any single layer. It is knowing when the human has stopped talking.
Wait too long and the agent feels slow and stupid. Cut in too early and it interrupts someone mid-sentence, which is worse, an agent that talks over you reads as broken in a way a slow one does not. There is no correct threshold, because the right answer depends on the caller, the phone, the background noise and what was just asked.
A pause after "my account number is..." means keep waiting. The same pause after "yes" means reply now. Systems that handle this well are using semantic cues, not just silence duration, and this is where the difference between platforms actually shows up.
Barge-in
When the caller starts talking while the agent is speaking, the agent must stop immediately, discard what it was going to say, and process the interruption. Getting this wrong produces the single most frustrating failure in the category: an agent that keeps reading its script while a human is trying to correct it.
Speech to speech, and the trade
Newer models collapse transcription, reasoning and synthesis into one hop, removing conversions and cutting latency meaningfully.
The trade is control. You cannot swap a cheaper component when one ships, you do not get a clean transcript for free, and debugging is harder because there is no intermediate state to inspect. For compliance-heavy deployments that need verbatim records, the assembled pipeline is still the safer choice.
What a minute of all this costs, with each component priced from its vendor's own published rate, is on the cost model.