A second and a half to answer: inside a real-time voice agent

Run speech recognition, a language model and speech synthesis one after another, each waiting for the last, and a caller waits three to five seconds for every reply. On the phone, that is unacceptable. Over the past months we have built several telephone AI agents, and each raised the same three questions: how long does it take from the caller’s last word to the answer, what happens when they interrupt, and how well does the system cope with Hungarian. This is the architecture we arrived at, pitfalls included.

Stream everything

The classic setup has three stages: speech recognition (STT), a language model (LLM) and speech synthesis (TTS). To avoid the three-to-five-second wait, all three run in streaming mode:

• the STT produces partial transcripts as the audio arrives, without waiting for the end of the sentence;
• the LLM streams token by token, and we pass the text on at the first sentence-ending punctuation mark;
• the TTS synthesises fragment by fragment, so the first audio chunk goes out while the model is still generating the rest.

The result is an asynchronous pipeline in which every stage has its own queue and its own cancellation signal. In Node.js this comes naturally: streams, async iterators, AbortController.

Where the silence ends

Most of the delay does not come from the models. It comes from a decision: when has the caller finished speaking? A fixed silence threshold, say 700 ms, is simple and wrong in two ways. After a short pause it cuts the caller off; set it long and the system feels sluggish.

We combine three signals: VAD (voice activity detection) on the raw audio, the STT’s own end-of-speech signal, and a simple linguistic heuristic that estimates from the partial transcript whether the sentence is syntactically complete. This is harder in Hungarian than in English because word order is freer, though question words and suffixes help a lot. A state machine weighs the three signals, and the threshold adapts: if the caller speaks slowly, with pauses, the system grows more patient.

The budget, in milliseconds

What the caller experiences is the gap between their last syllable and the first sound they hear back. We split it roughly like this, with indicative magnitudes:

• endpointing decision: 200-400 ms
• STT finalisation: 100-300 ms
• LLM first token: 300-600 ms
• TTS first audio chunk: 150-300 ms
• network and jitter buffer: 100-200 ms

That comes to about one to one and a half seconds, which already feels natural. We measure every stage separately, as spans, per call. With only the total, we would not know where things degraded. One trick works well: the filler response. If the LLM’s first token is late, a short, context-dependent acknowledgement (“Right, let me check.”) can go out first. It buys time and spares the caller a dead line.

Four sample rates

Telephony is the least glamorous part and takes the most work. From the PSTN, RTP arrives over a SIP trunk, typically G.711 (8 kHz, alaw/ulaw) or, with luck, Opus. From the browser comes WebRTC, Opus at 48 kHz. STT models mostly expect 16 kHz PCM, and TTS often produces 24 kHz. So the media layer resamples in every direction. That belongs in one place, under an unambiguous format contract; otherwise the audio turns chipmunk-like or plays at half speed.

RTP packets arrive unevenly, so the receiving side runs an adaptive jitter buffer in the 20-60 ms range. The sending side needs pacing: TTS produces audio faster than the line can carry it, so we send it in 20 ms frames on a clock rather than as it comes. Skip that and the jitter buffer at the far end drops packets.

When the caller cuts in

An interruption is called barge-in. When it happens, TTS playback stops immediately, the outgoing buffer is flushed, LLM generation is cancelled and the new input is handled as a fresh turn. The hard part is the truncated answer: the conversation history stores what was actually spoken, not the full text the model generated.

Every external call has a timeout. If the STT provider does not respond in time, we switch to a secondary provider; for the LLM we fall back to a smaller model. Retries are allowed only for idempotent steps, so a TTS clip that has partly played is never repeated. If everything fails, the agent apologises with a pre-recorded audio file and promises a callback. Worse than normal operation, much better than silence.

Dates with suffixes

Hungarian is agglutinative, so STT models are less sure of rarer word forms than they are in English. What we found:

• recognition of proper nouns and domain terms (street names, product codes) improves when the STT gets a glossary or a prompt hint;
• numbers, dates and times cannot go to the TTS raw: instead of “2024. 12. 10.” we pass “kétezer-huszonnégy december tizedike”, complete with the right suffixes (“tizedikén”, “háromkor”);
• phone numbers and identifiers are read out in groups, with pauses, then confirmed back;
• Hungarian TTS voices vary widely in prosody, so we compare several providers on the same set of sentences.

The main lesson: a voice agent is a real-time media system, and the LLM is only one stage of it. Latency, interruptibility and linguistic normalisation take at least as much engineering as the prompt.