Voice agents: architecture, latency and turn-taking
A voice agent listens, thinks and speaks in real time. The common architecture is a pipeline: speech-to-text, an LLM (with tools), and text-to-speech, streamed so each stage starts before the previous one finishes, with voice activity detection deciding when the user has stopped speaking. Newer speech-to-speech models skip the text steps. The hard parts are latency (people expect a reply in well under a second), turn-taking, and handling interruptions, which is why good voice agents need streaming, WebSockets, fast models and careful engineering at every stage.
Why voice is hard
In a text chat, waiting two or three seconds for a reply feels normal. In a conversation, a two-second silence feels broken.
Most voice bots today feel unnatural for that reason, and because they work like a walkie-talkie: you speak, you wait, the bot speaks, the bot waits. Real conversations aren't like that. People interrupt, talk over each other, say "hmm" while thinking, and change direction mid-sentence. A teacher answering a student's question in the middle of a sentence stops, answers, and goes back to what they were saying. That is what a voice agent has to handle.
Pipeline architecture
- Audio in: the user's microphone audio is streamed to your server, usually over a WebSocket.
- Voice activity detection (VAD): a small model detects when the user is speaking and when they've stopped.
- Speech-to-text (STT): audio is transcribed, ideally while the user is still speaking (streaming transcription).
- LLM: the transcript goes to the model, with the conversation history and tools, and the response is streamed token by token.
- Text-to-speech (TTS): the response is turned into audio, starting with the first sentence while the rest is still being generated.
- Audio out: the audio is streamed back to the user.
The pipeline is flexible: you can choose the best model for each stage, use any LLM with tools, log the text at each step, and swap parts independently.
Speech-to-speech models
Speech-to-speech models take audio in and produce audio out directly, without separate transcription and synthesis steps. They can be faster, and they keep tone, emotion and timing that text loses. They give you less control over each stage and can be harder to combine with complex tools and logging, so many production systems still use pipelines, or mix both.
Latency budget
To feel natural, the time from the user stopping to the agent starting to speak should be well under a second. Every stage takes part of that budget:
- Detecting the end of speech.
- Finishing the transcription.
- The LLM's time to first token.
- TTS producing the first audio.
- Network time in both directions.
Ways to stay within budget:
- Stream everything: transcribe while the user speaks, start TTS on the first sentence of the response.
- Use fast models for each stage, and a smaller LLM where it's enough.
- Keep the LLM's prompt short and use prompt caching.
- Run servers close to users, or run models locally.
- Say something short ("Let me check that") while a slow tool runs.
Turn-taking and interruptions
Deciding when the user has finished is harder than it sounds. A pause might be the end of a turn or just thinking. Waiting too long feels slow, and cutting in too early interrupts the user. VAD settings, end-of-turn detection models and the content of what was said all help.
Interruptions, often called barge-in, need the agent to listen while it speaks. When the user starts talking, the agent should stop its audio immediately, cancel or pause the response being generated, and process what the user said. That requires full duplex communication, where audio flows in both directions at the same time, and asynchronous code that can handle both streams at once. See async Python.
Tools in voice agents
Voice agents use tools like any agent: looking up an order, booking an appointment. But tools add latency, so keep them fast, run them in parallel when possible, and design what the agent says while waiting. Confirm important details out loud, like dates, amounts and names, because speech recognition errors are common.
Other challenges
- Noise, accents and poor phone lines affect transcription.
- Numbers, names and email addresses are hard to recognise and to say.
- Phone systems add their own latency and audio quality limits.
- Voices that sound human raise questions of disclosure: tell users they are talking to an AI.
From the floor
The third artifact in AI Engineer HQ was Echo Negotiator: a real-time, low-latency, full duplex audio streaming server for voice-to-voice AI that feels closer to talking to a person. The problem statement was that current voice bots feel unnatural, respond slowly and can't handle interruptions, which makes them unsuitable for high-speed or high-stakes conversations like negotiations, sales calls, coaching and interviews. The goals were full duplex communication, natural handling of interruptions with voice activity detection, and responses in under 500 milliseconds.
We built it to run completely offline, with no external APIs: FastAPI with WebSockets for the streaming server, faster-whisper for speech-to-text, Silero for voice activity detection so the AI can stop when the user interrupts, and a local LLM through Ollama.