How to Build an AI Voice Agent

A production AI voice agent is four components running in a real-time loop – speech-to-text, an LLM with tool access, text-to-speech, and a telephony layer – held to a hard budget of under 800 milliseconds from the moment the caller stops speaking to the moment your agent starts. Expect roughly $50,000–150,000 and 30–45 days for a focused voice MVP that handles one call type end to end, and roughly $0.05–0.20 per minute in runtime cost once it is live.

The latency target is not an arbitrary engineering preference. According to Stivers et al., PNAS, 2009, turn transitions in human conversation across ten typologically diverse languages are unimodal, with the highest number of transitions occurring between 0 and 200 milliseconds. Every caller arrives with that expectation wired in. You will not hit 200ms with a language model in the loop, but the further past it you drift, the more the caller stops treating the exchange as a conversation and starts treating it as a broken phone system.

Which stack should you build: cascading or speech-to-speech?

This is the first real decision, and it constrains everything after it. A cascading pipeline transcribes audio to text, runs your LLM on the text, and synthesizes the reply back to audio. A speech-to-speech model takes audio in and emits audio out, keeping tone, timing, and prosody inside one model.

Most production business deployments in 2026 still ship cascading, and the reason is not nostalgia. The text layer between transcription and synthesis is where you attach every guardrail, every tool call, every eval, and every compliance log. Giving that up buys you a few hundred milliseconds and a warmer voice, which matters enormously for a companion product and very little for a clinic confirming an appointment. A reasonable default: build cascading, prove the business logic, then pilot speech-to-speech on the specific flows where naturalness is the product.

Where do the milliseconds actually go?

Teams miss latency targets because they optimize the wrong stage. Here is where a typical turn spends its budget, from the caller's last syllable to your agent's first:

• End-of-speech detection: 100–300ms. Voice activity detection waiting to be confident the caller is finished. This is the single largest tunable cost and the most dangerous to cut – shave it too far and the agent interrupts people mid-thought.
• Final transcript: 100–200ms. Streaming transcription should already have partial text; you are only paying for the finalization pass.
• Time to first LLM token: 200–500ms. Driven by model choice, prompt length, and whether the turn triggers a tool call. A tool call that hits your own database adds its own round trip on top.
• Time to first audio: 100–300ms. Streaming TTS should begin synthesizing on the first sentence boundary, not the complete response.
• Network and jitter buffer: 50–150ms. Largely fixed. Co-locate your agent service in the same region as your telephony provider and stop thinking about it.

Only two of those five are worth serious engineering: end-of-speech detection and time to first token. The highest-leverage move is streaming everything – partial transcripts into the LLM, first sentence into TTS, first audio chunk onto the wire – so the stages overlap instead of queueing. The second is a filler strategy: a short, contextual acknowledgment while a slow tool call runs buys you two full seconds of perceived headroom. Measure at p95, not average. The mean looks fine while one call in twenty is unusable.

How do you handle interruptions and barge-in?

Barge-in is the feature that separates a voice agent from a talking IVR, and it is where most demos quietly fail. When a caller starts speaking over the agent, three things must happen within about 100ms: audio playback stops, the in-flight LLM generation is cancelled, and the conversation state is truncated to what the caller actually heard – not what the model generated. Skip that last step and the agent will confidently reference information the caller never received.

The hard part is distinguishing a real interruption from a backchannel. "Mm-hmm," "right," and "okay" are the caller confirming they are listening, and an agent that stops dead every time someone says "uh huh" is worse than one that never stops at all. The practical approach is a two-signal gate: require both sustained speech energy above a threshold and a partial transcript that is not on a backchannel list. Also handle the inverse failure – echo. If your agent hears its own output through a speakerphone and treats it as barge-in, it will interrupt itself in a loop until the caller hangs up.

How do you connect it to the phone network?

Telephony is the part that looks trivial in a diagram and consumes a third of the build. You need a provider that supports real-time media streaming – SIP trunking or a WebSocket audio stream – so raw frames reach your agent service and synthesized audio streams back with no buffering layer in between.

Four constraints reliably surprise teams. First, phone audio is narrowband 8kHz, and transcription accuracy on 8kHz call audio is measurably worse than the wideband audio in every vendor demo – benchmark on real calls or your numbers are fiction. Second, DTMF keypad input still matters, because callers press zero and expect a human. Third, warm transfer needs designing: the agent must summarize the call for the human it hands off to, or you have built an expensive way to make customers repeat themselves. Fourth, call recording and AI-disclosure rules vary by jurisdiction, and the consent prompt has to be part of the flow from day one rather than bolted on before launch.

How do you evaluate a voice agent before it takes real calls?

Text-agent evals do not transfer, because the failure modes live in the audio layer. Build a regression suite of recorded audio and score four dimensions:

• Transcription accuracy on your vocabulary. Vendor word error rates are measured on clean general speech. Yours needs measuring on your product names, drug names, street names, and account formats, over a phone line.
• Task success on complete calls. Not per-turn quality – did the appointment actually get booked, with the right details, in the real system?
• Latency at p50 and p95 over the telephony path, not in a browser tab.
• Interruption behavior. Barge-in success rate and false-positive rate on backchannels, scored separately.

Then run the adversarial set: heavy accents, background noise, hold music, two people talking at once, and callers who change their mind mid-sentence. The general methodology carries over from our guide to evaluating and testing AI agents; the audio harness is the voice-specific addition.

What does it cost to build and to run?

Two separate budgets, and teams routinely model only the first. On the build side, adding a single voice capability to an existing product runs roughly $15,000–50,000. A focused voice MVP – one call type, real telephony, evals, and monitoring – runs $50,000–150,000 over 30–45 days. A production system spanning multiple call types, warm transfer, compliance recording, and a full regression suite starts at $150,000. Voice sits at the upper end of comparable text-agent work because real-time infrastructure and the call-audio test harness are genuine engineering, not prompt iteration.

On the run side, budget roughly $0.05–0.20 per minute all-in across transcription, LLM tokens, synthesis, and telephony. Synthesis and telephony are the largest predictable lines; LLM cost varies most, because it scales with context length and tool calls per turn. Model cost per resolved call rather than per minute – a better model that resolves in ninety seconds beats a cheap one that runs four minutes and still transfers to a human.

What should you build first?

One call type, end to end, in production. Pick the highest-volume, lowest-variance call you handle – appointment confirmation, order status, qualification – and ship it with a clean human handoff path and a hard cap on what the agent may do without approval. That scope fits a 30–45 day build, produces real call recordings to evaluate against, and tells you within a month whether your unit economics work. Voice agents that fail almost always failed at scoping, not at modeling: they tried to answer every call on day one and were beaten by the twenty percent of callers who did not follow the script.

If you want the broader agent architecture context first – tools, memory, guardrails, human approval – start with our guide to building an AI agent for your business, then come back for the audio layer. And if you would rather have a team that has already paid the telephony tuition, that is what we do – see our services for how we scope and ship production voice systems in 30 to 45 days.