A voice agent that takes the call alone

A business owner calls to get insured. A voice agent picks up, finds their company from the number they're calling from, and works through what an underwriter needs: what the business does, how many people work there, what it wants covered, where it operates. When the call ends, a durable job turns the conversation into completed insurance applications and files them.

Somewhere in that call the agent heard "Suite 4B" as "Suite 48." Who catches that?

Before this, the answer was an operator. I'd built an intake copilot that listened to calls and filled the form live while an operator watched, corrected and confirmed each field. The model proposed and a person decided. When the brokerage moved to a voice agent that takes the call by itself, that person was gone, and a lot of things the operator used to absorb without thinking became engineering problems: when the caller has finished talking, what "don't email and text" means, how to end a call without cutting someone off, and who checks the facts now.

This is a team system. I'll describe the whole pipeline, because the parts I owned only make sense inside it, and say which parts were mine.

The problem

A voice agent on a phone line has less room for error than a chat agent, in four ways.

Latency is audible. In chat, a two-second pause is nothing. On a call it's dead air, and the caller starts talking again just as the agent does.

Turn-taking is a guess. Nobody presses Enter on a phone. The system has to decide from audio alone when the caller has finished a thought, and a pause while someone looks up their tax ID sounds exactly like the end of an answer.

Speech recognition fails on the words that matter. Business names, street names, insurance terms and industry jargon are exactly what a general speech model mishears, and they're exactly what ends up on the application.

Every action is consequential. Hanging up, transferring, sending a text and filing an application can't be undone. A model that says "let me connect you" and then doesn't is not a cosmetic bug.

How it works

A call arrives over a SIP trunk into a LiveKit room, and a Python LiveKit Agents worker joins it. The pieces, in the order a call meets them:

The voice is a realtime speech-to-speech model. The conversation runs on OpenAI's Realtime API audio-to-audio, with no separate text-to-speech stage. That keeps latency low and the voice natural, and it means the model also decides, through server-side voice activity detection, when the caller has finished speaking. The tuned settings are a 0.8 detection threshold, 300 ms of prefix padding and 900 ms of silence before a turn ends. The 900 is a scar: at 400 ms, a caller pausing mid-answer had the next question start over them. A per-call experiment arm can switch a percentage of calls to semantic turn detection, which judges whether a thought sounds finished rather than timing the silence.

The transcript comes from Deepgram. The realtime model transcribes the caller too, but the transcript that becomes the application comes from a sidecar that taps only the caller's microphone and streams it to Deepgram Nova-3. Each call gets up to 100 keyterms: a general list, then terms for the caller's industry and the forms on the call, then the company and contact names already on record. The terms were mined from words Nova-3 actually missed on production calls. If the Deepgram stream degrades, the worker falls back to the realtime model's own transcription so the caller's side is never blank.

The questions come from a talk path. Each industry has a reviewed question track, 29 of them in the repository, compiled into a short checklist the model works through and reports progress against. Fields the company record already answers are confirmed rather than asked open: if the record says what the business does, the agent confirms it with one yes or no.

Code owns the consequential moments. Keypad entry for numbers that are easier typed than spoken. Hold audio while slow work runs. A frustration gauge that scores every caller turn from 0 to 100 and, after three consecutive exchanges at 60 or above, offers a person. Counting exchanges instead of seconds means a caller's long pause can't manufacture an escalation. Transfer is a cold SIP REFER to a human. Hang-up deletes the LiveKit room, because disconnecting the agent alone doesn't end a phone call; after a successful transfer, the worker only leaves, because deleting the room would hang up on the person who was just handed the call.

Nothing is lost if the worker dies. Every lifecycle event is written to a spool on a persistent volume before delivery, retried with backoff and flushed in order. Events that can never be delivered become dead letters instead of blocking newer ones, and a sweeper retries stranded events every 30 seconds. The worker refuses to run live if the spool isn't on persistent storage.

The application is written after the call. A leased, retried job fills each application against its blank PDF. One rule from that path is worth stealing anywhere a model fills forms: there is no assumed default. An unasked yes/no is not No, and an unasked count is not 0. A field gets a value from what the caller said, from what their answers settle ("no employees" settles every age band), or from a default the form's own rules name. Once an application is filed, it doesn't change.

Two ears, one conversation

Running the conversation on one model and the transcript on another solved the accuracy problem and created a new one: the two don't always agree on what happened.

The sharpest version is the missed turn, which a teammate tracked down. A caller says "Yes." Deepgram hears it and writes it down. The realtime model's voice detection never commits it as a turn, so no reply is generated, and the line goes silent with the caller waiting. The fix listens for LiveKit's "away" signal, and if the caller's words are the last thing said, the agent is idle and nothing is in flight, it hands those words back to the model exactly once. A Deepgram final that lands within 2.5 seconds of the agent starting its answer is held back as a probable late copy, so the same "Yes" isn't answered twice.

Another came from session length. The LiveKit plugin recycles the realtime session after 20 minutes by default, and the replay drops every function call and tool result. A normal intake call can run past 20 minutes, and when it did, the model started reporting invalid tool-call IDs mid-conversation. The recycle now happens at 55 minutes, past the end of any real call and still five minutes under OpenAI's own one-hour close.

Making scripted lines audible

Some lines can't be left to the model: the confirmation offer, the hold message, the closing. On an audio-to-audio pipeline there is no text-to-speech stage to say a fixed sentence, and the LiveKit session can't say bare text, so each fixed line is requested from the realtime model already on the call, with tools turned off.

On 12 October the separate text-to-speech service we used for some of those lines began stalling on most calls, and the confirmation and closing lines turned into dead air. The short-term fix forced a stalled line to give way so the lines queued behind it could play, cached audio per voice and text, and warmed the fixed lines at session start. The lasting fix was to record every scripted line once in the realtime model's own voice and keep a take only when both the model's transcript and an independent transcription matched the script word for word. A test fails if a spoken line has no recording. The agent now sounds like one voice from start to finish, and its scripted lines no longer depend on that service.

Ending the call

How calls end was mine, and it's where a voice agent is most likely to do something a person never would.

Goodbye means hang up, and only hang-up says goodbye. The model sometimes said goodbye and stayed on the line, and sometimes said goodbye before the call's work was done. The rule is now strict: after the last question, the model calls hang_up without speaking, and the hang-up path speaks the closing that fits the call. It never says a goodbye without hanging up in the same turn. If the caller jumps in during the closing, it answers briefly.

Frustration isn't a request to leave. Swearing or "I quit" in the middle of a question used to read as an ask to end the call. Now it gets one short sentence and the question again. Asking for a person is different, and a second ask counts only when it's more than two real seconds after the first, so a transcript that repeats one sentence can't double it.

A promise to transfer has to be kept. When the model said "let me connect you with someone," that line was ordinary speech, so nothing dialed, and the silence timer ended the call on a person waiting to be connected. A spoken promise now starts the transfer path, and a close that says a person is coming either starts the dial itself or keeps its one hold line in the transcript.

The model stays quiet while code holds the line. During a hold with no words from the caller at all, the realtime model's voice detection heard hold music and line noise and started replying: "Please hold for a moment," then "It's still processing," then "Still running." While the worker owns a hold, any reply the model starts on its own is now interrupted. The worker's own check-ins still play.

Letting the caller check

The bigger piece I built is customer confirmation. Before the agent says goodbye, it offers to send a link by text, email or both. The caller picks, holds briefly while the link is prepared, hears once that it was sent, and the call closes. On the page they see only what they can verify about their own business: legal name, address, tax ID, website, and the identity questions each form on the call asks. Things they can't vouch for, like the broker's own details, never reach the page.

A correction is bound to the exact field on the exact version of the form the fill job froze, and it's applied under the same lock the fill uses. If the fill is still running, the correction lands and a stale copy can't overwrite it. If finalization got the lock first, the correction is stored as evidence and the filed packet stays as it was, because rewriting a document someone already received is worse than a known, recorded discrepancy. What already sits on the company record never counts as the caller agreeing to it. A newer rejection invalidates an earlier yes. An expired link lets the fill go ahead with what the call captured, and a decline revokes the prepared link.

Two parts of this turned out to be harder than the data model.

Parsing the channel. "Text me" is easy. "Don't email and text" was parsed as SMS, because once "and" split the phrase only the "text" half matched. Negations joined by and, or and nor now refuse both, while "skip the email and text me" still means text. "That works," "Go ahead" and "I don't mind" all count as yes to a single-channel offer. The model also tended to say goodbye before asking at all, so the confirmation step now overrides any script's closing on calls that offer the link.

Racing the goodbye. Preparing the link means reading back everything the call extracted, and on a long call that made the hold feel like being dropped. Preparation now starts when the agent asks the channel question, and a link is prepared speculatively while the caller answers. Speculation produced most of the race conditions I fixed: a decline has to revoke the speculative link before anything can send it, a changed answer has to remake it, and if revoking fails the system keeps the link's ID so a later send can't deliver the stale one. Each was a separate bug, found by asking what happens when two events arrive in the other order.

Results

I'm not publishing call volumes, confirmation rates or anything about customers. What I can describe is the shape of the work. Over September and October I made about 350 commits to the intake agent and another 69 to the voice backend, almost all of them small fixes to specific call behaviors. Each behavior above is pinned by a test in the repository, several of them named after the sentence that broke it, like test_frustration_is_not_an_ask_to_end_and_a_goodbye_always_hangs_up. The confirmation loop is live, and it can be switched off for new sends with a single flag while links already sent keep working.

Limits

Every other case study on this site links to its receipts, and this one can't: the code belongs to my employer, and the parts that would identify its systems are left out. The number that matters most, how often callers actually change something on the confirmation page, is in the logs and stays there with the rest of the metrics. Without it, this write-up can show that the loop is built carefully, not that it catches a given share of errors.

Much of the pipeline isn't mine. The Deepgram sidecar and its keyterms, the missed-turn replay, the session recycle fix, the frustration gauge and the fill job were built by teammates. I've described them because the call doesn't make sense without them.

What's next

The copilot taught me to let the model propose and a person decide. The voice agent removed the person I'd been relying on, and the answer wasn't to trust the model more. It was to find the next person who knows the truth, and that turned out to be the one on the other end of the call. Irrevon settles an ambiguous payment from the destination's own records for the same reason.

The open questions are measurement ones. Semantic turn detection is already wired as a per-call experiment arm, and the confirmation logs can say whether the loop is catching real errors or mostly adding a step. Both should be answered with data from real calls before anyone tunes them further.