Dumbly

An intake call is a person reading a form out loud to another person who types it back into the same form. The caller knows the answers. The operator knows the fields. Between them sits a keyboard and a lot of "sorry, can you say that again." That queue was a real cost: hundreds of these calls a day, each one an operator half-listening and half-typing, and the typing is where the mistakes live. You mishear a number, you tab to the wrong field, you catch it three fields later and now you're scrolling back up while the caller waits. The work isn't hard. It's just constant, and constant is expensive.

Dumbly was my attempt to take the typing off the operator and leave them the part that needs a human: deciding whether what the machine heard is actually true. It's an Electron desktop app that sits on the operator's machine, listens to the live call through the OpenAI Realtime API, and does structured extraction from the audio as the conversation happens. The form fills itself while the caller talks. The operator watches, corrects, and confirms. Today 20+ operators run it, handling 300 to 400 intakes a day, and manual intake work is down roughly 70%.

That last paragraph is the version I'd put on a slide. What actually happened is that the first thing I built worked in the demo and then broke in two specific, memorable ways the moment real people used it on real calls. This is the story of those two failures and the redesign they forced, because the redesign is the only interesting part.

The problem

Real-time voice extraction sounds like one problem and is actually three, stacked, each one making the next harder.

The first is latency. If the form fills a beat after the caller speaks, fine. If it fills five seconds later, the operator has already given up and grabbed the keyboard, and now you've built a slower version of what they already had. The Realtime API exists precisely so you don't round-trip audio to text to a second model and back; the model handles live audio directly over a streaming transport. That solved the latency I was worried about and handed me the two I wasn't.

The second is that speech isn't structured and a form is. People don't say "annual revenue: four hundred thousand dollars." They say "we did, uh, call it four hundred last year, maybe a little more." Somewhere between that and a number in a box, a decision got made, and the question is who makes it and whether they're allowed to be wrong.

The third is trust. An operator who has to double-check every single field the machine fills is doing more work than if they'd typed it themselves, because now they're reading and verifying instead of just transcribing. For the tool to be worth anything, the operator has to be able to trust most of the fields most of the time, and know which ones to distrust. That last clause is the whole project. I did not understand that when I started.

The naive version

The first build was about as direct as it gets, and I want to describe it honestly because it's the version most people would build and it's the version that taught me the most.

The app captured the call audio and streamed it to the Realtime API. I gave the model the intake schema as a set of tools, one per section, using function calling so that as the model recognized a field it would call the corresponding function with the value. A tool call came back, I wrote the value into the form, the box lit up. I leaned on the built-in voice activity detection to chunk the audio into turns and let the model extract off each turn as it settled. The whole design rested on an assumption I didn't examine: that a turn in a phone call is a finished answer worth writing down.

In the demo this looked like magic. I'd read a fake application out loud and watch the form populate in near real time. I showed it around. People nodded. Then it went in front of operators on live calls, and it failed in the two ways that gave this project its actual content.

Failure one: the Eager Scribe

The first failure I named the Eager Scribe, because the model behaved like a court stenographer who writes down your answer while you're still deciding what it is.

Callers revise themselves constantly. "It's about forty employees, well, forty-two counting the part-timers, actually call it forty." The naive version would hear "forty," fire the tool call, write 40 into the field, and light it up green. Then the caller says "forty-two," and, depending on how the turn got chunked, the model either didn't re-fire or re-fired into a field the operator had already glanced at and mentally checked off. The wrong-but-confident value just sat there. Nobody re-reads a field that already looks done.

The Eager Scribe was, at least, an honest kind of wrong. The value in the box genuinely corresponded to something the caller said at some point. It was stale, not invented. Which meant the downstream damage was the ordinary kind: a value that has to be caught and corrected later, by someone with less context than the operator who was on the call, or not caught at all until it's inconvenient. A number that's merely out of date is still a number someone has to un-trust after the fact, and un-trusting is the exact work the tool was supposed to remove.

The root cause wasn't the model being dumb. It was me treating a live conversation like a stream of final answers. A turn in a phone call is not a committed fact. It's a draft the caller might revise in the next breath, and I'd wired the form to commit drafts.

Failure two: the Phantom

The second failure was worse, and I didn't see it for a while because it was quiet.

Call the model's output what it is: a very good guess. Most of the time the guess is right. But when the audio was unclear, a caller trailing off, a bad connection, cross-talk, the model did not say "I didn't catch that." It filled the field anyway with something plausible. A number that could be a revenue figure. An address that parses. A date in range. I named this one the Phantom because it produced fields the caller had never actually said, and they were indistinguishable, on the screen, from the fields the caller had said.

This is the failure that matters, and it's a property of the whole approach, not a bug I could patch. A model asked to extract a field will extract a field. It is not, by default, in the business of abstaining, and there's a good argument that the way these models are trained and scored rewards guessing over admitting uncertainty in the first place. A filled box carries no memory of how sure the machine was when it filled it. Green is green. The operator, reasonably, trusted green.

So the tool was quietly training its own users to trust it exactly when they shouldn't. The clearer the caller, the more accurate the extraction, the more the operator relaxed, and the better positioned a Phantom was to sail through unnoticed on the one field in the call that happened to be mumbled. A stale value announces itself eventually; the caller said something else and the record disagrees. A Phantom has nothing to disagree with. The caller never said it, so there's no later moment where the truth resurfaces on its own. You only catch it if the machine told you, at the time, that it wasn't sure, and the machine wasn't telling anyone anything.

If I'm honest, the Phantom is what turned Dumbly from a neat demo into a thing I actually respected, because it's the failure that generalizes. Any system that turns a fuzzy signal into a crisp field has a Phantom in it somewhere. The only question is whether you've given it a place to show up other than a confident green box.

The redesign

The fix was one idea applied everywhere: the model proposes, it does not commit. Nothing the model extracts writes itself into the form. It becomes a proposal, and a proposal has to earn its way into a committed field.

Three changes carried the weight.

First, I stopped committing drafts. Extraction now runs off settled utterances rather than every partial turn, so the Eager Scribe writes down your final answer instead of your first one. The lever here is turn detection: instead of chunking on raw silence, the Realtime API can run semantic voice activity detection, a mode that uses a classifier to judge from the words themselves whether the caller has actually finished the thought rather than just paused mid-sentence. Waiting for the thought to close, instead of for the audio to go quiet, is what keeps a mid-revision "forty" from being treated as an answer. And because the result is a proposal and not a committed field, when a caller does revise, the proposal revises in place, which is expected behavior rather than a surprise overwrite.

Second, every proposal carries evidence. When the model proposes a value, it also has to point at the span of transcript that value came from, and attach how sure it is. This is the part that kills the Phantom. A value with no transcript span behind it is, definitionally, invented, and the system can now tell the difference between "the caller said four hundred thousand" and "the model felt like four hundred thousand." I treat the grounding, whether the value traces to something in the transcript, as the real signal, and the model's own stated confidence as a weak secondary hint, because a model can be fluent and wrong in the same breath. Proposals that don't clear the bar don't get shown as confident fields. They get held back and surfaced as "I didn't catch this one, you fill it," which is the honest thing to say and, it turns out, the thing operators trust.

Third, the operator confirms. A proposal is visibly a proposal, not a filled field, until the operator accepts it. Most proposals are right and accepting them is one gesture, so the operator isn't retyping, they're ratifying. But the screen never lies to them about what's been checked and what hasn't. A confirmed field looks confirmed. A proposal looks like a proposal. A held-back field looks like a hole. The operator's attention goes exactly where the uncertainty is, instead of being spread flat across a form where everything looks equally done.

The through-line, and the reason I keep coming back to it, is the same conviction that drove the human-in-the-loop gates in DaVinci: the human isn't a nag bolted onto a finished system. The human is a load-bearing step, and the system's job is to route their attention to the fields that actually need it. Get that right and the tool is fast because the human is trusting it correctly. Get it wrong and you've built the Phantom's delivery mechanism.

How it works now

Today the app runs on the operator's desktop and captures the call audio locally. That it's Electron is not incidental. A voice tool has to live where the call already lives, capture audio off the machine reliably, and sit in an always-there window beside whatever else the operator has open, rather than competing for attention as one more browser tab that can be closed or backgrounded. A desktop shell that owns its own window and captures audio through the platform's media APIs fit those constraints in a way a web page did not.

Audio streams to the Realtime API over its low-latency transport, negotiated the same way a browser sets up a peer connection. The model both transcribes and extracts, and turn detection decides when a stretch of speech has settled enough to extract from. Each settled utterance produces zero or more field proposals, each with its transcript span and a confidence. Grounded, confident proposals show up in the form as pending values the operator can accept in a gesture. Ungrounded ones get held back and marked as needing a human. Only accepted values commit.

The operator is on the call the whole time, so a correction is a normal part of the conversation. Caller misspeaks, operator sees the proposal update, confirms the corrected one, moves on. What used to be "listen, type, scroll back, fix, apologize" is now "listen, glance, confirm." That's where the roughly 70% reduction in manual intake work comes from. The machine isn't doing 70% of the thinking. It's doing the typing, and the operator is left with only the judging, which was always the smaller job.

The latency budget

The loop I just described has a clock in it, and it's worth taking apart, because the whole design lives or dies on where the time goes. One pass through the loop: the caller talks, the semantic VAD decides the thought has closed, extraction runs over the settled utterance, the proposal renders with its evidence, and the operator confirms it with a gesture. Five stages. They are nowhere near equal, and the useful move is to sort them by who pays for them.

The caller's speech is the longest stage by an order of magnitude and it costs the system nothing, because it was going to happen anyway. The audio streams while they talk, so by the time an answer ends, the model has already heard all of it. The confirm gesture at the far end is nearly free too, since the operator does it while the caller is already answering the next question. Neither of those sits on the path that matters. What sits on the path is the gap between the caller finishing an answer and the proposal appearing on screen, and only two stages live in that gap. The settle decision can't overlap anything, by definition, because its whole job is to notice the utterance is over, and it can only do that after the last word. Extraction starts the moment the turn commits. Rendering the proposal is a local write and effectively instant.

So if I call the settle delay $t_s$ and the extraction time $t_e$, the delay the operator actually experiences, answer done to proposal visible, is $t_s + t_e$ and nothing else. Illustrative: suppose an answer takes six seconds to say, the settle decision costs half a second after the last word, and extraction over the utterance takes another second and a half. The proposal lands two seconds after the answer ends. Two seconds would be an embarrassing latency for a text box and is nothing at all on a phone call, because the operator spends those two seconds saying the next question out loud, which they were going to do regardless. The machine's entire visible delay fits inside a stretch of conversation that already existed. That overlap is the design, and it's why the loop being paced by conversation is a gift rather than a constraint: the conversation keeps generating dead air, and the system spends its inference time there.

The budget's hard ceiling has a name from earlier in this piece: the operator grabbing the keyboard. A proposal that lands after the operator has already started typing is worth nothing, and in practice slightly less than nothing, because now it's fighting the operator's own half-typed answer for the same field. That ceiling is the binding constraint, and it's why the settle decision deserves more suspicion than it usually gets. Semantic VAD buys correctness, waiting for the thought to close instead of the audio to go quiet, and it pays for that correctness in exactly the currency this budget is denominated in, delay after the last word. The dial that made the Eager Scribe survivable and the dial that blows the latency budget are the same dial turned in opposite directions.

What I never measured, and would want to before defending any of this with numbers, is the ceiling itself: how long an operator actually waits, on a live call, before deciding the machine isn't coming and typing the value themselves. Measuring it well means logging the moment a cursor enters a field that had a proposal in flight, and I didn't build that log. The two-second figure above is a budget I believe the system meets by feel, not a number I can show you. That gap is what the next section is about.

How I'd measure it

The numbers I've published about Dumbly are outcome numbers: 20-plus operators, 300 to 400 intakes a day, manual intake work down roughly 70%. Outcome numbers are what the business cares about, and they say almost nothing about whether the loop itself is healthy. A system can hit those numbers while quietly degrading in ways nobody has instrumented. So here is what I think honest measurement of this system looks like, written partly as a plan and partly as a confession about what I didn't build.

Start with timestamps at the boundaries the transport already exposes. The Realtime API emits events at exactly the seams the budget cares about: speech starting and stopping, the turn committing, the response streaming and completing. Stamp each event as it arrives at the app, and stamp the two purely local events, proposal rendered and proposal confirmed, on the same clock. That decomposes every field into the stages from the budget above without inventing any new instrumentation, and because every stamp lands on the operator's machine, there's no cross-clock skew to argue about. The one number the events can't give you is the ceiling, the operator's give-up threshold, which is why the cursor-enters-a-pending-field log matters more than any of the machine-side stamps.

Then report percentiles within a call, and distributions across calls, and never a fleet mean. Latency on a live call is heavy-tailed in the ways you'd expect: a mumbled answer takes longer to settle, a long utterance takes longer to extract. The statistic I'd watch is each call's p50 and p95 settle-plus-extract delay, then the distribution of those per-call p95s across the day. A fleet-wide mean would average a crisp caller on a good line together with a mumbler on speakerphone and report that everything is fine while one operator's whole afternoon was quietly awful. The operator's trust is set by the worst field on their own call, so p95 within the call is the number that predicts whether they grab the keyboard. And the sample is not the problem: 300 to 400 intakes a day across 20-plus operators is 15 to 20 calls per operator, every working day, which makes one day of production traffic a bigger and more representative sample than any eval set I would have assembled by hand.

Third, treat the grounding bar as a curve, not a setting. The bar that decides whether a proposal gets shown or held back is a dial, and the honest way to look at a dial is to sweep it: replay logged proposals at each candidate threshold and plot the held-back rate against the Phantom escape rate. The labels come free, which is the quiet gift of keeping a human in the loop. A proposal the operator corrected before confirming was a machine error the design caught. A held-back field the operator typed was the cost side of the dial. A confirmed field that got corrected later, downstream, was an escape. Every operator correction is ground truth somebody was already being paid to produce. The curve doesn't tell you where to set the dial, because that's a judgment about which failure is more expensive, but it tells you exactly what each setting buys and costs, which beats the way I actually set it, by feel, against a handful of test calls.

Last, measure reliability across repeated trials, because a single good run tells you close to nothing. The same recorded call, replayed through the extraction pipeline twice, will not produce identical fields; sampling and turn-boundary jitter see to that. The tau-bench authors, benchmarking tool-using agents in settings a lot like this one, made the point in the way that stuck with me:

We also propose a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials.

Yao, Shinn, Razavi, and Narasimhan, tau-bench

Their headline finding, that state-of-the-art function-calling agents succeeded on under half their tasks and were inconsistent enough that pass^8 fell below 25% in one domain, is the finding to take personally. For Dumbly the analogue is replaying each recorded call through the pipeline k times and counting the fields correct in all k runs, not the fields correct once. A field that comes back right in four replays out of five is a field that's wrong for one caller in five, and neither the operator nor I can tell, on any given call, which run we're in. The demo that impressed everyone was one run.

None of this instrumentation is exotic, and most of it is a logging statement away from the events the system already handles. The honest state of things is that Dumbly's design guards against the failures I could see, and the measurements above are how I'd find the ones I can't. The next system I build like this gets the logging before it gets the demo.

The honest residue

The redesign fixed the two failures it was built to fix. It did not make the tool perfect, and a few rough edges are load-bearing enough that I'd be lying to leave them out.

Held-back fields are a cost, not a free win. Every time the system says "I didn't catch this, you fill it," it's handing work back to the operator. Set the grounding bar too conservative and you've built a tool that shrugs at everything and makes the human type it all anyway. Too permissive and the Phantom creeps back. That bar is a dial with a bad failure mode on both ends, and I don't think there's a setting that's right for every call, because the calls aren't the same: a crisp caller on a good line and a mumbler on a bad one want different amounts of caution, and the tool is choosing one policy for both.

Accents, cross-talk, and bad phone lines are still where accuracy drops, and those conditions aren't evenly distributed across callers, which is a fairness problem hiding inside an accuracy problem. The tool is most confident exactly where the audio is cleanest, which is not the same population as the callers who most need to be understood. I don't have a clean fix for that. The propose-don't-commit design at least routes the harder calls toward more held-back fields rather than more silent Phantoms, so the failure mode on a bad line is "the human types more," which is the survivable one. But a tool that quietly asks more of some callers' operators than others is still doing something I'd want to measure before I'd want to defend.

And the tool is, by design, only as good as the operator's willingness to actually look. The redesign routes attention to uncertainty, but a tired operator at the end of a shift can still rubber-stamp a screen full of proposals. The system makes the right thing easy. It can't make it mandatory without becoming the slow thing again, and I don't think the answer is to make it mandatory. A tool that forces a human to re-verify every field has reinvented the typing it was supposed to remove.

What operators actually did

The part I got most wrong was predicting how operators would react to the tool admitting it wasn't sure.

My assumption going in was that operators would want the tool to seem confident, that visible "I'm not sure" states would read as the tool being weak, and that hiding the uncertainty would make it feel more finished. The opposite held. A tool that admits when it didn't catch something is a tool you can hand the boring majority of the work to, precisely because you know it'll tap you on the shoulder for the part it couldn't. The held-back field, the thing I'd assumed operators would resent, is the thing that let them stop double-checking everything else. Confidence theater would have gotten exactly one Phantom through before they stopped trusting any of it, and a tool you've stopped trusting is a tool you've gone back to typing around.

That's the counterintuitive win, and it's the one I'd carry into any tool like this: making the uncertainty visible didn't cost trust, it bought it. People don't need the machine to be right about everything. They need it to be honest about which things it isn't sure of, so they can spend their attention there and nowhere else.

What I'm still not sure about

I'll end where I'm honest with myself, which is a list of things I believe are true and am not certain about.

I'm not sure the desktop app was the right call long term. It solved the audio-capture and always-there problems cleanly, but it's a heavier thing to ship and update than a web surface, and if the call platform ever exposed the audio stream in a way a browser tab could reliably grab, a lot of the reason to be on the desktop would evaporate. I'd want to rebuild it the moment that constraint changed, and I'm not certain I'll notice the moment it does.

I'm not sure confidence from a language model means what I want it to mean. I lean on "grounded in the transcript" as the real signal and treat model-reported confidence as a weak secondary hint. But grounding has its own hole: the model can ground a value in a transcript span that the transcription itself got wrong, and then the Phantom comes back wearing evidence. A confident, well-grounded, wrong field is the hardest kind to catch, because everything the system checks for says it's fine, and I don't have a clean answer for that one. The best I've got is that a transcription error is at least a different failure than a hallucination, and sometimes it's audible enough on the call that the operator catches it live. Sometimes.

I'm not sure where the line between copilot and autopilot should sit. Every held-back field is a small admission that the tool won't fully close the loop, and there's constant pressure, from myself as much as anyone, to let the confident proposals just commit without a human. The whole argument of this write-up is that you shouldn't. But I notice I want to, on the easy calls, and I don't fully trust my own judgment about which calls are easy. That's the same discomfort I wrote about when I argued that agents need a nervous system before they earn autonomy, and Dumbly is the version of that argument with a human's voice on the other end of the line.

What I am sure of is the one thing the two failures taught me. A system that turns speech into structure is going to guess, and the entire design problem is not making it guess less. It's making it tell you, honestly and in the right place, when it just guessed. Everything good about Dumbly is downstream of that.