Dumbly

An intake call is a person reading a form out loud to another person who types it back into the same form. The caller knows the answers. The operator knows the fields. Between them sits a keyboard and a lot of "sorry, can you say that again." That queue was a real cost: hundreds of these calls a day, each one an operator half-listening and half-typing, and the typing is where the mistakes live. You mishear a number, you tab to the wrong field, you catch it three fields later and now you're scrolling back up while the caller waits. The work isn't hard. It's just constant, and constant is expensive.

Dumbly was my attempt to take the typing off the operator and leave them the part that needs a human: deciding whether what the machine heard is actually true. It's an Electron desktop app that sits on the operator's machine, listens to the live call through the OpenAI Realtime API, and does structured extraction from the audio as the conversation happens. The form fills itself while the caller talks. The operator watches, corrects, and confirms. Today 20+ operators run it, handling 300 to 400 intakes a day, and manual intake work is down roughly 70%.

That last paragraph is the version I'd put on a slide. What actually happened is that the first thing I built worked in the demo and then broke in two specific, memorable ways the moment real people used it on real calls. This is the story of those two failures and the redesign they forced, because the redesign is the only interesting part.

The problem

Real-time voice extraction sounds like one problem and is actually three, stacked, each one making the next harder.

The first is latency. If the form fills a beat after the caller speaks, fine. If it fills five seconds later, the operator has already given up and grabbed the keyboard, and now you've built a slower version of what they already had. The Realtime API exists precisely so you don't round-trip audio to text to a second model and back; the model handles live audio directly over a streaming transport. That solved the latency I was worried about and handed me the two I wasn't.

The second is that speech isn't structured and a form is. People don't say "annual revenue: four hundred thousand dollars." They say "we did, uh, call it four hundred last year, maybe a little more." Somewhere between that and a number in a box, a decision got made, and the question is who makes it and whether they're allowed to be wrong.

The third is trust. An operator who has to double-check every single field the machine fills is doing more work than if they'd typed it themselves, because now they're reading and verifying instead of just transcribing. For the tool to be worth anything, the operator has to be able to trust most of the fields most of the time, and know which ones to distrust. That last clause is the whole project. I did not understand that when I started.

The naive version

The first build was about as direct as it gets, and I want to describe it honestly because it's the version most people would build and it's the version that taught me the most.

The app captured the call audio and streamed it to the Realtime API. I gave the model the intake schema as a set of tools, one per section, using function calling so that as the model recognized a field it would call the corresponding function with the value. A tool call came back, I wrote the value into the form, the box lit up. I leaned on the built-in voice activity detection to chunk the audio into turns and let the model extract off each turn as it settled. The whole design rested on an assumption I didn't examine: that a turn in a phone call is a finished answer worth writing down.

In the demo this looked like magic. I'd read a fake application out loud and watch the form populate in near real time. I showed it around. People nodded. Then it went in front of operators on live calls, and it failed in the two ways that gave this project its actual content.

Failure one: the Eager Scribe

The first failure I named the Eager Scribe, because the model behaved like a court stenographer who writes down your answer while you're still deciding what it is.

Callers revise themselves constantly. "It's about forty employees, well, forty-two counting the part-timers, actually call it forty." The naive version would hear "forty," fire the tool call, write 40 into the field, and light it up green. Then the caller says "forty-two," and, depending on how the turn got chunked, the model either didn't re-fire or re-fired into a field the operator had already glanced at and mentally checked off. The wrong-but-confident value just sat there. Nobody re-reads a field that already looks done.

The Eager Scribe was, at least, an honest kind of wrong. The value in the box genuinely corresponded to something the caller said at some point. It was stale, not invented. Which meant the downstream damage was the ordinary kind: a value that has to be caught and corrected later, by someone with less context than the operator who was on the call, or not caught at all until it's inconvenient. A number that's merely out of date is still a number someone has to un-trust after the fact, and un-trusting is the exact work the tool was supposed to remove.

The root cause wasn't the model being dumb. It was me treating a live conversation like a stream of final answers. A turn in a phone call is not a committed fact. It's a draft the caller might revise in the next breath, and I'd wired the form to commit drafts.

Failure two: the Phantom

The second failure was worse, and I didn't see it for a while because it was quiet.

Call the model's output what it is: a very good guess. Most of the time the guess is right. But when the audio was unclear, a caller trailing off, a bad connection, cross-talk, the model did not say "I didn't catch that." It filled the field anyway with something plausible. A number that could be a revenue figure. An address that parses. A date in range. I named this one the Phantom because it produced fields the caller had never actually said, and they were indistinguishable, on the screen, from the fields the caller had said.

This is the failure that matters, and it's a property of the whole approach, not a bug I could patch. A model asked to extract a field will extract a field. It is not, by default, in the business of abstaining, and there's a good argument that the way these models are trained and scored rewards guessing over admitting uncertainty in the first place. A filled box carries no memory of how sure the machine was when it filled it. Green is green. The operator, reasonably, trusted green.

So the tool was quietly training its own users to trust it exactly when they shouldn't. The clearer the caller, the more accurate the extraction, the more the operator relaxed, and the better positioned a Phantom was to sail through unnoticed on the one field in the call that happened to be mumbled. A stale value announces itself eventually; the caller said something else and the record disagrees. A Phantom has nothing to disagree with. The caller never said it, so there's no later moment where the truth resurfaces on its own. You only catch it if the machine told you, at the time, that it wasn't sure, and the machine wasn't telling anyone anything.

If I'm honest, the Phantom is what turned Dumbly from a neat demo into a thing I actually respected, because it's the failure that generalizes. Any system that turns a fuzzy signal into a crisp field has a Phantom in it somewhere. The only question is whether you've given it a place to show up other than a confident green box.

The redesign

The fix was one idea applied everywhere: the model proposes, it does not commit. Nothing the model extracts writes itself into the form. It becomes a proposal, and a proposal has to earn its way into a committed field.

Three changes carried the weight.

First, I stopped committing drafts. Extraction now runs off settled utterances rather than every partial turn, so the Eager Scribe writes down your final answer instead of your first one. The lever here is turn detection: instead of chunking on raw silence, the Realtime API can run semantic voice activity detection, a mode that uses a classifier to judge from the words themselves whether the caller has actually finished the thought rather than just paused mid-sentence. Waiting for the thought to close, instead of for the audio to go quiet, is what keeps a mid-revision "forty" from being treated as an answer. And because the result is a proposal and not a committed field, when a caller does revise, the proposal revises in place, which is expected behavior rather than a surprise overwrite.

Second, every proposal carries evidence. When the model proposes a value, it also has to point at the span of transcript that value came from, and attach how sure it is. This is the part that kills the Phantom. A value with no transcript span behind it is, definitionally, invented, and the system can now tell the difference between "the caller said four hundred thousand" and "the model felt like four hundred thousand." I treat the grounding, whether the value traces to something in the transcript, as the real signal, and the model's own stated confidence as a weak secondary hint, because a model can be fluent and wrong in the same breath. Proposals that don't clear the bar don't get shown as confident fields. They get held back and surfaced as "I didn't catch this one, you fill it," which is the honest thing to say and, it turns out, the thing operators trust.

Third, the operator confirms. A proposal is visibly a proposal, not a filled field, until the operator accepts it. Most proposals are right and accepting them is one gesture, so the operator isn't retyping, they're ratifying. But the screen never lies to them about what's been checked and what hasn't. A confirmed field looks confirmed. A proposal looks like a proposal. A held-back field looks like a hole. The operator's attention goes exactly where the uncertainty is, instead of being spread flat across a form where everything looks equally done.

The through-line, and the reason I keep coming back to it, is the same conviction that drove the human-in-the-loop gates in DaVinci: the human isn't a nag bolted onto a finished system. The human is a load-bearing step, and the system's job is to route their attention to the fields that actually need it. Get that right and the tool is fast because the human is trusting it correctly. Get it wrong and you've built the Phantom's delivery mechanism.

How it works now

Today the app runs on the operator's desktop and captures the call audio locally. That it's Electron is not incidental. A voice tool has to live where the call already lives, capture audio off the machine reliably, and sit in an always-there window beside whatever else the operator has open, rather than competing for attention as one more browser tab that can be closed or backgrounded. A desktop shell that owns its own window and captures audio through the platform's media APIs fit those constraints in a way a web page did not.

Audio streams to the Realtime API over its low-latency transport, negotiated the same way a browser sets up a peer connection. The model both transcribes and extracts, and turn detection decides when a stretch of speech has settled enough to extract from. Each settled utterance produces zero or more field proposals, each with its transcript span and a confidence. Grounded, confident proposals show up in the form as pending values the operator can accept in a gesture. Ungrounded ones get held back and marked as needing a human. Only accepted values commit.

The operator is on the call the whole time, so a correction is a normal part of the conversation. Caller misspeaks, operator sees the proposal update, confirms the corrected one, moves on. What used to be "listen, type, scroll back, fix, apologize" is now "listen, glance, confirm." That's where the roughly 70% reduction in manual intake work comes from. The machine isn't doing 70% of the thinking. It's doing the typing, and the operator is left with only the judging, which was always the smaller job.

The honest residue

The redesign fixed the two failures it was built to fix. It did not make the tool perfect, and a few rough edges are load-bearing enough that I'd be lying to leave them out.

Held-back fields are a cost, not a free win. Every time the system says "I didn't catch this, you fill it," it's handing work back to the operator. Set the grounding bar too conservative and you've built a tool that shrugs at everything and makes the human type it all anyway. Too permissive and the Phantom creeps back. That bar is a dial with a bad failure mode on both ends, and I don't think there's a setting that's right for every call, because the calls aren't the same: a crisp caller on a good line and a mumbler on a bad one want different amounts of caution, and the tool is choosing one policy for both.

Accents, cross-talk, and bad phone lines are still where accuracy drops, and those conditions aren't evenly distributed across callers, which is a fairness problem hiding inside an accuracy problem. The tool is most confident exactly where the audio is cleanest, which is not the same population as the callers who most need to be understood. I don't have a clean fix for that. The propose-don't-commit design at least routes the harder calls toward more held-back fields rather than more silent Phantoms, so the failure mode on a bad line is "the human types more," which is the survivable one. But a tool that quietly asks more of some callers' operators than others is still doing something I'd want to measure before I'd want to defend.

And the tool is, by design, only as good as the operator's willingness to actually look. The redesign routes attention to uncertainty, but a tired operator at the end of a shift can still rubber-stamp a screen full of proposals. The system makes the right thing easy. It can't make it mandatory without becoming the slow thing again, and I don't think the answer is to make it mandatory. A tool that forces a human to re-verify every field has reinvented the typing it was supposed to remove.

What operators actually did

The part I got most wrong was predicting how operators would react to the tool admitting it wasn't sure.

My assumption going in was that operators would want the tool to seem confident, that visible "I'm not sure" states would read as the tool being weak, and that hiding the uncertainty would make it feel more finished. The opposite held. A tool that admits when it didn't catch something is a tool you can hand the boring majority of the work to, precisely because you know it'll tap you on the shoulder for the part it couldn't. The held-back field, the thing I'd assumed operators would resent, is the thing that let them stop double-checking everything else. Confidence theater would have gotten exactly one Phantom through before they stopped trusting any of it, and a tool you've stopped trusting is a tool you've gone back to typing around.

That's the counterintuitive win, and it's the one I'd carry into any tool like this: making the uncertainty visible didn't cost trust, it bought it. People don't need the machine to be right about everything. They need it to be honest about which things it isn't sure of, so they can spend their attention there and nowhere else.

What I'm still not sure about

I'll end where I'm honest with myself, which is a list of things I believe are true and am not certain about.

I'm not sure the desktop app was the right call long term. It solved the audio-capture and always-there problems cleanly, but it's a heavier thing to ship and update than a web surface, and if the call platform ever exposed the audio stream in a way a browser tab could reliably grab, a lot of the reason to be on the desktop would evaporate. I'd want to rebuild it the moment that constraint changed, and I'm not certain I'll notice the moment it does.

I'm not sure confidence from a language model means what I want it to mean. I lean on "grounded in the transcript" as the real signal and treat model-reported confidence as a weak secondary hint. But grounding has its own hole: the model can ground a value in a transcript span that the transcription itself got wrong, and then the Phantom comes back wearing evidence. A confident, well-grounded, wrong field is the hardest kind to catch, because everything the system checks for says it's fine, and I don't have a clean answer for that one. The best I've got is that a transcription error is at least a different failure than a hallucination, and sometimes it's audible enough on the call that the operator catches it live. Sometimes.

I'm not sure where the line between copilot and autopilot should sit. Every held-back field is a small admission that the tool won't fully close the loop, and there's constant pressure, from myself as much as anyone, to let the confident proposals just commit without a human. The whole argument of this write-up is that you shouldn't. But I notice I want to, on the easy calls, and I don't fully trust my own judgment about which calls are easy. That's the same discomfort I wrote about when I argued that agents need a nervous system before they earn autonomy, and Dumbly is the version of that argument with a human's voice on the other end of the line.

What I am sure of is the one thing the two failures taught me. A system that turns speech into structure is going to guess, and the entire design problem is not making it guess less. It's making it tell you, honestly and in the right place, when it just guessed. Everything good about Dumbly is downstream of that.