How to Build the Next Generation of Voice Model

You're talking to a voice assistant, you pause mid-sentence to find the word — and it cuts in. Or you ask something that needs a lookup, it says "hmm…", and then there are five seconds of nothing.

Cutting you off, and leaving you hanging. Every voice assistant so far has had both. They look like two separate bugs. They're the same structural problem.

Voice has always been an adapter for something else

The first generation ran a relay: speech into text, text into a model, the model's text back into speech. Three models in a chain. It was the first time anyone could talk to a frontier model, and the cost was written into the structure — tone, hesitation, overlap, all of it disappears the moment it passes through text. Every handoff also costs time, which is why it was slow and why it sounded stilted.

The second generation collapsed the three legs into one model that eats audio and speaks audio. Much faster, much smoother. But it still took strict turns: it had to be sure you were finished before it started. How does it know? It listens for you to stop. Which means the two seconds where you're searching for a word and the two seconds where you're genuinely done look exactly the same to it.

Both generations were optimizing speed along the same road. Speed was never the problem.

First thing we got clear on: people don't take turns

Almost everything that makes a real conversation feel natural happens in the overlap. You're talking, I'm going "mhm." I hear you winding down and come in early. You say something wrong and I cut in immediately.

None of that is "faster turn-taking." On a single lane, you can optimize every segment to perfection and still never produce overlap — overlap simply isn't on that road.

So the first decision: input and output have to run at the same time. Able to speak while listening, able to listen while speaking. The model stops working in "receive a chunk, return a chunk" mode and instead listens continuously while continuously deciding: speak now, keep listening, hold, cut in, or reach for a tool.

That isn't a performance optimization. It's a different shape of time.

Second thing: speaking and thinking shouldn't share a brain

This one matters more.

The deadlock in the previous generation is that the voice model itself was the intelligence. Ask it something that needs a search or some real thought, and its only option is to stop the conversation and go do that. So the deeper it thinks, the more rigid the conversation gets — and you're forced to choose between smart and natural.

Split them and the tradeoff disappears. One up front handling the conversation, one in the back doing the work. When a question needs a lookup or actual reasoning, the front hands it off and keeps talking to you; when the result comes back, it folds into the conversation naturally.

Split that way, three things become true at once:

• The voice model is no longer the ceiling. Swap a stronger model in the back and the whole experience improves without touching the front.
• "How long to think" becomes a dial, not a downgrade. Depth is tuned on the back side; it doesn't touch the structure of the conversation up front.
• The front end gets to be good at one thing: holding the conversation.

That, we think, is the real dividing line for "next generation." Not a more human voice, not another few dozen milliseconds off the latency. This: conversational continuity and depth of thought stop charging each other rent.

Once those two are settled, a lot of the design grows on its own

Our framework is really just three lines running at once: you, something that talks, something that works. Plus a layer in the middle deciding who speaks, who goes quiet, and which sentence is allowed out.

The interesting part is that once you accept the two premises above, most of the remaining decisions don't need much debate:

Waiting can't be silence. During the seconds the back end is working, the front end has to still be there. Even "let me check on that" beats a gap — a gap makes people think it died.

Interruption is the main path, not error handling. The user can cut in at any moment; that isn't an edge case to cover, it's what the system looks like when it's working. And there's a subtle trap here: once you've been cut off, the answer nobody wants anymore has to be genuinely dead. What really breaks the illusion isn't "it didn't stop in time" — it's the thing you interrupted half a minute ago suddenly speaking up on its own.

No escape hatch. When the realtime path fails, dropping back to the old way — transcribe, think, synthesize — is the easy move. We decided not to leave that door open, because once it exists it becomes the default: it's cheaper, easier to wire up, and completely invisible in a demo. Reconnect, or say honestly that it isn't working right now. Don't pretend to still be working.

On safety, half of it can only happen live. What the back end concludes can be reviewed before it's ever spoken. But the little things the front end says on its own — "mhm," "let me look" — can't be vetted word by word in advance by any realtime voice system. That half can only be handled live: listen while it talks, steer when it drifts, end the call if it doesn't. Those are two different mechanisms, and blurring them turns into a promise you can't keep.

How you'd know it worked

You don't need specs, and you definitely don't need a demo — every system is natural in a demo. Put on headphones, talk for ten minutes, and watch four things:

Can you get a word in while it's talking. After you do, does the discarded answer resurface on its own. When you ask something that needs a live lookup, is anyone still talking to you for the next few seconds. And when something breaks, does it say so — or does it quietly drop back a generation and act like nothing happened.

What's next: Interaction Live

None of the above was written to be an essay.

Lumirain builds an AI coworker — one that delivers outcomes, not process. But there's still a layer between you and it: you have to compress a half-formed thought into a finished paragraph before you can even start.

Voice should be where that layer disappears. It doesn't occupy your hands and it doesn't ask you to think it through first. And voice only earns real work — rather than weather and alarms — once speaking and thinking stop dragging each other down.

Interaction Live is exactly that: talking to your AI coworker, seamlessly. You change your mind halfway through a sentence and it keeps up. It looks things up and gets work done in the background without the conversation stopping. When the work is finished, it tells you along the current thread of the conversation — instead of leaving you watching a spinner.

This isn't bolting a microphone button onto a product. It's wiring those three lanes into an AI coworker that actually does the work.

What we do not claim

Research writing puts its boundaries in the body, not in a footnote.

This is a piece about design thinking, not a performance report. There isn't a single benchmark number of ours anywhere in it — Interaction Live hasn't shipped, and we don't publish figures we haven't measured. When it does ship, we'll account for what it does and doesn't do in something you can hold us to.

Until then, this is our judgment on the problem, and nothing more.