What Full-Duplex Voice Agents Must Coordinate Beyond Speech

Real-time voice assistants are often judged by how quickly they begin speaking. That is useful, but it misses the harder problem: deciding whether to speak at all. A person may pause to think, say a brief acknowledgment, address somebody else, revise a request, or interrupt with a genuinely new priority. An assistant that treats every sound as a turn boundary can be fast and still feel clumsy.

MOSS-Live, introduced by the OpenMOSS team, presents full-duplex interaction as a coordination problem rather than a speech-output feature. The system is described as combining simultaneous listening and speaking, live visual input, selective attention, and background tool execution. Its central design separates the conversational foreground from longer-running work, while sharing context and task state between them. That framing offers a useful checklist for anyone evaluating a real-time agent, even when the underlying implementation differs.

Two timelines, one conversation

A spoken exchange and a tool call run on different clocks. Conversation unfolds in fractions of a second: the assistant must notice an interruption, interpret a backchannel, and decide whether to continue. Research, booking queries, or multi-step reasoning can take seconds or minutes. Serializing those activities forces the user to wait whenever a tool is active.

A foreground/background design avoids that artificial queue. The foreground remains responsible for turn-taking and immediate context. The background owns work that can continue independently. A coordination layer then decides when a completed result should re-enter the dialogue. The important detail is not merely concurrency; it is controlled re-entry. A result delivered during an unrelated thought can be as disruptive as an unnecessary interruption.

Consider a travel-planning request. While a hotel search runs, the user asks whether a ferry accepts cars and then changes the destination. A capable coordinator should keep the ferry discussion responsive, determine whether the destination change invalidates the hotel task, and either cancel, revise, or label the old result. Simply running both threads in parallel is insufficient. Their assumptions must remain visible to the system.

Full duplex is a policy, not a microphone setting

Simultaneous audio input and output provides the channel, but conversational behavior comes from policy. The agent needs to classify events such as an acknowledgment, a hesitation, unrelated nearby speech, and a direct interruption. Each classification implies a different action: continue, wait, ignore, yield, or ask for clarification.

This makes error costs asymmetric. Yielding unnecessarily is mildly awkward but usually recoverable. Ignoring a correction can cause the assistant to continue with invalid assumptions. Responding to background speech can expose private context or trigger unintended actions. Product teams therefore need thresholds and recovery behavior that reflect the application. A casual companion, an accessibility tool, and a voice interface for financial operations should not share identical turn-taking rules.

Visual input adds another timing question. A camera frame may help resolve what “that building” means, yet visual evidence changes as the user moves. The agent should associate an observation with the moment and request that produced it, rather than silently treating an old frame as current reality. Streaming multimodality is most useful when temporal provenance is part of the state.

Reading the benchmark claims carefully

The OpenMOSS article reports comparisons across six benchmark groups and 36 metrics. It says MOSS-Live tied the best tool-use success result on FDB v3 at 51%, placed second on the reported τ-Voice multi-turn task metric at 57.93%, and reached 88.40% overall accuracy on Big-Bench Audio. The tables also expose tradeoffs: no model leads every measure, and metrics for interruption response, resumption, latency, selectivity, and task completion can point in different directions.

That is the right way to read a full-duplex evaluation. A single composite score can hide a system that yields reliably but resumes poorly, selects the correct tool but supplies weak arguments, or answers quickly while reacting to speech that was not directed at it. Benchmark numbers are evidence about the tested conditions, not proof that an agent will behave safely in every room, language, microphone setup, or workflow.

Before adopting any system, teams should reproduce the scenarios that matter to their users. Test noisy speakers, overlapping voices, long pauses, corrections after a tool starts, and tasks that return out of order. Record both the classification and the visible recovery. A useful test log asks: what did the agent believe happened, what action followed, and could the user repair the mistake without restarting?

A practical evaluation sequence

Start with turn behavior before adding tools. Create a small script of directed questions, backchannels, hesitations, and speech aimed at another person. Measure false takeovers and missed interruptions separately. Then test whether the assistant resumes the correct thought after yielding.

Next, introduce one background task and deliberately change its assumptions while it runs. The expected outcome should be explicit: cancel the task, update it, or return the result with a warning that it uses earlier constraints. Add multiple tasks only after this state transition is dependable.

Finally, test result re-entry. A background answer should arrive when it helps, not merely when computation finishes. Let users postpone it, ask what assumptions it used, or discard it. These controls turn concurrency from a demo effect into a manageable interaction model.

MOSS-Live’s most useful contribution is the architectural question it highlights: how should an agent coordinate attention, dialogue, perception, and work that outlives a turn? Smooth speech matters, but reliable continuity depends on explicit state, selective responses, and recoverable handoffs between the foreground and the background.