Designing a Reliable Offline Voice Pipeline for On-Device AI
Google AI Edge Gallery's new local speech features offer a useful reference architecture for combining streaming recognition, small language models, and speech synthesis without making the network part of every interaction.

Treat offline voice as a coordinated, interruptible pipeline and validate the complete experience under real device constraints before expanding model or language coverage.
Google AI Edge Gallery now demonstrates an end-to-end voice loop that can remain on the device: speech recognition turns audio into text, a small language model produces a response, and text-to-speech reads it aloud. The release covers an Android-oriented route through ML Kit as well as cross-platform options built around MediaPipe Tasks and LiteRT. It also packages several open-weight speech models into .litertlm bundles.
The most useful lesson is architectural rather than numerical. A local voice feature is not one model call. It is a pipeline whose perceived quality depends on how recording, partial transcripts, language generation, sentence boundaries, synthesis, playback, and cancellation fit together. Teams evaluating this approach should therefore test the complete interaction, not select components from isolated benchmark tables.
Why the modular pipeline matters
A single multimodal model may accept audio directly, but separate automatic speech recognition (ASR), language, and text-to-speech (TTS) stages create useful control points. An application can show partial words while the user is speaking, revise the transcript after an utterance ends, filter content before it is spoken, and swap a voice model without replacing the reasoning model. Each stage can also have its own fallback and resource budget.
That modularity has a cost: every boundary introduces buffering and state. The ASR system may revise its last few words just as the language model begins answering. The generator may emit Markdown that sounds awkward when read literally. TTS may be ready to play while the model is still consuming the device's accelerator or memory bandwidth. A robust design needs explicit contracts between stages: whether a transcript is provisional or final, which text is safe to vocalize, and what happens when the user interrupts.
A practical state machine can stay small. Treat listening, recognizing, generating, speaking, and interrupted as distinct states. Preserve final transcripts, but regard partial transcripts as replaceable. Start generation only from a stable segment unless the product truly benefits from speculative work. When new speech arrives during playback, stop queued audio first and then decide whether the new utterance continues or replaces the previous request.
Stream sentences, not arbitrary token chunks
Text generation and audio synthesis prefer different units. Language models stream tokens, while listeners need coherent phrases with stable pronunciation and prosody. Sending every token to TTS creates choppy playback and makes corrections difficult. Waiting for the entire response produces a long silent delay. Sentence-level scheduling is a sensible middle ground.
The application can accumulate generated text until it reaches a dependable sentence boundary, normalize it for speech, and enqueue that sentence for synthesis. Normalization should remove formatting markers and omit elements that do not translate well to audio, such as code blocks, raw URLs, and dense tables. The visual response can remain complete even when its spoken representation is shorter.
Sentence queues also make interruption predictable. Finished audio remains immutable, the current sentence can be stopped, and unsynthesized sentences can be discarded. For multilingual output, structured fields are safer than guessing language changes from punctuation. A response schema can identify explanatory text, target-language phrases, and display-only metadata, allowing the application to choose an appropriate voice for each segment.
Choose the integration layer from product constraints
The announcement presents three layers with different responsibilities. ML Kit offers platform-managed speech capabilities for supported Android environments. MediaPipe Tasks provides higher-level ASR and TTS interfaces across several operating systems and the web. LiteRT exposes more control for teams that need custom models or processing. These are not simply faster and slower versions of the same choice.
A platform-managed route can reduce application size and model-lifecycle work, but availability depends on the target device and service support. A cross-platform task API can make behavior more consistent across products, while still requiring teams to validate microphones, audio routing, and hardware variation. Direct runtime integration offers the most flexibility, but the application then owns more packaging, compatibility, and performance decisions.
Before choosing, define the supported device floor, required languages, acceptable download size, update policy, and whether offline operation must work immediately after installation. Also decide whether models may be downloaded later and how users will understand that state. “Runs locally” does not automatically mean “always available”; a missing language pack or model asset can still block the feature.
Evaluate the experience under contention
Model-level speed is only one part of responsiveness. Test time to the first stable transcript, delay from speech end to response generation, time to first audible output, and interruption latency. Run those measurements on representative low, middle, and high-end devices, both cold and warm. Averages can hide pauses, so record tail latency as well.
Then repeat while the complete pipeline is active. ASR, the language model, and TTS can compete for memory, thermal headroom, and accelerators. Long conversations may behave differently after several minutes of sustained use. Include background audio, Bluetooth devices, noisy rooms, rapid turn-taking, and screen-off transitions. Check battery drain and temperature rather than assuming local execution is free.
Accuracy review should include names, numbers, mixed-language phrases, accents, and corrections after partial transcription. TTS review should cover pronunciation, voice switching, sentence highlighting, and accessibility controls. Privacy claims need equally concrete testing: verify which assets are downloaded, which telemetry remains enabled, and whether any fallback sends audio or text to a server.
A useful reference, not a universal default
The Gallery release shows how local speech can turn an on-device model from a typed demo into a hands-free interface. Its strongest reusable ideas are modular stages, sentence-oriented synthesis, explicit multilingual structure, and integration choices at different abstraction levels.
The right production design will still depend on device coverage and product risk. Teams should begin with one narrow voice journey, instrument every transition, and make interruption and unavailable-model states first-class behavior. If that complete loop remains understandable and responsive under real device constraints, expanding to more voices, languages, and multimodal inputs becomes a controlled engineering decision rather than a collection of model demos.
Source: On-Device Speech Recognition and Text-to-Speech in Google AI Edge Gallery ↗. How we write


