How Do Voice Assistants Actually Understand You?

A voice assistant does not understand you the way another person does. It converts the sound of your voice into text with a neural network, then runs a second set of models that classify what you want from that text, and finally speaks its answer back to you with synthesized speech. Every exchange — wake word, transcription, interpretation, reply — is a relay race between small models on the device and giant ones in the cloud.

The journey from sound wave to answer

Each interaction follows a four-stage chain. First, wake-word detection wakes the device when it hears a trigger phrase. Then automatic speech recognition, or ASR, turns the spoken audio into a written transcript. Next, natural language understanding, or NLU, reads that text to work out what the user wants. Finally, text-to-speech, or TTS, converts the answer back into audible speech. Audio goes in one end, meaning is extracted in the middle, and audio comes out the other end. The chain matters because each stage solves a different problem. Recognizing words is not the same as grasping intent — "play the weather" and "what's the weather" share words but demand different actions — and modern assistants handle that split explicitly. This division is part of a broader pattern in machine intelligence, where narrow systems master one kind of pattern at a time, as neural networks learn to master one pattern at a time rather than thinking like a mind.

The tiny model listening for a name

Before any of the heavy lifting starts, a small model stands guard. Wake-word detection, also called keyword spotting, keeps the device idle until it hears its trigger phrase — "Alexa," "Hey Siri," and their cousins. Incoming audio is first preprocessed: noise is reduced and the sound is converted into compact features, such as Mel-frequency cepstral coefficients, that capture how speech sounds to a human ear. A compact neural network then scores a sliding window of audio against a confidence threshold, firing only when the pattern matches. Engineers judge these guards on two error rates: false acceptance, where the device wakes with no real trigger, and false rejection, where it sleeps through a genuine one. False activations are a familiar nuisance — a television ad saying "Hey Google" can set living rooms of devices off at once — because the model only hears a sound pattern, not a speaker's intent. Nothing about this stage is intelligent in the human sense, and that is the point. The guard model is small enough to run continuously on almost no power, and until it fires, the assistant has no idea what anyone said.

How neural networks learned to hear

Speech recognition was not always neural. Before about 2012, systems chained three separately tuned pieces: an acoustic model mapping short audio frames to phonemes, a pronunciation model joining phonemes into words, and a language model judging which phrases were plausible. Each piece was optimized on its own, and errors compounded down the chain. Deep learning broke the chain apart and rebuilt it. Neural networks — first deep neural networks, then recurrent and convolutional variants — replaced the old statistical acoustic models from 2012 onward, driving a long run of accuracy gains in products like Google Voice Search. Around 2014, researchers began training a single network to map audio waveforms straight to output sentences, giving rise to end-to-end architectures like listen-attend-spell and RNN-Transducer. The payoff was models that could shrink onto the device itself. In 2019, Google described an all-neural on-device recognizer that streamed its output character by character as the user spoke, matching the accuracy of the company's old server-side models. The old system had relied on a roughly 2-gigabyte search graph; the new one was a single 450-megabyte neural network, compressed to 80 megabytes through quantization — roughly a fourfold shrinkage and a fourfold runtime speedup — small enough to run on a single phone core with no network at all. Amazon took a similar path for Alexa. Its on-device recognizer is an end-to-end RNN-Transducer mapping the speech signal directly to words, trained with a teacher-student trick: a small model learns to mimic a bigger, slower one, using up to a million hours of unannotated speech. Quantization-aware training and weight sparsification then shrank it further, co-designed with Amazon's AZ neural edge processors, which natively handle the 8-bit and lower-bit math these compressed models need. One elegant detail survives from the cloud era. ASR does not produce a single answer; it produces a ranked set of hypotheses about what was said, represented as a lattice — a graph of candidate words with the probability that each follows the last. Because words late in a sentence can change the best guess for the whole thing, the transcript is only finalized once the speaker finishes. A dedicated "end-pointer" model decides when that moment arrives; Alexa runs two of them, a speculative one tuned about 200 milliseconds faster to kick off understanding early, and a slower, more accurate one that can correct it.

From transcript to meaning: intents and slots

Once the transcript exists, the audio is done — NLU works on text, not sound. A series of models classifies the intent behind the words and fills slots with the details. "What's the weather in Lahore tomorrow" becomes intent: weather query, with slots for place and date. Amazon built its MASSIVE dataset to train exactly these two tasks: one million labeled utterances across 51 languages, spanning 18 domains, 60 intents, and 55 slots.

Large language models have since pushed the frontier further. Alexa+, announced in early 2025, was rebuilt around a new architecture that routes each request among several large language models — Amazon's Nova and Anthropic's Claude, served through Amazon Bedrock — picking the best one per task to balance speed and reasoning quality. Since language models do not natively call APIs, Amazon built an orchestration layer that lets them drive tens of thousands of services, stringing multiple API calls together: booking a lunch reservation, then texting the plan to a friend. Grounding systems, built with news partners including the AP and Reuters, keep the models tethered to real-time facts instead of hallucinating. The return trip has its own story. The assistant's text answer is spoken aloud by text-to-speech, and here too scale has unlocked surprises: Amazon's experimental BASE TTS model, trained on 100,000 hours of speech with roughly 980 million parameters, showed emergent abilities it was never explicitly taught — natural prosody on complex sentences, compound nouns, emotion, foreign words, even punctuation and questions — and it can generate speech streamable moment by moment rather than one whole sentence at a time.

The mic is not recording everything

The most persistent myth about voice assistants is also the easiest to deflate. The microphone is technically always on, but until the wake word is heard, nothing is recorded, stored, or transmitted. Audio is analyzed locally for the trigger sound pattern and discarded; only after detection does the device record the command and send it to the company's servers. It does not understand speech the way people do — it converts sound to text, then classifies intents from that text. When a television line that merely sounds like the wake word triggers an accidental recording, it can feel like the device was listening all along, but the mechanism is a pattern match, not eavesdropping. The industry is also pushing more of the pipeline onto the device. Amazon's newer on-device processing handles voice requests without sending them to the cloud at all; even in hybrid mode, only the text-like hypothesis lattice — not raw audio — goes up for a final language-model rerank. The drivers are practical: lower latency, less bandwidth, and the ability to work without connectivity. And for the truly cautious, there is always the physical mute button, which electronically disconnects the microphone. So the next time you ask a question out loud and an answer comes back, picture the relay: a tiny guard that knows one sound, a neural network that turns your voice into ranked guesses about words, models that classify what those words mean, and a voice built from a hundred thousand hours of human speech reading the answer back. None of it understands you — and yet, working together, it gets remarkably close. Much like the humble autocorrect that quietly fixes your typing, the magic is not comprehension but prediction, refined at scale.