The robot is an ESP32-S3, a microcontroller that costs a few dollars, with a microphone and a speaker. It talks to a voice model in the cloud. The obvious way to build that is to stream the microphone up to the cloud all the time and let the big model figure out when you are talking to it. That is also the wrong way. It is expensive, it is a privacy problem, and it means a chip on your desk is shipping your living room to a server every second of the day.
So I wanted a wake word: the microphone stays local and silent until it hears one specific word, and only then does it open the stream. Every good voice assistant works this way. "Alexa", "Hey Google", "Hey Siri" are all wake words. The point of this post is that the little model doing that job is much simpler than people think, and the hard part is not the model at all. It is the word, and the data.
The model is a pile of numbers
I used microWakeWord, the same open framework Home Assistant's voice hardware uses. The trained model is about 30,000 numbers and weighs 60 KB. That is it. It is not a speech recognizer. It does not turn audio into text. It is a binary classifier: it answers one yes-or-no question, many times a second.
"Do the last one and a half seconds of sound resemble the wake word?"
The audio never touches the model as raw sound. First a small signal-processing step turns each 10 milliseconds of audio into 40 numbers describing its frequency content, a kind of fingerprint of that instant. The model reads a rolling window of those fingerprints and outputs a single probability between 0 and 1. High means "that sounded like the wake word", low means "it did not". Because it only recognizes one pattern, it is small and cheap enough to run forever on a chip with half a megabyte of RAM.
40 frequency numbers every 10 ms go in. One probability comes out, every 30 ms. Thirty thousand weights in between decide the answer. Training is the act of setting those thirty thousand numbers.
How training actually works
At the start, those 30,000 weights are random, so the model is useless. Feed it the wake word and it might say 0.5. Feed it a cough and it might also say 0.5. Training is a loop that repeats, in my case, 12,000 times, and each pass does four things.
Guess, measure, find the downhill direction, step. Repeat until the guesses are good. There is no magic in the loop. Everything interesting happens in what you feed it and how you score it.
Two knobs matter here. I set a negative class weight of 30, which tells the loss to treat a false alarm as thirty times worse than a miss. A wake word that fires at random noise is far more annoying than one you occasionally have to repeat, so I deliberately bias the whole thing toward silence. The second knob is augmentation: every time a clip is used, it gets distorted differently, with room echo, background music, noise, and pitch shifts. A model with 30,000 free parameters is perfectly capable of memorizing a few thousand clips and then failing on anything new. Never showing it the same exact input twice forces it to learn the general sound of the word instead of specific recordings.
The data is the whole game
I never recorded my own voice for this. The positive examples are synthetic. A text-to-speech engine, Piper, generated a few thousand spoken versions of the wake word across roughly 500 different synthetic voices, speeds, and pitches. That variety is what lets the model generalize past any single speaker.
The negatives are the interesting half. General speech, music, and household noise, all labeled "not the wake word", teach the model what the world sounds like. But the real work is the hard negatives: clips of words that sound dangerously close to the wake word. The model only learns to separate the things you put in front of it. If it never hears a confusable word during training, it has no pressure to score that word low, and it will happily fire on it later. You steer the model by choosing its mistakes.
"Limbo" was a terrible wake word
My robot is called Limbo, so the wake word was going to be "Limbo". This was a mistake, and the model told me exactly why. My verification harness generates fresh, held-out clips the model has never seen and scores them. A good wake word scores high; everything else should score low. Here is what plain "Limbo" produced.
| Word | What it is | Peak score | Verdict |
|---|---|---|---|
| limbo | the wake word | 1.00 | fires, good |
| jumbo | rhyme | 1.00 | false alarm |
| banana | unrelated | 0.92 | false alarm |
| mumbo | rhyme | 0.79 | false alarm |
| hello | common word | 0.95 | false alarm |
| limousine | shared onset | 0.93 | false alarm |
The problem is structural, and no amount of training fully fixes it. "Limbo" is two syllables of very common sounds, and English is full of near-twins: jumbo, mumbo, combo, bimbo, gumbo. Every one of those differs from "limbo" only in the very first consonant, which is the single least reliable thing for a microphone to catch from across a desk. This is exactly why real assistants use long, distinctive triggers. "Alexa" and "Hey Google" have no rhymes. Short, common-sounding words do, and you cannot out-train physics.
One honest caveat: that table takes the single highest spike anywhere in each clip, which is the harshest possible read. The real on-device detector does not fire on one spike. It waits for the score to stay high across several frames in a row. So "hello" and "limousine", which spike once and settle, would mostly not trigger the actual device. But jumbo stayed high across whole clips. That one was real.
The fix: Hey Limbo
The cleanest fix was to stop fighting the word and change it. Not away from the name, but around it: "Hey Limbo". Adding "Hey" turns a weak two-syllable target into a distinctive three-syllable phrase. Jumbo, mumbo, and combo instantly stop colliding, because none of them start with "hey". The robot keeps its name, and the phrasing is exactly what people already say to Siri and Google. This is not a hack; it is why those products are named the way they are.
Retraining on the phrase also flips which hard negatives matter. The dangerous confusions are no longer jumbo and combo. They are other "hey" phrases: "hey there", "hey you", "hey Google", "hey guys". So those became the new hard negatives, along with the bare word "limbo" on its own, so the model insists on the full phrase and does not wake to just the name.
Alongside the model, I tightened the firmware. Instead of firing on any single frame crossing a threshold, the device now requires the score to stay above roughly 0.8 across a short sliding window, plus a cooldown before it can fire again. That smoothing kills the one-off spikes on its own, regardless of which word you train. The model and the firmware are two layers of the same defense.
Shrinking it onto a $6 chip
When training finishes, the 30,000 weights are stored as high-precision decimals. That is wasteful for a microcontroller. The last step, quantization, rounds every weight to an 8-bit integer. The model gets about four times smaller and runs far faster, for a tiny accuracy cost. That 8-bit version is the 60 KB file I flash to the device, where it runs an inference every 30 milliseconds using a few tens of kilobytes of working memory. There is plenty of room left for everything else the robot does.
The whole arc, then, is short: random numbers, then twelve thousand rounds of guess-measure- correct on carefully chosen positive and negative sounds, then shrink the result to integers small enough to live on a chip that costs less than lunch. The neural network was the easy part. The lesson that stuck was older than any of this: a model is only as good as the mistakes you teach it to avoid, and I should have picked a better name.