The Week My Desk Robot Started Talking Back

I've been building at the intersection of IoT and AI, and this week my little desk companion finally talked back.

It has eyes on an OLED, a mic, a speaker, and a wake word.

Say "hey limbo" and it wakes up, listens, and starts a real conversation.

Right now it can:

1. check, search, read, and summarise my emails
2. Be omnipresent across social media
3. Control my laptop by voice: "Open Excalidraw", "Play lofi", "Pause"
4. change between 10 facial expressions while talking
5. remember things across conversations
6. control its own volume, mute, sleep, etc.

connect to a PWA where I can see the live transcript, change its face, volume, WiFi, and control the overall harness

But the part I found most interesting is the wake word.

It runs completely on the ESP32-S3. No audio leaves the device until it hears "hey limbo."

I trained the model myself: ~30K parameters, 60 KB, 8-bit, running with TensorFlow Lite Micro.

And I learned two things the hard way.

1. A wake word is a modelling decision, not just a branding decision.
Initially I used just "limbo".
Bad idea.

It would confidently trigger on words like "jumbo", "hello", and even random sentences.

Adding one extra word made a huge difference:

Decoy "limbo" vs " hey limbo"
jumbo = 1.00 vs 0.16
hello = 0.95 v s 0.38

Turns out longer, more distinct phrases are much easier to separate acoustically.

2. Embedded "random crashes" usually aren't random.
The device kept rebooting while speaking.
Every crash pointed somewhere different: WiFi, LEDs, audio...
I spent almost a day blaming my ML model.

The actual problem was the SPI flash running at 80 MHz. Under WiFi transmit load, it occasionally returned corrupted data.

Since the ESP32 executes code directly from flash, corrupted flash reads basically meant corrupted instructions.

One clue finally gave it away: an illegal instruction at an address that should have contained valid code.

Dropped the flash speed to 40 MHz.
Stable ever since.

Still very much a prototype.

But it can now listen, think, remember, answer, and actually do things in the real world.

That's starting to feel pretty cool.