WALDO

Inspiration

We’ve all had smart home devices that turn into expensive paperweights the second Wi-Fi drops, or "smart" cameras that take two business days to process a frame in the cloud. We wanted to build something that felt alive, responsive, and completely self-reliant.

Our question: Can a single microcontroller act like a living, seeing, hearing robotic companion that's running real-time computer vision, voice recognition, and physical motor control 100% on-device with zero PC or cloud tether?

Enter WALDO: part Pixar’s Luxo Jr. lamp, part sci-fi desktop sentry, and 100% edge-powered.

What it does

WALDO is an autonomous, voice-commanded pan/tilt searchlight turret that patrols your desk or room, spots people and objects, and tracks them with a focused beam of light.

Tell it to "stop", "patrol", or nudge "left" / "right", and it snaps to attention. Ask it to track a "cup", "phone", or "bottle", and it locks on. Behind the scenes, WALDO runs two concurrent ExecuTorch neural networks entirely on a single Alif Ensemble E8 DevKit. No hidden laptops under the table, no AWS bills, and zero cloud latency.

How we built it

We designed WALDO around a "two brains in one chip" architecture, taking full advantage of the E8’s dual-core, dual-NPU silicon:

The Always-On Ears (HE Core): The Cortex-M55 High-Efficiency core never sleeps. It captures audio from onboard microphones and routes it into the Arm Ethos-U55 NPU, running a fine-tuned MicroNet-M keyword spotter to listen for 10+ voice commands in real time.
The High-Speed Eyes (HP Core): When duty calls, the 400 MHz Cortex-M55 High-Performance core spins up the Ethos-U85 NPU, running INT8-quantized YOLOv8n at a blistering 29 ms per frame on JPEG camera frames streamed over a 2 Mbaud UART link from an ESP32-S3 head.
The Reflexes: The two cores sync over a lockless, cache-coherent mailbox in SRAM0. When a target is acquired, a Kalman-predictive tracker drives the pan/tilt servos via hardware UTIMER PWM for butter-smooth, jitter-free movement.
The Chassis: A custom 2-DOF gimbal turret designed in OpenSCAD and 3D-printed overnight on a Bambu Lab printer.

Challenges we ran into

The "Float Tax": Initially, our YOLO inference was throttled by CPU float-conversion kernels (eating up over 30 ms on their own!). We kicked floating-point ops out of the critical path by building a 256-entry INT8 lookup table on the M55, cutting overall frame time in half to 29.7 ms.
Dual-Core Synchronicity: Juggling two cores with distinct clock speeds, memory topologies, and separate NPUs meant inter-core communication had to be bulletproof. Getting our SRAM0 mailbox to pass commands without race conditions or memory faults was a true trial by fire.
3:00 AM Mechanical Gremlins: Tuning servo deadbands so the turret didn’t violently vibrate itself off the desk, and fitting dual ExecuTorch runtime weights cleanly into tight embedded MRAM/SRAM budgets.

Accomplishments that we're proud of

3rd Place at the Arm + ExecuTorch Edge AI Challenge Hackathon!
Pulling off two concurrent neural networks on two separate NPUs on bare metal. Not in an emulator, but live on silicon.
Watching WALDO seamlessly sweep a room, lock onto someone walking past, and track them across the room while obeying verbal steering commands.

What we learned

Edge AI is an art of data plumbing, not just raw FLOPS. The biggest performance wins didn't come from squeezing model parameters as they came from optimizing memory layouts, bypassing CPU float conversions, and leveraging asymmetric cores to divide sensing and heavy compute.

Also: never underestimate how much you can build in 24 hours when fueled by coffee and 3D printer filament.

What's next for WALDO

Direct MIPI Camera Link: Swapping the UART bridge for direct native MIPI capture on the E8 to shave off even more latency.
Dialing in "Hey Waldo": Benchmarking and deploying our dedicated on-device wake-word model so WALDO only perks up when his name is called (and stops eavesdropping on hallway conversations).
Smart Assistive Spotlight: Expanding the object detection palette so WALDO can automatically illuminate tools on a workbench, track actors on a stage, or aid in search-and-rescue inspection!