## Inspiration
Sundowning hits up to 66% of Alzheimer's patients. It's that late-afternoon window where they get confused, restless, scared. If you're a family member living hours away, you don't know it's happening. Cameras feel invasive. Wearables get pulled off. We wanted something that could sit in the room without feeling like a medical device. A teddy bear felt right. There was a show about a horse named Mr. Ed who could talk, but only to the people who needed to hear him. We built a bear that does the same thing.
## What it does Ed sits on the couch with your grandmother. When she gets restless, he notices through accelerometer, touch, microphone, and heart rate sensors. He responds with her grandson's recorded voice, guided breathing through a haptic motor, or a warm LED pulse. Then he tells the family what happened in plain language. Before that message reaches anyone, three AI critics review it: is it medically sound, will it scare the daughter, does it say too much. The full debate is logged. A real-time dashboard shows agitation patterns, episode history, circadian heatmaps, and a daily digest that reads like a note from a caregiver, not a spreadsheet.
## How we built it
Three ESP32s inside a bear. One reads the accelerometer and capacitive touch pads. One runs the microphone through voice activity detection, Whisper transcription, and emotion classification. One plays TTS audio through a speaker. Each connects to a Python bridge over USB serial, which forwards everything to a FastAPI backend over WebSocket. The backend runs a LangGraph agent pipeline: perception translates raw sensors to plain English (following the IoT-LLM paper), risk scores agitation with a sundowning time multiplier, memory retrieves what worked before using a three-tier system with Thompson sampling, the planner picks a comfort recipe via Claude Sonnet, and the executor fires TTS through Cartesia and runs the MAR safety gate on notifications. The dashboard is Next.js 15 with server-sent events for live updates.
## Challenges we ran into
Serial audio was the hardest part. At 921600 baud, data arrives three times faster than the ESP32 can play it. We went through five firmware rewrites before landing on raw PCM with no framing protocol and a 32KB ring buffer on a dedicated core. Our original approach used sync markers, but the byte sequence kept appearing inside the audio data and corrupting the stream. We also dealt with zombie processes on Windows, three USB serial ports fighting for attention, and WebSocket connections that silently died because blocking serial reads were starving the async event loop.
## Accomplishments that we're proud of
The MAR gate works. Three AI critics review every notification in parallel, rewrite it if needed, and log the full debate. You can expand any notification on the dashboard and see exactly why it was worded the way it was. The comfort recipe memory system actually learns. Thompson sampling ranks interventions by success rate, so Ed gets better at calming each patient over time. And the bear talks. Cartesia TTS streams through a 3W speaker inside a stuffed animal and it sounds warm, not robotic.
## What we learned
You can't send raw sensor numbers to an LLM and get useful output. The IoT-LLM translation layer that converts accelerometer jerk and touch states into plain English before the agent sees them was the single most important architectural decision we made. We also learned that flow control matters more than baud rate when streaming audio over serial, that pyserial's blocking reads will silently kill your async WebSocket connections, and that three AI critics reviewing a message before it reaches a worried daughter at 4am is not overkill.
## What's next for Mr.Ed — The Horse that Cares
Docker Compose for one-command deployment across multiple bears in a care facility. Privacy-preserving computer vision using thermal cameras and depth sensors for fall detection and sleep monitoring, with no RGB cameras ever. FHIR data export for clinical EHR integration. Voice cloning through Cartesia so Ed can speak in a family member's voice. And circadian-aware pre-emptive interventions that start calming routines before the sundowning window even opens.
Log in or sign up for Devpost to join the conversation.