Poco the Penguin

Inspiration

1 in 31 children in the U.S. is diagnosed with autism (CDC, 2025), and children with autism are 4× more likely to struggle with strong emotions (Conner et al., 2021). About half of people with autism also have trouble naming their own feelings (Kinnaird et al., 2019).

Yet emotion practice is still often a printed cartoon face and the same question over and over: "Is this person happy or sad?" Nothing responds, nothing adapts, and nothing about it feels like play. And recognizing "sad" on a flashcard is very different from spotting it on a real person's face, mid-conversation.

What it does

Poco is a plush robot penguin powered by AI, with three modes, all switched from an iPad:

Social mode: Poco's camera and mic face the friend the child is talking to, and everything Poco says goes to the child. Friend: "I've got three exams and I'm really stressed." Poco, to the child: "You could ask which exam worries them most." Poco's default is silence; it only speaks when it has something worth saying.
Teaching mode: tap a feeling and Poco acts it out, with the same face, color, movement and words every time. Lessons, calm-down tools like deep breaths, and custom feelings and moves (1,960 movement combinations per step).
Play mode: the child talks to Poco directly. Poco answers, dances and makes faces, and teaches one small piece of each feeling the way therapists do.

The iPad app is for the grown-up. A 7-step setup personalizes Poco to each child (what calms them, how they communicate), and a session log shows what Poco noticed, what it did, and why.

How we built it

iPad app → WebSocket → laptop server → USB serial → Arduino → 11 servos + 8×8 LED belly

Vision: OpenCV with the YuNet face detector and the HSEmotion emotion model, smoothed so Poco doesn't flicker between labels, at 30 fps
Speech: Silero VAD to detect speech and Whisper to transcribe it, fused with facial emotion on one timeline
Judgement: Meta Muse returns a schema-validated choice: what Poco should say (or nothing), a gesture, a belly color, and the reason
Voice: ElevenLabs v3, with Poco's tone matched to the moment
Memory: Backboard, opt-in, so Poco remembers the child between sessions
Hardware: Arduino Uno, a PCA9685 servo shield, 11 servos (11 degrees of freedom), 23 keyframed gestures, and an 8×8 NeoPixel belly
App: React + TypeScript + Vite, wrapped for iPad with Capacitor and Xcode

Challenges we ran into

Poco kept hearing itself. Whisper transcribed Poco's own voice as the friend's words, so we mute the mic while Poco speaks.
Emotion models are less reliable than they look. We swept 300 threshold combinations on recorded faces, and dropped "disgust" because the model couldn't tell it apart from anger.
Chatbot vs. coach. Early Poco answered the friend directly. Separate Social and Play modes with separate prompts fixed it.
Servo limits. Poco's arms rest at their limits, which flattened idle motion, so Poco now measures each servo's range at startup.

Accomplishments that we're proud of

Full loop on real hardware: face + voice → understanding → voice, gesture and belly, at a steady 30 fps
Vision + speech fused: Poco matches words to the speaker's face, so it catches "I'm fine" said with a sad face
Dynamic, not scripted: Poco's AI decides in the moment what to say, how to move, and how to help.
Built to move: 11 servos, 1,960 movement combinations per step, and a 64-LED belly showing 12 feelings

What we learned

Robotics: designing natural motion within real physical limits: calibrating each of Poco's 11 servos, building keyframed gestures, and keeping movement smooth and safe
Computer vision and AI: how pretrained emotion models behave on real faces, and how to tune them with our own recorded data instead of guessing thresholds
Designing for human connection: building technology that supports a relationship between two people instead of replacing it, which is why Poco coaches the child rather than talking for them

What's next for Poco

Run the AI fully on Poco, no laptop needed, with a local model for families who want nothing to leave the room
Pilot Poco with kids, teachers and therapists, starting with our teammate's brother
Help more children: ADHD, anxiety and speech delays