ESu

Inspiration

A hospital stay can feel especially long when you’re five to nine years old. Between treatments and visits, children may spend hours without someone beside them to talk, laugh, or play with.

We wanted to make those quiet moments feel a little warmer. That inspired ESu: a little companion for long hospital days.

What it does

ESu is an animated AI buddy that children can talk to and share little moments with.

It watches YouTube Shorts alongside them, remembers recent videos, and reacts with expressions and short comments. Children can ask questions, discuss something funny, or simply chat. ESu also knows when to stay quiet, giving children space to enjoy what they’re watching.

With optional camera features, ESu can respond to visible expressions. When a child points to a comment with the cursor, ESu can react to it and talk about it with them.

How we built it

We built ESu as a Chrome extension using TypeScript, React, and Vite, connected to a Python FastAPI backend.

The backend combines AI video understanding with Gemini Live for real-time voice conversations. A small memory connects recent videos, while a deterministic timing engine decides when a reaction should appear.

We use MediaPipe to process optional facial-expression signals locally, without sending webcam frames to the backend. The extension also detects which comment the child points to, giving ESu context for reactions and conversation.

Challenges we ran into

The hardest challenge was making ESu feel attentive without becoming distracting. Generating a response was only half the problem; choosing the right moment to deliver it mattered just as much.

We also had to coordinate voice, animation, camera processing, and Chrome’s extension lifecycle. Missing model files, incompatible dependencies, and stale connections showed us how easily a small technical issue can interrupt the experience.

Accomplishments that we're proud of

We brought together an expressive character, continuous voice conversation, video understanding, and memory in one working prototype.

We’re especially proud of the quieter details: ESu can connect moments across videos, change its expression, and wait before speaking. Those choices help it feel like a companion sharing an activity.

What we learned

Companionship depends on listening and timing as much as speaking. More reactions do not necessarily create a better experience.

We also learned to treat facial expressions as imperfect clues. Letting children point to a comment gives them a direct way to show ESu what they want to discuss.

What's next for ESu

We want to give children more ways to make ESu their own, with more avatars and personalised personalities, voices, and interests.

We also plan to bring ESu beyond YouTube Shorts to platforms like TikTok, and expand into articles, stories, and other text. Whether children are watching, reading, or exploring, ESu can be there to share the moment.