Mira

Mira is a companion robot that detects real-time behavioral cues to identify distress or falls, helping caregivers stay connected and to support dementia patients when they can't be there.

Comment

Inspiration

There are over 55 million people worldwide living with dementia. One of the hardest parts isn’t just memory loss—it’s the confusion and disorientation that can happen in the moment, when caregivers or family members aren’t there to help.

We were inspired by that gap: what a patient is experiencing in real time versus what others are able to see and respond to. We wanted to build something that could stay with the patient and help bridge that disconnect.

What it does

Mira is a conversational companion robot designed for individuals with dementia, acting as both a support system for patients and a tool for caregivers and families.

It uses real-time, multimodal AI to both listen and see the user through audio and visual inputs. By observing cues like facial expressions, movement, and behavior, Mira can detect when something might be wrong—such as confusion, distress, or fall-like situations.

Mira responds with a warm, patient conversational style, while also translating AI intent into physical actions—adjusting its expression, lighting, or posture to show engagement and empathy. This allows it to both ground the user in the moment and help caregivers stay informed and respond faster, even when they’re not physically present.

How we built it

We built Mira as a system that connects sensing, processing, reasoning, and action.

  • On the frontend, we used React + TypeScript (Vite) to orchestrate the system.
  • For AI, we integrated the Gemini Multimodal Live API, continuously streaming microphone audio and camera frames to give the model real-time context.
  • We convert raw inputs into structured, time-based events, making the data more stable and meaningful for reasoning.
  • AI-generated responses are streamed back as audio and played seamlessly using a Web Audio scheduler, ensuring natural, uninterrupted speech.
  • For hardware, we used the Web Serial API to communicate directly with an Arduino, allowing Mira to control servos, LED states, and visual expressions.
  • A custom BehaviorRouter parses AI outputs into synchronized physical actions, enabling the robot to respond in real time alongside conversation.

Challenges we ran into

  • Handling noisy, real-time data from both vision and audio streams without overwhelming the system.
  • Managing bidirectional streaming of audio and video in the browser while keeping the UI responsive.
  • Implementing smooth, gapless audio playback for streaming AI responses.
  • Synchronizing AI output with hardware actions so the robot doesn’t glitch or respond mid-sentence.
  • Avoiding context overload by carefully throttling visual input to maintain stable AI performance.

Accomplishments that we're proud of

  • Built a working end-to-end prototype that connects real-time sensing, AI reasoning, and physical interaction.
  • Successfully bridged multimodal AI with hardware, allowing the system to both understand and respond in the physical world.
  • Designed a system that transforms raw signals into structured, meaningful behavior, rather than simple detections.
  • Created a modular and scalable architecture that cleanly separates AI, media processing, hardware control, and UI.

What we learned

  • Raw sensor data is not enough—structuring it over time is critical for meaningful interpretation.
  • Multimodal AI becomes significantly more powerful when combining audio and visual context.
  • Building real-time systems requires careful handling of buffers, streaming data, and synchronization.
  • Prompt design plays a major role—guiding the AI to be calm, patient, and human-like drastically changes the user experience.
  • Clear system architecture is essential when integrating AI, frontend, and hardware components.

What's next for Mira: Dementia Robot Companion

  • Improve detection accuracy and stability, especially for complex behaviors like falls or distress.
  • Integrate a C++ computer vision backend (OpenCV) to enhance visual cue tracking and maintain longer-term behavior history.
  • Incorporate hardware sensor feedback (motion, proximity, temperature) into the AI context for better awareness.
  • Expand caregiver tools with real-time alerts and monitoring dashboards.
  • Move toward fully embedded deployment on a portable device, allowing Mira to function independently as a real-world companion system.

Our goal is to evolve Mira from a prototype into a system that provides reliable, real-time support for both patients and caregivers.

Share this project:

Updates