Mirror Mind

Inspiration

Most AI assistants today focus only on what users say, not how they feel. Human communication is deeply influenced by emotions, facial expressions, and non-verbal cues. We wanted to bridge this gap by creating an AI companion that can recognize emotions in real time and respond with empathy.

MirrorMind was inspired by the growing need for accessible emotional support and the potential of multimodal AI. We envisioned a future where AI interactions feel more natural, human, and emotionally aware.

What it does

MirrorMind is a real-time emotion-aware AI companion that combines computer vision, generative AI, voice interaction, and 3D avatars.

The platform:

Analyzes facial expressions through a webcam
Detects emotions such as happy, sad, angry, surprised, and neutral
Uses emotional context to personalize AI responses
Provides empathetic coaching and wellness-oriented conversations
Displays a 3D avatar that mirrors detected emotions
Speaks responses using text-to-speech technology
Tracks emotional trends through an analytics dashboard

Unlike traditional chatbots, MirrorMind understands both the user's words and emotional state.

How we built it

MirrorMind was built using a modern AI-powered web stack.

Frontend

React
TypeScript
Tailwind CSS
Framer Motion

Computer Vision

We used MediaPipe Face Landmarker to process webcam input and track facial landmarks in real time.

AI Layer

Detected emotions are passed to an LLM, which adapts its responses based on emotional context.

Example:

{
  "emotion": "sad",
  "confidence": 0.91,
  "message": "I'm feeling overwhelmed today."
}

3D Avatar

We integrated a Ready Player Me avatar rendered with Three.js and React Three Fiber. The avatar changes facial expressions based on the detected emotion.

Voice Interaction

Speech-to-Text for user input
Text-to-Speech for AI responses
Lip-sync animation for natural avatar conversations

Analytics

Session emotions are stored and visualized using dashboards that display mood history and emotional trends.

Challenges we ran into

Real-Time Performance

Running emotion detection, AI inference, avatar animation, and voice synthesis simultaneously while maintaining a smooth user experience was challenging.

Emotion Recognition Accuracy

Human emotions are complex. A smile does not always indicate happiness, and facial expressions can vary significantly between individuals.

Synchronizing Multiple Systems

Keeping the following systems synchronized required extensive testing:

Webcam feed
Emotion detection
AI response generation
Voice synthesis
Avatar animations

Responsible AI Design

We wanted MirrorMind to provide support without acting as a therapist or making medical claims. Designing safe and empathetic AI responses was an important challenge.

Accomplishments that we're proud of

✅ Built a fully functional emotion-aware AI companion

✅ Successfully integrated computer vision with conversational AI

✅ Created a responsive 3D avatar that mirrors emotions

✅ Implemented voice interaction with speaking AI responses

✅ Designed a modern and engaging user experience

✅ Demonstrated real-time multimodal AI working together in a single platform

Most importantly, we transformed a complex idea into a compelling live demo that users can immediately understand and interact with.

What we learned

Throughout this project we gained hands-on experience with:

Real-time computer vision
Facial landmark tracking
Human-centered AI design
Prompt engineering
3D avatar development
Voice AI technologies
Full-stack application architecture

We also learned that combining multiple AI modalities creates experiences that feel significantly more natural than traditional text-based interactions.