MoodMirror

Inspiration

Mental health awareness is growing, but most people lack tools to understand their emotional patterns. Traditional mood trackers rely on manual input, which is subjective and often forgotten. We wanted to create something that captures emotions naturally—through facial expressions, voice tone, and personal notes—using AI to provide genuine insights.

What it does

MoodMirror is an iOS app that performs multimodal emotion analysis using Google's Gemini 3 API. Users complete daily check-ins by:

1. Taking a selfie (analyzed for facial expressions)
2. Recording a voice note (analyzed for vocal sentiment)
3. Adding optional text notes (analyzed for context)

Gemini 3 synthesizes all three inputs simultaneously, returning emotion classifications across 8 categories with confidence scores and personalized AI insights. Over time, users can view their emotional journey on a timeline and receive pattern-based insights.

How we built it

• SwiftUI for a modern, responsive iOS interface
• AVFoundation for camera and audio capture
• Apple Vision & Speech frameworks for face detection and voice transcription
• Gemini 3 Flash API (gemini-3-flash-preview) for multimodal emotion analysis
• Core Data for local, privacy-first storage

Challenges we faced

• Threading issues with HealthKit integration on iOS required careful @MainActor annotations
• Ensuring the multimodal API call correctly includes base64-encoded images alongside text
• Balancing privacy (local storage) with AI analysis (API calls)

What we learned

Gemini 3's multimodal capabilities are remarkably powerful for combining visual, audio, and text signals into cohesive emotional understanding. The model's structured JSON output made integration straightforward.

What's next

• HealthKit integration for sleep/exercise correlations
• Apple Watch companion app
• Trend predictions using Gemini's reasoning capabilities

Built With

• avfoundation
• core-data
• google-gemini-3-api
• speech
• swift
• swiftui
• vision
• xcode