MoodMuse: Emotion-Aware Music for Mental Wellness
Inspiration: Mental Health Through Music
In today’s fast-paced world, emotional wellness often takes a back seat. We believe music can help change that. Mental health is deeply personal—and so is music. Whether you're overwhelmed, anxious, or just having an off day, the right song can ground you, comfort you, or even bring you back to yourself.
As a team of tech enthusiasts who care deeply about mental well-being, we saw a powerful opportunity: what if music could respond to how you’re really feeling in the moment? MoodMuse was born from that question. Our goal was to build a system that passively supports mental health by detecting emotional distress and responding with music that soothes, uplifts, or energizes—depending on what you need most.
What It Does
MoodMuse is a context-aware, emotion-responsive music companion. Using a standard webcam and machine learning, it recognizes your facial expressions and real-time emotional state—like stress, sadness, or calm—and responds with curated music to match or gently shift your mood.
Whether you're a parent caring for a newborn, a patient in therapy, or a student working late, MoodMuse picks up on non-verbal cues and adapts to help. It's music that listens to you—before you even press play.
Demo Overview
We designed two core use cases:
Infant scenario: A baby begins to cry. The system detects distress using facial emotion analysis and instantly plays gentle lullabies to calm the environment—supporting both the child and caregiver.
Therapy session: A patient’s facial expression subtly shifts, showing signs of growing anxiety or sadness. MoodMuse detects this rising emotional state and starts playing soft, therapeutic background music to restore a sense of calm—enhancing the therapeutic atmosphere without interrupting the session.
Both scenarios demonstrate how technology can create emotionally intelligent spaces that support mental wellness without intrusion.
How We Built It
Our system is built for real-time emotion detection and music adaptation:
Emotion Detection: We used PyTorch and a pre-trained model from Hugging Face (
dima806/facial_emotions_image_detection) to identify core emotional states from video frames in real time.Multithreading for Performance: To maintain responsiveness, we implemented a two-thread system. One thread handles continuous facial emotion analysis, while the other runs the music mixer. This separation allows us to avoid processing delays and keep the audio experience in sync with the user’s current emotional state.
Music Integration: We’re currently using curated local audio files mapped to emotion labels. While Spotify integration is still in development, future iterations will pull playlists dynamically via the Spotify API for more personalization.
Frontend: A lightweight React interface displays camera status, detected emotion, and current music playback. It also includes a manual override option and basic user preferences.
Backend: Built in Python, the backend coordinates data flow between the facial emotion model and the music controller, using asynchronous logic to handle real-time updates smoothly.
Challenges We Ran Into
Thread Management and Real-Time Performance
Maintaining smooth, low-latency performance was one of our biggest technical hurdles. In early versions, the facial recognition and audio processing components ran on the same thread, leading to noticeable lag and dropped frames during video capture and playback.
To fix this, we split the system into two concurrent threads: one for the face analyzer and another for the music mixer. This allowed both processes to run independently and in parallel, reducing delay and making emotion-to-audio reactions feel significantly more immediate.
Thread synchronization, however, required careful coordination to ensure that emotion data was consistently passed between modules without creating race conditions or glitches. We fine-tuned the threading model to allow for consistent, reliable mood detection and corresponding music playback.
Smooth Music Transitions and Crossfading
Designing seamless musical transitions based on shifting emotions was another challenge. In particular, we struggled to implement smooth crossfades between tracks when a user’s emotional state changed. Without crossfading, the switch between songs felt abrupt and could break the immersive experience.
We explored several approaches using the Web Audio API and buffer manipulation but encountered limitations in achieving clean, real-time fades without audio artifacts. While we achieved basic fade-outs and fade-ins, continuous overlapping crossfades between two tracks with emotion-based triggers remains an area we’re actively improving.
Accomplishments We’re Proud Of
We built a system that goes beyond utility—it feels personal. MoodMuse reads subtle emotional cues and responds with music in a way that feels empathetic and human. That emotional connection is our proudest achievement.
Other highlights include:
- Implementing real-time emotion recognition with minimal latency
- Solving performance issues with multithreaded architecture
- Creating a calming, minimal UI that supports mental wellness
- Building a flexible foundation for future music and wellness integrations
What We Learned
We learned that designing for emotional impact requires both technical precision and a deep sense of empathy. It’s not enough for AI to recognize a frown—it has to respond in a way that feels natural, helpful, and non-intrusive.
Technically, we deepened our experience in:
- Real-time video processing with PyTorch
- Threading and parallelization for performance
- Audio system design and transition handling
- Model deployment and backend/frontend synchronization
What’s Next for MoodMuse
We’re excited to take MoodMuse further. Upcoming goals include:
- Integrating Spotify’s API for dynamic, emotion-aware playlist selection
- Building custom emotion and activity models for greater accuracy
- Adding support for audio-only cues (like tone of voice)
- Expanding compatibility with smart speakers and IoT devices
- Allowing users to train the system with their favorite tracks for personalized responses
- Exploring partnerships with mental health platforms to integrate music as therapeutic support
MoodMuse is more than a hackathon project—it’s a step toward building emotionally intelligent systems that understand and support us through music.
Log in or sign up for Devpost to join the conversation.