Mood Theremin

Meet Our Team! (L-R: Tri, Ryland, Yurie, Caleb)

Inspiration

The theremin, an electronic musical instrument invented in the 1920s, is played without physical contact—pitch and volume are controlled by moving hands near two antennas. Its eerie sound has been featured in various music genres and movie soundtracks.

What it does

Mood Theremin translates your facial expressions into music - no musical theory needed! Smile, frown, or be surprised – your emotions become beautiful chords and melodies in real time. Explore your feelings and get inspired through music!

How we built it

Our project integrates facial expression analysis, emotion sentiment analysis, machine learning, and audio synthesis to generate music based on users' facial expressions. Through video input, we captured user's facial expressions, analyzed emotional sentiments using the HUME streaming API, and fed this data into a Scikit-Learn K-Nearest Neighbors (KNN) regressor model. We trained the model real-time and it translates facial expressions into frequencies for a 7th chord and a corresponding volume level based on the intensity levels of users. Then, the chords are played using the Web Audio API, creating a unique melody that mirrors user's emotions.

Challenges we ran into

Integrating different components seamlessly, ensuring smooth data flow.
Fine-tuning the machine learning model to accurately capture emotional nuances.

Accomplishments that we're proud of

Getting this to work from a concept in 24 hours!
Integrating multiple elements, emotion analysis, machine learning models, and music generation.
Generating an immersive 3D user interface for a rich interactive experience.

What we learned

Enhanced our respective skillsets in integrating diverse technologies into a cohesive project.
Deepened understanding of facial expression analysis and emotion sentiment interpretation.

What's next for Mood Theremin

Multi-user Support: Adapting the model for multiple users, considering complex emotional interactions.
Emotion Sentiment Refinement: Fine-tuning emotion sentiment analysis for greater accuracy.
Diversifying Audio Output: Add different classes of instruments (piano, string ensemble, etc.)
User Experience Improvements: Enhancing the interface and feedback for a seamless user experience.
Customization options: Offer the user options to customize certain chords corresponding to certain emotions.

Built With

Submitted to

Cal Hacks 10.0

Created by

I developed the backend algorithm responsible for translating emotional sentiment vectors into frequencies and volume levels. My efforts included experimenting with KNN regressor, random forest, and neural network models. After exploration under limited time, I found that the KNN regressor outperformed others in terms of mean squared error for the volume output channel, which is the most valuable one as the pitches are dependent on it.

Additionally, I helped with the debugging process for establishing the connection between the backend and our frontend interface.

Ryland Hu
I worked on the front-end component for this project. This was my first time integrating the front end and the back end, and it inspired me to further develop this using 3-D animations in the future.

Tri Pham
I built the back end service to connect our machine learning model to our front end. This involved creating a Flask service to accept HUME data and responded with frequency/amplitude information from our model.

I also wrote the code that fetches data from the HUME api and creates a request to the back end.

Furthermore, I wrote all of the audio synthesis code using the WebAudio API.

Caleb
Yurie Uratsu

Updates

Ryland Hu started this project — Oct 29, 2023 01:16 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.