MoodyTunes

You walk into the 2nd floor of CDS and notice a monitor that says, "What song are you feeling right now?" You walk up to it and click the "Take a Picture" button. Suddenly, a tune plays for your mood!

Comment

Inspiration

MoodyTunes was born out of the excitement and challenges presented by the hackathon's Spark subtrack. We were particularly inspired by the examples that explored creative uses of music and visual emotion detection, especially the "Room Mood Visualizer." This immediately sparked the idea to combine these concepts into a practical, engaging product: an application that could instantly analyze a person's current emotional state via camera vision and respond by curating a perfectly corresponding song, bridging the gap between mood and music.

What it does

MoodyTunes is a powerful "Music Emotion Based Application" designed to provide an instant, personalized soundtrack to the user's feelings.

The seamless, five-step process is as follows:

  1. Capture: The user grants access to their webcam and captures a photo of themselves directly within the application's simple, clean interface.
  2. Submission: The application converts the captured image into base64 data and sends it to the backend server.
  3. AI Analysis: The powerful Gemini 2.5 Flash Lite model analyzes the photo for emotions and is prompted to suggest a song title and artist that perfectly matches the user's emotional state.
  4. Music Lookup: The suggested song title and artist are used to automatically search the Spotify Web API for the exact track.
  5. Playback: The application retrieves the Spotify track ID and displays the result with an embedded, immediately playable Spotify player.

How we built it

MoodyTunes is a robust, Python-based web application built on the Flask framework. The core of the project is a complex integration of multiple external APIs:

  • Front-end: A minimal, responsive interface uses vanilla HTML, CSS, and JavaScript. The client-side logic handles camera access and image capture via the HTML5 getUserMedia API.
  • AI Emotion Analysis: We leveraged the Gemini API with the powerful gemini-2.5-flash-lite model. We designed a custom prompt to force the model to respond with a clean array containing only the suggested song and artist, which was critical for the next step.
  • Music Integration: The Spotify Web API was integrated using the spotipy library. This component handles authentication and searches the Spotify catalog to retrieve the unique track ID required for the embedded playback widget.
  • Dependencies: Key Python dependencies included flask, google-genai, spotipy, and python-dotenv for secure environment variable management.

Challenges we ran into

The primary hurdles we encountered stemmed from the complexity of chaining multiple external API calls seamlessly.

  1. Google Gemini AI API Parsing: The core difficulty here was reliably parsing the unstructured text response from the Gemini model. We needed the model to return a perfectly formatted song title and artist, which then had to be extracted and cleaned so that it could serve as compatible, accurate arguments for the next API call to Spotify.
  2. Spotify API Embedding: The second major challenge was with the final output: embedding the Spotify player directly into the web application and ensuring it would play the selected track immediately upon loading the results page. This required careful manipulation of the Spotify IFrame API to ensure a smooth, functional user experience.

Accomplishments that we're proud of

We are most proud of successfully achieving the seamless, end-to-end user experience. Specifically, the technical triumph of embedding the Spotify player into our web application and having it play the suggested track immediately upon navigating to the results page was a huge accomplishment. This proved to be the most difficult and time-consuming part of the project, and getting it to work reliably validated our entire multi-API integration approach.

What we learned

MoodyTunes was a fantastic opportunity for growth. We gained practical experience in:

  • AI as a Teammate: We learned how to use AI in our coding workflow not just as a service or a simple function call, but as a more collaborative tool that actively shapes the application's core logic.
  • New API Mastery: The entire group gained valuable hands-on experience experimenting with and mastering two major external services—the Google Gemini AI API and the Spotify API—both of which were new territories for us.
  • Full-Stack Integration: We significantly improved our skills in combining front-end (HTML5 camera access and JavaScript embedding) and back-end (Flask and Python API handlers) technologies to create a cohesive, functional product.

What's next for MoodyTunes

Our immediate next step is to transition MoodyTunes from a hackathon prototype into a public, accessible utility. We plan to deploy the application with functionality to run 24/7 at the Spark! Center at CDS. This will allow the application to provide continuous, real-time emotion-based music recommendations for anyone visiting the center. Future features could include: tracking a user's emotional history over time, creating a dynamic playlist based on emotion shifts, or utilizing a more granular emotion detection model.

Link to Presentation: https://docs.google.com/presentation/d/1Mce40Rldm9pBF8cAI6zN9MKRAweg894Ui1hPYMf1F2g/edit?usp=sharing

Built With

Share this project:

Updates