Moodscapes

Inspiration

Any time I get into a car, the first thing I do is play music. But if I'm gonna be driving, how do I know the perfect song to play for my mood? Our program provides a soundtrack for your car ride. Rather than trying to improve or aggravate your mood, we just play the most appropriate music based on your current facial expression. Car rides are like a movie, and if the mood is hype, you need the perfect upbeat songs to keep you company. If it's a lonely solemn evening, you need the melancholy music.

What it does

We use a client-side facial expression recognition model to identify the user's live expressions as they change, and send the expression to our backend server implemented using python flask. We then use the OpenAI API to make use of the user's music preferences and the current mood and environment to generate the perfect music recommendations.

These recommendations are then sent back the client, implemented using React and Vite.js, to add the songs to the queue periodically. The user should be able to view, modify, and give feedback to the queue from the front-end.

How we built it

We have a separate client-side server running on Vite.js using mantine for front-end components that makes calls to the Spotify API and uses the Spotify Web SDK to instantiate a separate client instance. This means the music can also be controlled by other people in the car, because the Spotify queue is stored in the user's Spotify account.

The music suggestions are generated using OpenAI's GPT-4, and dynamically takes into account multiple emotions.

Challenges we ran into

The biggest challenge we faced was integrating with the Spotify API, because the Web SDK is not well-supported and parts of it have been deprecated. A lot of endpoints haven't been updated in a long time and don't work well with other new technologies like React.

Accomplishments that we're proud of

This is the first in-person hackathon for most of our members, and it was a truly educative experience. Despite the short amount of time available to us, we're really proud of the progress we were able to make and the fact that we were able to come up with a fun project that was engaging and made use of new and upcoming technologies.

We're really proud to work as a team and synergize to make such a beautiful product. This project is really meaningful to us and we've dedicated every minute we've been given to carefully put together unique technologies.

Learning can be hard, and rightfully so in the course of 25 hours when there's more than one thing to deal with. Our team is proud to say that, regardless of the difficulties we've faced, all our problems had well-considered solutions. We've been able to expedite the learning process and make something we hope is useful to drivers.

What we learned

Our team adopted a new frontend framework--Mantine. We've been able to utilize its various components and integrate them into a beautiful, easy-to-use interface.

Additionally, we gained plenty of exposure to AI-based technologies, namely in the form of OpenAI and Face API (a JS Library). Integrating an LLM was a bit of a hurdle for us, but we've managed to make the most of it's capabilities to build an amazing project. Face API was quite new--unlike anything we've seen. Unlike libraries like OpenCV, there was a significant learning curve but it's adoption was made easier since we were working with a web-based tech stack.

And besides the technologies managed above, we've been able to learn so much more. Everything picked over this weekend has been leveraged to make the best product under our time constraints. Enjoy!

What's next for Moodscapes

Given the time crunch, we weren't fully able to integrate everything that we wanted to do.

Actually take into account the user's music preferences in conjunction with the OpenAI prompt
Integrate into car play and android auto, and possibly separate hardware dashcams
Add more queue modification functionality to the interface, and add voice commands so the driver doesn't have to lose focus from the road.