Vibe

Inspiration

Music provokes us. It influences our moods. If you play the right song are the right time, you can can help a classroom of students focus, facilitate conversation in a fancy restaurant, or get crowds pumped at large outdoor events. But how do you find the perfect playlist based on the vibe?

What it does

With Vibe, you snap a picture of your surroundings, and the app returns the playlist with the best-suited songs for the current physical environment. In addition to extracting clues from the picture, Vibe measures the current ambient noise level in the room and accounts for current time when making its music selection. This information is fed into a trained machine-learning algorithm that intelligently finds the type of music best-suited for the situation.

For example, if you’re at a library looking for study music, you would open Vibe and take a picture of, say, the bookshelves around you. Then Vibe listens to your environment and notices that it’s really quiet around you, and that separately checks that it’s close to nighttime. It spits out a Spotify playlist that makes sense for your quiet, focused library session.

How we built it

The mobile app is built in React Native. We used the Google Cloud Vision API to extract labels from the user’s picture of their surroundings. Using a pretrained word2vec model from Google News, we ran k-means to cluster the words. We then labeled clusters based on what Spotify genre we believe these word groups best fall in so that the next time so that it becomes a quick look up process every time a user takes a picture.

Challenges we ran into

Each disparate response on its own presented it’s own challenges, but the most difficult part was putting each section (computer vision analysis, k-means algorithms, Spotify integration) into an app with easy user flow. We spent a lot of time on compatibility issues between our various APIs. For example, since there is no convenient wrapper for React Native to use Google Vision, so we had to complete the Vision callback on our Python backend instead (temporary solution).

Accomplishments that we're proud of

We're proud that, on a surface level, Vibe is simple idea with immediately apparent value. At the same time, we know that under the hood, there are a lot of working parts which, in the tests we've performed so far, are giving us playlists that really make sense for the environment. We're also proud of our recommendation engine--incorporating k-means and word2vec models to arrive at genres.

What we learned:

This project improved our efficiency as a team, since we quickly realized that a lot of our code was overlapping between teammates due to lack of communication. We also learned how to troubleshoot vague errors, especially when working with an XDE like Expo (StackOverflow and Google are key!). On a high-level, we learned to narrow down broad ideas; Vibe went through several different versions before we settled on a concise pitch.

What's next for Vibe:

There's potential for an even more intelligent recommendation engine. This comes from 1) using our own word2vec rather than using Google's (which had a lot of unhelpful clusters) and 2) adding more sensor information like temperature and location.

Built With

• google-computer-vision
• k-means
• python
• react-native
• spotify