People want music ubiquitously. Plenty of playlists exist for showering, studying, and working out - so why don't we generate a playlist from text by analyzing the emotions from it? Notes for Notes generates music playlists for the book that you will be reading tonight. It can understand your youtube video just by reading the transcript and then recommend the background music just for your video. It can take in a scene from a movie script and give you the soundtrack to live the moment as you read it. Wouldn’t video editing be much more dynamic?

What it does

First, choose a file that contains text. It can be a jpeg, jpg, or png file, which means our application will recognize characters from the photo of handwritings too! Then, the Natural Language Understanding will go through advanced text analysis, yielding emotion keywords and conceptual keywords, all with their respective levels of confidence. Using those keywords, we analyzed their connotations and co-occurrences of topics in order to associate them to Spotify genres in order of importance. Finally, using those genres, we yielded your top Spotify playlists to represent your text.

How we built it

We used _ Google Cloud OCR(optical character recognition) API _ to recognize a text from jpg, jpeg, png. For emotion analysis, we used _ IBM Watson’s Natural Language Understanding API _. From the result, we used the Spotify API to generate playlists. Finally, we used Vue.js framework for Frontend and NodeJS for backend to make the web application.

Challenges we ran into

The first challenge was to connect the existing APIs to our local machines. Since we haven't dealt with this many API in one web application before, it was difficult to be brave at first. Yet we figured out this part by asking a lot of questions and watching youtube tutorials!

The second was that Spotify API doesn't support the recommendation fully since that is one of the most popular features of Spotify. Also, the authorization for Spotify API was very tricky. This prevented us from using our original design.

Accomplishments that we're proud of

We did it !

What we learned

  1. Web application-related knowledge. One of our teammates learned javascript here!
  2. Linguistic analysis

What's next for Notes for Notes

Expansion: We would like to be able to work up to an implementation where one could scroll through the text in real time and see how the generated playlists change as the storyline evolves! They could also track which genres dominate which parts of the text, and examine the patterns and relationships between those genres. We would also be interested in a multimodal approach that does not only utilize text but expands its computer vision applications to a movie or show; in others words, the combination of the analysis of what's on the screen would be in combination with the script/text analysis to generate an even more representative playlist.

A new B2B model for media companies: This is perfect timing for media companies to use Notes for Notes. Link metadata to the music that goes well with their products. Advertisement, Trailers, and even E-books, the possibilities are endless.

Work in reverse!: If we discover enough patterns from textual concepts to musical genres, we can perhaps go in reverse to analyze a song and predict what kinds of themes a text related to that song would pertain to. Considering the popularity that movie soundtracks and lucrative artists can bring to media, it would be useful to take the most popular genres of music and see what kinds of movies/show/texts would create the ultimate marketable compatibility.

Built With

Share this project: