I read a paper that predicted user mood based on his cell phone usage pattern, and I thought of using a similar template for web browsing.

What it does

The browser (called 'Spirited') tracks the user's browsing pattern (activity and emotion) and guesses his mood. Mood is also guessed using his facial expressions through the webcam. Then, it tries to bring it to an optimal level by playing certain songs. For example, if the user is sad, the browser would play more upbeat songs. If he is ecstatic, it would play slower songs.

How we built it

We built the browser in Python, the music player in Django, and the facial recognition in OpenCV. The browser opens the website in the window; simultaneously, in the background, it performs sentiment analysis on the website itself to get the mood of the user. This is also aided by the webcam, where we run an instantaneous facial analysis to determine mood based off of expressions.

Challenges we ran into

A couple of the really tough challenges were: 1) We had to analyze a song to evaluate its intensity, so a suitable heuristic had to be designed 2) Classification of emotions using facial expressions is still a little buggy

Accomplishments that we're proud of

1) The song does change based on your mood, which I can attest to; while tesing, it did indeed give me a more upbeat song if I spent too much time on a website, and encouraged me to be more active. 2) The facial recognition is able to differentiate between a smile and a frown, which when plugged in to our application enhances the accuracy of the results

What we learned

1) How to do facial recognition and analysis using OpenCV 2) Basics of making a browser 3) Performing sentiment analysis on websites 4) Song intensity analysis

What's next for Spirited

1) The browser is very rudimentary; it currently only renders the response websites give on a window. We're going to have to include a lot of other features including encryption to make it a real, full-fledged browser 2) We'll be refining the song selecting heuristic to improve performance and accuracy 3) We're also planning to include an indexing mechanism, so that if the user visits the same site multiple times it doesn't have to analyze it for sentiment again and again

Share this project: