Stream of Consciousness

Inspiration

As researchers, we're interested in how multiple media (e.g., music, visuals, text) can work together to reinforce the perceived immersion of people while experiencing content. If the different media used to deliver content jointly point in the same emotional direction, then the experience will be even more immersive. As developers, we love to play around with these ideas and to test them by building interactive systems. This is why we decided to get simple text content (tweets) and enhance them, by creating a VR scene that adapts all-around to the emotional content of a tweet stream. Think of Stream Consciousness as an emotional amplifier of the tweets you read!

What it does

First of all, put the Oculus Rift headset on! You're now in a low-poly riverside scene. You can choose among different "hot topics" organised into different types (e.g., people, events), which have been previously populated in the UI by making a call to the ContentPool API. Once you select a topic, Stream of Consciousness makes another call to the ContentPool API to get tweets about the topic. The tweets are then displayed in the scene one by one, as in a stream of consciousness. If you like, you can select another hot topic and get a new tweet stream.

The VR scene responds to the emotional content of the tweets. When a tweet is displayed, the system uses AI to automatically extract its emotion and uses the information to influence the emotional content of the music and of the scene. Depending on the _ intensity _ level of the tweet and on its _ positivity _, the system renders the same basic scene in different seasons. For example, if a tweet is "sad" the system switches to the winter season. The music also changes its emotional state dynamically. The music is generated from scratch in realtime by the Melodrive AI music engine. Please note that we didn't implement the music generation system during the hackathon - and this is freely available online as Early Access software. When the emotion of a tweet is detected, this information is passed to the Melodrive music engine which changes the emotional content of the music to reflect that of the tweet. As a nice add-on feature, Stream of Consciousness has musical trees that "dance" following the beat!

How we built it

Stream of Consciousness is made up of multiple parts. First, we developed an algorithm that could extract emotions from tweets in Python. Most sentiment analysis algorithms can extract a score to describe the polarity of a text on a unidimensional scale only (e.g., bad vs good). This is OK for sentiment extraction, but it is limited for emotion extraction, which warrants at least a couple of dimensions to be modeled (_ intensity _ and _ positivity _). Our algorithm extracts 2 continuous scores for describing the _ intensity _ of a tweet and its _ positivity _, employing a corpus-based approach. We also built a VR scene in Unity. From the Unity scene, we used the ContentPool API to get hot topics and tweets in the experience. We made the Unity scene dynamic, by programmatically changing its visual aspect depending on the emotional state of the tweets. We also created an interface to the Melodrive music engine to influence its generation modules. In the Unity scene, we created a choreography with the trees by tracking the beat of the music.

Challenges we ran into

We had a few issues integrating the emotion extractor algorithm coded in Python in the Unity project. Also, we noticed that the emotion of the majority of tweets tend to be quite "neutral", so we had to amplify the values of the _ intensity _ and _ positivity _ scores we extracted, in order to have a bigger impact on the music generated.

Accomplishments that we're proud of

We're really proud of how Stream of Consciousness is able to detect emotions and to dynamically adapt its visual and musical content to the tweets. The full pipeline works nicely! We also tested the system with non-tweet text (e.g. fairy tales), and it performs really well.

What we learned

By developing and playing around with Stream of Consciousness we realised that music can have a massive impact on how people read and interpret news. On a more sociological side, we learned that most tweets tend to be quite neutral in terms of emotional charge. We also found out that everyone's tweeting about Donald Trump and Angela Merkel.

What's next for Stream of Consciousness

Stream of Consciousness will become a demo for our AI music startup Melodrive, which will showcase the potential of AI music as a means to enhance textual content with real-time music generation. In terms of improvements, we're envisioning integrating a speech-recognition system in Stream of Consciousness, in order to enable people to influence the visual and musical content of the experience just with their voice.

Built With

• python
• unity