Hunter: a video sentiment analysis platform

Hunter is a video conferencing app which provides realtime sentiment analysis of the participants of the meeting and detailed feedback regarding the overall mood and attentions levels.

Comment

Inspiration

Our inspiration to build this app was to further human communication, interaction, and connection in the era of online meetings. Specifically, we were recently on a video conference with our research lab and were not completely sure how the rest of the lab was responding to our work and presentation. Therefore, we were unable to determine how best to craft our presentation to ensure its appeal to everyone in the meeting room while simultaneously providing us with the most accurate, unspoken feedback.

What it does

Our application is built on top of a live video conferencing platform; it provides a dashboard to manage your own speaking attitudes and to view the emotions of your presentation audience over time. Specifically, it shows any of 50 different emotions that you or your audience may be feeling while also highlighting the most prevalent emotion present among your audience.

How we built it

We built our application on top of DolbyIO's Communication UIKit along with Hume.AI's live streaming sentiment analysis APIs. Our application made heavy use of Web APIs for video capture, recording, and transmission. Furthermore, due to the live nature of our application, we used WebSockets for nearly all communication between our front and back end applications. From each client, we were able to make continuous requests to Hume's streaming API, with both facial images and audio transcripts in order to generate a complete emotional summary of each participant. These summaries were then aggregated on the server side, which communicated continuously with each client using the WebSocket API.

Challenges we ran into

We ran into quite a few challenges with WebSocket, as initialization and connection led to quite a few errors. Additionally, getting familiar with the DolbyIO Communications libraries took some getting used to in order to become proficient with the entire stack. Lastly, due to some API overloading, we faced some outages with Hume's API, complicating our development, which was centered around live insights and low latency.

Accomplishments that we're proud of

We are proud of building a live application with such complex business logic using low latency web-based connections provided by WebSockets. Furthermore, we think our application has significant real world potential due

What we learned

We learned how to ensure reliability of continuous data streams, to work with live video, and to decrease latency in live applications.

What's next for Hunter: a video sentiment analysis platform

Next, we would build a full fledged speech coach, which would help users improve their presentations after the fact based on insights provided by sentiment analysis and next-token prediction provided broader context regarding the presentation at hand.

Built With

Share this project:

Updates