Streaming Sentiments

Streaming sentiments webapp: sentiment score, main syncon, background webcam...
Web home. No installation required.
Chrome extension: features.
Using in Google Meets.
Using webcam in OBS.
Using in Teams meeting.
Chrome extension: settings page.
Emotions in streaming.
CSV exported with transcript, emotions, sentiment score and main syncon.

Inspiration

This project has been created for the expert.ai hackathon (https://expertai-nlapi-042021.devpost.com/).

The project consists of 2 parts: a web application (which uses the webcam, microphone and the expert.ai APIs) and a Chrome Extension (which uses the expert.ai APIs). I will explain it below.

Inspiration: every day there are more people streaming their content: professional streamers, workers in virtual meetings, teachers in class, virtual events, etc...

There is a risk that in these broadcasts a screen (or a web page) will be broadcast on which personal/sensitive information appears. For this problem I created this Chrome extension, which hides the private information of the web pages (anonymizes) and also shows the feeling score of the web page, the emotions of the web page and the main topic.

On the other hand, also for streaming, there is a problem when people connect to the streaming after the streaming has started. In these cases, the new attendees have difficulty knowing what topic the speaker is talking about (many attendees ask "What are you talking about?" In the chat). To solve this problem, I created the webapp for webcam and I use the API of expert.ai to detect the main syncons and display it in the upper left corner of the webcam.

In addition, it is difficult to know if negative or positive sentiments have been used in a streaming, it is also difficult to manually detect emotions. This information is important for content generators and event organizers, to verify that no harmful content is generated for the community. For this reason, I have implemented that the webapp shows in real time the sentiment score and the emotions detected (both features use APIs from expert.ai). In addition, after the streaming, the organizers can download all the information in Excel (CSV) format.

Finally, for fun and to increase engagement, I have implemented that the webcam background changes automatically when the streamer changes the subject. The backgrounds are GIF images obtained with permission through the Giphy API.

This NLP superpowered webcam (with background, sentiment score and main syncon) can be used in any video conferencing application, such as Microsoft Teams, Zoom, Google Meets, Twitch, etc... with OBS.

What it does and How we built it

For the Chrome Extension:

Chrome Extension

Anonymizer web. It hides sensitive content on web pages and replaces it with asterisks (*****). It allows you to configure what type of entities to view: names of people, addresses, telephones, etc ... It uses the API of expert.ai to detect these entities.
It calculates the sentiment score of the web and displays it in a "gauge" type graph.
It shows the emotions of the web in a radar chart (or spider), using the API of expert.ai.
Displays the main topic of the page, using the expert.ai API for main syncons.
For testing purposes, this chrome extension allows you to change the API Token of expert.ai very easily.

For the web application (webcam):

Webcam

You don't need to install anything to work.
Uses the recent web standard to convert the user's voice into text (SpeechRecognition) and thus be able to send the message to the API of expert.ai to obtain the sentiment score, the emotions and the main theme (main syncons).
The sentiment score is shown in a chart in the upper left corner, next to the main topic.
Emotions are displayed in a spider chart at the bottom.
The webcam background is replaced by a gif image related to the current topic that the streamer is talking about. The Giphy API is used with permission.
All the information is downloadable in Excel (CSV) format, including the transcripts.
It uses TensorFlow.js and BodyPix to detect the user's body on the webcam and separate the background.
A "playground / settings" section is included to test the APIs and SpeechRecognition separately. Also for testing purposes, it is allowed from this screen to change the API Token of expert.ai.
This webcam enriched with sentiments can be used in any video conferencing and streaming application, such as Google Meets, Twitch, Microsoft Teams, Zoom, etc.
For users who don't want to modify the background of their webcam but do want the "current theme" and "sentiment score" display, there is the option "Display only" in the home page.

Challenges we ran into

Creating the chrome extension, it is my first extension for chrome.
Manipulating the HTML of web pages to hide sensitive information (anonymizer).
Using "speechrecognition" in continuous mode and then use the API of expert.ai and obtain all the information. I wanted to show the headline "we are talking now about ..." and for this I tried several APIs of expert.ai until I saw that the API of main syncons was the one that best suited this feature.
Generating CSV from javascript is also new to me.
With the API of expert.ai I have not had any problems. It is very well documented and its online version for testing helps a lot.

Accomplishments that we're proud of

I am very proud of the final result, because it is a project with 2 very useful parts for a growing sector (streaming and video conferencing sector).
I believe that sentiment analysis and NLP can be applied in many products, not only for written text but also for voice, especially in virtual events or recorded events.

What we learned

I have learned more about NLP, entity classification, emotion types, syncons, etc ...
I have learned a lot about javascript programming and APIs, creating my first Chrome Extension and learning more about SpeechRecognition from the web browser.

What's next for Streaming Sentiments

I have reached the deadline very right, so the source code is not as clean as I would like. In the next few days I will continue working on making a more readable and documented source code.
I hope to be able to publish the project soon and continue to improve it with user feedback.