AI Emotions Godview – track how speech content spurs emotion

Inspiration

For politics: Emotion-invoking narratives play are major role in driving world events today like the election of Trump, Brexit, the US-Iranian conflict escalation and the HK protests. Politicians, lobbyists, lawyers and activists often use speech to stir up stir up strong emotions amongst their audience and grow a following. Unfortunately, till date there has not been any notable work linking the exact words used in the speeches of such figures with the effect of such words on the emotions of the audience members, even though recent advances in AI have allowed such tracking to be possible. The problem of fake news, which is very much in the public consciousness today, is also partially addressed here, by creating BERT sentence-embeddings of the sentences which invoke the strongest emotions, for semantic search of news articles relevant to it.

For classroom use: Teachers can track the student reactions and engagement to their teaching, like a personal AI coach to help the teacher constantly improve.

For meetings: Meeting leaders can check the web app and visualisations in post-meeting evaluations

What it does

Speech recognition engine transcribes speech in real-time, and matches that with facial emotions of the audience detected by our facial emotions recognition engine. Our data visualisation platform allows the user to explore the data, relating the sentences with the emotions detected. BERT-sentence embeddings are also created for the sentences which invoked the most emotions, for semantic search of recent news which are relevant to such sentences.

How we built it

Within the tight timeframe of a 24-hour hackathon, we used our ML expertise to choose suitable open-source libraries which implemented models we thought would be suitable for the use case, rewrote and tuned some of them before integrating into our FERN stack web app, along with Chart.js visualisations of the data.

Challenges we ran into

Scaling the facial emotion recognition to more audience members was tough for a 24-hour hackathon. We didn't have enough time to build a scraper to scale our semantic search with more web data beyond limited headline news, though our choice of the Siamese network-enhanced BERT implementation would have been able to handle such scaling of the semantic search gracefully. We tried to adapt a driver drowsiness detection model to our use case to measure how focused the audience members were, but the results were not satisfactory.

Accomplishments that we are proud of

We managed to put everything together in such a short time of 24 hours and got it to work decently! More importantly, we persevered through the hackathon by staying at the venue to work together all the way as a team, with minimal sleep!!!

What we learned

We got much more familiar with the tools we have been using for a while, because of the time-limit of the hackathon which forced us to dig deep. Don't spend too much time dwelling on the same problem, be disciplined in setting time limits for yourself to go deep in one approach and navigate the obstacles, then cut it and try another approach once the hard deadline is up