Get your voice heard

Inspiration

Work from home has changed the way we work. Most meetings shifted to be online and remote, making communications sometimes more difficult, as delays make it easy to get interrupted and dominant speakers might limit the interactions and ability to contribute to the meetings success.

A lot of recent studies and surveys¹²³⁴ have shown, that especially woman are more like to get interrupted or having difficulties to speak up in remote meetings.

¹https://www.linkedin.com/pulse/women-virtual-meetings-your-voice-getting-heard-andrea-grossman/ ²https://www.nytimes.com/2020/04/14/us/zoom-meetings-gender.html ³https://janicetomich.com/women-speaking-while-female/ ⁴https://covergalls.com/blogs/news/sit-tall-speak-up-the-challenges-women-face-in-virtual-meetings

What it does

A Chat Bot can be added to any Teams Tenant, which is able to join voice or video calls, listening in and detecting interruptions and who is speaking. Based on post-call analysis it provides private messages with some insights to the call and highlighting potential habits/biases (to each individual privately) and/or encourage participants to be speak up and/or provide coaching on what behaviors can be used to get more engaged into the call.

How we built it

Through a Microsoft Bot Framework Chat Bot, call interactions will be anonymously monitored to provide post-call reports and gentle coaching to dominant speakers, to raise awareness to give others the opportunity to talk up and share their insights/opinion/knowledge. Through whispers AI model a custom model is planned to be built to detect more extensive insights of the call than just speaker/gender and transcription.

Challenges we ran into

Multimedia processing is difficult as it only comes with either commercial licenses, or extensive other restrictions (where it can run, what it can do, etc.) making it difficult
Time, the task is quite extensive and duration limited. Training a model on a view voice samples already took ~6 hours industrializing the model to be performant enough to be used in a near-realtime environment like a call will take some further refinement
Machine Learning Bias, build a Model to detect female/male voices is easy, however with an inclusive background some male's might identify themself as she/her and vice versa, providing a capability to define own pronouns or retrieving this over the Graph API will be another challenge/problem which would need to be solved

Accomplishments that we're proud of

Created the Bot which is able to join a direct, group call and send out notifications to end-users
Integration with Microsoft Cognitive Services (for now) to provide insights to end-users

What we learned

Teams only supports certain media codec's and certain restrictions of the Teams SDK make it difficult to integrate in some solutions (.Net Core)

What's next for Get your voice heard

Create a custom AI Model which can be trained on interruptions, gender detection, and/or other bad behaviors in online meetings to provide more extensive coaching for meeting participants.