Sign translator

Global Architecture
How to get the app on teams
Integration in teams desktop app on a team meeting

Inspiration

Helping other people

What it does

Deaf people have a hard time communication with their peers when remote working, they are forced to communicate by writing. So when using our service (Sign translator), which will translate the sign language into text in real time and display the text in a Microsoft Teams Tab during the meeting. They will be able to express themselves normally and physically.

How we built it

We built it using python, tensorflow and mediapipe for the sign language recognition. Mediapipe handles the strategic points of a person, which we then use to train the machine learning model. React, typescript, socket.io, REST API, Azure and MS Teams sdk, for the custom Tab in teams. Communication betwwen our API and the ML model are through a Rest Api while the communication between the api and the MS Teams Tab is through sockets. The Tab is deployed in the cloud with Azure.

Challenges we ran into

Choosing the correct data to train the model with. Hard time understanding the Disco framework, and we didn't manage to make it work. Getting started with developping apps on MS Teams, the intro tutorial is supposed to take 30 mins, and it tooks us 5 hours.

Accomplishments that we're proud of

We are proud to have a functional Proof of Cocept. Getting to translate from sign language to words is really satisfying. Learning about app developpment for MS Teams, and deploying in the cloud, it was our first time doing it.

What we learned

Creating a machine learning model with multiple inputs (a video). Creating a pluging for MS Teams. Cloud deployment and SDKs.

What's next for Sign translator

Finish developping the training Web app, so multiple people can contribute to training of the ML model. And add integration for other communication services such as Google Meets, Discord, kMeet. Maybe sell it to Microsoft :)