GestureLink

Inspiration

We recognise the frustration and challenges for deaf and mute people when they are communicating with people via video call. Typing back and forth forever to express thoughts during a video call is a waste of time and frustrating for people having disabilities like this to communicate with other participants. Thus, our desire is to connect and enhance communication barriers between the deaf, mute community and normal individuals. We aim to develop a video call meeting platform that can detect and translate hand sign language into text subtitles in real-time, making everyone engage comfortably without frustration and inconvenience from both sides. We believe this will somewhat “bring” the unfortunate community closer to the society, enhance connectivity, dissolve the communication barriers for disabled people (deaf and mute community).

What it does

It plays as a platform for live video calls where multiple users can create live video call meetings and others can join by the room ID. During video call, the platform supports hand sign recognition for (deaf and mute) people who only do hand sign languages and translate into text subtitle continuously in real-time for every other participant. For normal people to communicate with deaf or mute people, they can hand sign to them or they can simply speak and there will be auto-generated subtitles by voice recognition. Notice that the auto-generated subtitle by voice recognition is not included in the demonstration because our focus for this project is for the disabled community side.

How we built it

We first discussed and came up with the idea of our project. We then defined the objectives as well as major components, technologies for the projects. The building of the project is broken into sub-parts, we first had to manage to build the video call web with proper connection and setup. Meanwhile, dedicated UI/UX members started to design and build the interface for the application. We then used our model and trained it separately to recognise different hand signs before the integration stage. We finally integrated the model into our web application to have a complete project, but this final stage is the most difficult part and we failed in this stage.

Challenges we ran into

None of us has done video call web applications before, so we had to look up and try different versions of the application. Throughout the progress, we struggled a lot with errors regarding networking and connections between participants. Another challenge was that we do not know how to integrate our hand sign recognition model into the web application to run in the background and detect hand signs in real-time. We have researched and tried different ways to integrate but we still have not found a way to integrate the model into the application properly.

Accomplishments that we're proud of

We managed to build the video call platform with basic features and tackled problems regarding connections. Moreover, the model was successfully trained to be able to recognise simple hand signs.

What we learned

We definitely improved our teamwork skills a lot by cooperating with team members throughout the progress. We have learned how to build a real-time video call web application using js, html, css, with firebase. We now understand the high-level mechanism of sockets and how the TCP, UDP protocols are used before and after the establishment of connection for a video call. We have also learned how to optimise the efficiency of a LSTM (Long short-term memory) model during our training phase.

What's next for GestureLink

Looking ahead, GestureLink is set to undergo significant enhancements to serve its users better. Improving accuracy of hand sign recognition is one of the key upgrades to ensure seamless communication. In the future, the platform will support multiple hand sign languages and a version for mobile devices. For even further goals, we look forward to collaborating with big organisations and integrate this hand sign recognition project into popular and standard platforms for video call today such as Zoom, Microsoft Teams, and Google Meet.