GestureSync

Inspiration

In 2022, I participated in the Microsoft Imagine Cup, where I came across the idea of converting sign languages into understandable text language. This idea struck my mind and ever since, I have wanted to do a project on this, KU HackFest 2023 would be the best platform for bringing the idea to life. I found a team with the same enthusiasm as me for working on this project, and here we are. Our team advocates for equity and inclusivity, and creating an impact is what our team strives for. Empowering minorities and backward communities while promoting inclusivity is what keeps us going. Using technology to create something that could make a lot of lives better has always been our motto. Building "Gesturesync" is like building hopes and bridging communication gaps. Our commitment to inclusivity and breaking barriers inspired us to create an equitable world. Transforming the lives of people belonging to the specially-abled community using technology brings us nothing but pure joy. Expressing oneself without hindrance using technological advancements would make one feel grateful for belonging to the 21st century. We are inspired by the thought of creating an empathetic world where people feel heard. Nowadays, news media broadcasting is assigned to specially-abled individuals and they use sign language to convey the news to others. However, it is not understandable to everyone and we feel that those people are getting fewer opportunities in all areas of life just because they happen to be disabled. Through this project, we can help them secure a permanent job, as sign languages are translated into understandable text languages, which makes them feel included. They won’t have to feel bad for being different, which inspires us the most. Being able to solve real-time problems fills our hearts with pure joy.

What it does

Contrasting to normal sign languages that recognize each character, we detect “actions” that translates into word directly. Our project is based on train, translate and communicate. Here, the user will be able to train and save gestures with different keywords such as Hello, Bye, and so on. To start from the beginning, our project is made to translate and detect gestures in real-time. Users can use our pre-defined gestures or they can train their own gestures for use. That being said, our main feature is to provide users with live translations of gestures to the keywords.

How we built it

Our plan was to build a web app for our project which would be dynamic and easily accessible. React, as our frontend, and Django as the backend are the tech stacks used to develop a functional web app. Since we were to deploy a Machine Learning model that will be trained on different gestures with user-defined labels, our project is heavily inclined to the Python language. We used opencv-python for getting the video streams, then used the Mediapipe library to extract the key points from the hands of the user, and finally used those collected key points to train on the corresponding labels using the LSTM layers. That’s our predefined model for a prototype. Not limited to using the pre-trained model, we built ML pipelines to make the users access the model itself to train their own custom gestures. With this users can have their own custom gestures at their hand for use.

Challenges we ran into

We faced a lot of challenges and we are glad we were able to overcome them all. There was no internet access at the start, and poor network connection while still connecting to the mobile data was something very frustrating to hackers trying to win the game. The internet connection was interrupted just when we were about to start the project from scratch. The research process took quite a long time, there were a lot of algorithms we could choose, and choosing one of them was such a tough task. Algorithm switching created challenges in between. We needed to find the right APIs, but the limited open-source APIs created challenges. The best-fit algorithm was usually paid and we couldn’t use it due to monetary constraints. The Real-time video streaming from frontend to backend connection was such a challenge that took so much of our time. There were external disturbances as well which we found challenging to ignore. Despite less to no sleep, the team with very minimal energy managed to handle all the challenges.

Accomplishments that we're proud of

What we are very proud of is making an impact. Being able to build sign language to text will help facilitate meetings for the specially-abled community. Helping a whole community with the technology that we have access to makes us feel proud.

What we learned

During the intense 48-hour Hackfest, we learned the essence of working in a team and the difference it can make. Our learning started from the basics to the complex of what it could have been. From searching the basic Python syntax to learning vast ML concepts. We also learned a few sign languages of American Sign Language (ASL) that would help us communicate with specially-abled people in real life. Our learning revolves around gesture recognition, Machine Learning models, Natural Language Processing (NLP), and its integration. We learned to solve a real-life problem using computer vision. We learned that no matter how confident we are, problems arise, and we need to find people who can provide us with mentorship with whom we can communicate and find a solution from a different perspective and approach. We learned how documentation is important while creating. We came with the intention of just completing our project but we got to know so many ideas coming from brilliant minds and also learned new things right here.

What's next for GestureSync

We see the future in GestureSync. Live translations that are pretty accurate and really fast are our main focus. We can also integrate pre-built APIs and IOT. The project is limited for now to just hand motion, but we are planning to track head motions so as to improvise the precision. We are also planning to create a learning platform for ASL, where interested people find a learning space to practice gestures so as to communicate with their loved ones from the specially-abled community.