hAIndle

Inspiration

We were inspired by the difficult experiences persons with disabilities faced and we wanted to enable them to be able to connect and integrate better into their communities.

What it does

With our Mobile Application, persons with disabilities would be able to translate their ASL to text in real time, allowing them to communicate with their peers who may not be well versed with ASL. With our Web Application, the same individuals would be able to translate their ASL to text and send it virtually over their browser. Their peers can then use our speech to text feature to communicate back to them.

How we built it

First we trained our model using a popular publicly available dataset from kaggle, the respective hand signs for the appropriate sign language. We then made a program that captures hand gestures and with reference to the model, would return the correct letter in ASL.

We then built a server to be able to receive images and process the images in real time. We then moved onto the frontend with a mobile application and Google Chrome extension to finalise our product.

Challenges we ran into

The model was difficult to train as there were many initial inaccuracies in the dataset. Putting together the server and frontend was also complicated. Achieving real time connection was our toughest challenge as we have little to no knowledge on networks.

Accomplishments that we're proud of

Having never dealt with AI Modeling and Socket Connections, we are proud that we have made a working application that involves multiple frameworks.

What we learned

We have learned how to use Socket.IO and AI Modeling to some extent and would love to dive deeper into those fields in future projects.

What's next for hAIndle

Our current AI model detects individual letters in ASL, and translates it into text. We would hope to expand our model to be able to accept British Sign Language (BSL) and be able to translate full words to text. At the same time, we would hope to increase the reliability of our model, improving its accuracy.

We also hope to integrate our application into more mainstream web services such as Zoom Meetings, to better empower persons with disabilities.