AI Sign Language Interpreter

AI Sign Language Interpreter is a software that enables people who are hearing impaired to be able to communicate with someone who does not have the knowledge of sign language.

Comment

Inspiration

My inspiration of this project was a case that I came across in a University setting. In a case where there are students of all abilities in a class, sometimes it is difficult to communicate with someone especially for the hearing impaired people who find it hard to communicate with people who do not understand sign language. This project is meant to bridge just a little bit of that gap.

What it does

The system is a Sign Language Interpreter. It reads signs according to the trained machine learning model in the system and gives output inform of text. These are the features of the system: Real-Time Interpretation- It translates signs from a live webcam feed by finding the most similar gesture in its knowledge base. AI Pipeline - it uses MediaPipe to extract hand landmarks, which are then converted into powerful 128-dimensional "embedding" vectors by a custom CNN. Scalable Vector Database that is Powered by TiDB Cloud, which stores and searches through thousands of sign embeddings in milliseconds using approximate nearest neighbor (ANN) search. Teachable AI: An "Admin Mode" allows users to easily teach the system new signs, continuously expanding its vocabulary. Secure User Authentication: Features a full registration and login system to manage user sessions.

How we built it

The application is built on a modern, decoupled architecture designed for real-time AI inference and data management. The following architecture is adapted: Webcam(Input) ---> MediaPipe(Landmark Extraction) ---> Custom NN (Embedding Model) ---> TiDB Cloud (Vector Search) ---> Streamlit (UI)

Challenges we ran into

The following were the challenges:

  1. Low CPU strength in my machine prevented me from running the model locally. Which led to adoption of Google Colab.
  2. Google colab was also a challenge in UI output due to conflict in components and poor server client connection.

Accomplishments that we're proud of

Model training was accomplished using Google Colab. Responsive UI

What we learned

Lessons learnt: Machine learning and AI project take up a lot of CPU strength and slow down PC performance sometimes leading to the PC crushing. Google Colab is an advanced platform that provides GPUs to train machine learning models at a faster rate. Functionality of TiDB Cloud in terms of storage.

What's next for AI Sign Language Interpreter

The is still a lot of improvements to be made since the Machine Learning needs some adjustments making the system not 100 percent useful its only about 40 percent useful in saving information in TiDB Cloud. NB: To run the project locally a dataset from Kaggle needs to be downloaded the whole dataset was not able to be pushed to GitHub but instructions on how to download and run them is present in the README.md in my GitHub Repository.

Built With

  • cnn
  • mediapipe
  • python
  • streamlit
  • tidb
Share this project:

Updates