• 

  Home Screen

• 

  Options Screen

• 

  Hand Sign Language for "Yes"

• 

  Hand Sign Language for "Hello"

• 

• 

Bridging Communication, One Sign at a Time

Inspiration

Communication is a fundamental human right, yet millions of deaf and hard-of-hearing individuals face barriers in their daily interactions. We wanted to create a solution that fosters inclusivity by leveraging technology to bridge this communication gap. Inspired by the power of AI and machine learning, we set out to develop SignSpeak—a real-time sign language translation web tool that enables seamless communication for all.

What We Learned

Throughout this project, we gained valuable insights into:

• Machine Learning & TensorFlow – Understanding how to train and deploy models for real-time sign language recognition.
  
• Computer Vision – Processing and interpreting hand gestures with accuracy.
  
• Web Development with React – Building a responsive and intuitive user interface.
  
• The Importance of Accessibility – Designing with inclusivity in mind, ensuring our tool is user-friendly for diverse communities.
  

How We Built It

Our project was developed using:

• React – For the front-end interface.
  
• TensorFlow.js – To run machine learning models directly in the browser.
  
• WebCam API – For real-time hand gesture recognition.
  
• Custom Trained AI Model – Using sign language datasets to improve recognition accuracy.
  
• GitHub – For version control and team collaboration.

Challenges We Faced

• Model Accuracy & Training Data – Finding high-quality datasets for training was a hurdle, and ensuring accurate gesture recognition required multiple iterations.
  
• Performance Optimization – Running AI models in the browser efficiently without latency was a challenge we tackled through model optimization.
  
• User Experience & Accessibility – Designing an intuitive UI that serves both sign language users and those unfamiliar with it required continuous testing and refinement.
  
• Keras to TensorFlow.js Conversion – We initially trained our model in Keras, but converting it to TensorFlow.js for in-browser execution proved to be difficult due to compatibility issues and model size constraints.
  

Acknowledgement

We would like to express our gratitude to Kazuhito Takahashi, whose pre-trained model served as the foundation for our AI system. His work provided us with a strong starting point, allowing us to fine-tune and adapt the model for real-time sign language translation.

Conclusion

SignSpeak was built with a vision to empower communication and inclusivity through technology. While we've made significant progress, there's still room for improvement—such as expanding sign language support and enhancing model accuracy. We hope to continue refining our project and making a meaningful impact on how people connect, one sign at a time.

Built With

• github
• javascript
• keras
• mediapipe
• opencv
• python
• react.js
• tensorflow
• tensorflow.js
• webcamapi