AI SignLang Tutor

breaks communication barriers for the deaf people by enabling seamless two-way interaction through text, voice, and Indian Sign Language using AI and gesture recognition.

Comment

💡 Inspiration

The theme “Breaking Barriers” inspired me to address a challenge close to my heart the communication gap faced by the deaf community.
Millions of deaf individuals struggle with limited resources, social isolation, and inaccessible education systems due to a lack of sign language awareness.
Through this project, I wanted to break that barrier by building an AI-powered system that enables two-way communication using Indian Sign Language (ISL).

🤖 What it does

AI SignLang Tutor is a fully featured educational and assistive web app that enables:

  • ✅ Text/Voice ➡️ ISL Video translation
  • ✅ Ask AI Tutor (using Amazon Q) for ISL explanations (e.g., “How to sign father?”)
  • ✅ Sign ➡️ Text + Voice (using webcam gesture recognition and speech output)
  • ✅ Friendly, accessible UI with speech + visual feedback
  • ✅ Uses pre-recorded ISL videos, AI predictions, and real-time microphone & webcam support

🛠️ How I built it

  • Frontend: Built using Streamlit, styled with custom layout and visual feedback.
  • Backend Components:
    • Speech Recognition for capturing voice queries.
    • Amazon Q Developer for intelligent responses to sign-related questions.
    • pyttsx3 for converting responses to speech.
    • MediaPipe + TensorFlow to detect and classify hand gestures.
  • Video Integration: Pre-recorded ISL videos mapped to recognized words/phrases.
  • Custom Training: Collected webcam data to train a gesture model for phrases like “hello”, “how are you”, “thank you”.

🚧 Challenges I ran into

  • Lack of open datasets for ISL phrases .
  • Streamlit Cloud doesn’t support webcam/mic-dependent features needed local deployment.
  • Training gesture models required good lighting, angle consistency, and lots of retakes.
  • Amazon Q Developer cannot be called directly via API, so we simulated intelligent responses.

🏆 Accomplishments that I'm proud of

  • Built an app that supports two-way sign language interaction — both text ➡️ sign and sign ➡️ text/voice.
  • Trained a custom model for gesture-to-phrase recognition in ISL.
  • Successfully integrated Generative AI, Speech, and Computer Vision in one project.
  • Designed a user-friendly, accessible interface with inclusive design principles.

📚 What I learned

  • Accessibility is not just a feature it’s a responsibility.
  • Gained deep experience in combining GenAI, vision, and audio systems.
  • Explored practical deployment limitations of mic/webcam apps and planned workarounds.

🚀 What's next for AI SignLang Tutor

  • 🔠 Expand phrase dataset to support full conversational ISL.
  • 🌐 Add support for multiple languages & sign language standards ( ASL, BSL).
  • ☁️ Deploy via AWS EC2 or other full-featured environments for webcam/mic support.
  • 📱 Build a mobile app version with offline capability.
  • 🧠 Integrate true Amazon Bedrock or SageMaker for advanced GenAI interaction.

Built With

Share this project:

Updates