SignAura

Every voice will be heard

About the Project: Signaura – Sign Language Learning and Communication Platform

Inspiration

In India, millions of deaf and mute individuals face daily communication and accessibility challenges due to the limited awareness and adoption of Indian Sign Language (ISL). Despite being a structured and expressive language, ISL is often underutilized, both in education systems and in everyday interactions. This communication gap leads to exclusion, reduced opportunities, and social isolation for individuals who rely on sign language.

The idea for Signaura was born from this critical need to make communication more inclusive. Our team envisioned a solution that leverages artificial intelligence and computer vision to create a bridge between sign language users and the wider population, enabling mutual understanding and empowering those with speech and hearing impairments.

What We Built

Signaura is an interactive web-based platform designed to make learning and using Indian Sign Language accessible, engaging, and intuitive. It features:

ISL Learning Modules: Structured lessons for alphabets, numbers, and common vocabulary, supplemented by images or video placeholders for each sign.
Sign to Text Translation: Real-time webcam-based gesture recognition using hand landmarks and a trained GRU model.
Text to Sign Conversion: Converts typed or spoken input into corresponding sign language outputs using animated/static signs.
AI Sign Tutor: A chatbot-style assistant to guide learners, answer queries, and help track learning progress.
Sign Dictionary: A searchable repository of signs with descriptions and visual references.
Translation and Learning History: Keeps track of the user’s translations, chats, and progress for future reference.

Technology Stack:

Frontend: Streamlit, HTML/CSS, JavaScript (for interactive UI elements)
Backend: Flask (Python)
Machine Learning: GRU-based RNN for gesture sequence classification
Computer Vision: MediaPipe for real-time hand tracking
Database: MongoDB for storing user profiles, history, and analytics
Deployment: Designed for local testing and web deployment, modular and scalable

What We Learned

Throughout the development of Signaura, our team gained deep insights into several technical and human-centered domains:

Machine Learning and Deep Learning: We trained and optimized a GRU model to classify sequences of hand landmarks into meaningful ISL signs, learning about time-series data handling, model tuning, and accuracy improvement.
Computer Vision Integration: Real-time landmark detection using MediaPipe taught us how to preprocess and normalize visual data efficiently for model input.
User Experience Design: Developing for an audience with diverse needs emphasized the importance of accessibility, clarity, and intuitive navigation.
Web App Architecture: We explored the integration of Flask APIs, MongoDB storage, and front-end tools to maintain seamless functionality across components.
Dataset Creation and Augmentation: We collected a custom dataset of ISL gestures, involving multiple users and lighting conditions to make the model more robust and generalizable.

Challenges We Faced

Lack of Public Datasets: Most ISL datasets were either unavailable or lacked the diversity required for effective model training. We had to create a balanced dataset with variation in users, gestures, and capture conditions.
Real-Time Accuracy and Latency: Ensuring the system could accurately predict signs without delay was a major technical challenge, especially on resource-constrained environments.
Gesture Ambiguity: Similar hand signs sometimes led to misclassification. This required improving preprocessing steps and refining the model with more examples and better feature extraction.
UI/UX for Accessibility: Designing a platform that could be comfortably used by both deaf/mute users and complete beginners to sign language required extensive feedback cycles and testing.
Modular Architecture: Integrating gesture recognition, sign rendering, AI assistance, and user tracking in a seamless, bug-free web interface required careful coordination across multiple components.

Impact and Future Scope

Signaura aims to promote inclusion, empathy, and awareness by making sign language education mainstream and technology-driven. It has the potential to be deployed in: