Inspiration
🚀 About the Project Signapse is an AI-powered real-time sign language translation system built to break down communication barriers between the hearing and hearing-impaired communities. Our mission is to foster inclusivity through technology by enabling seamless two-way communication—translating sign language to text/audio and vice versa.
💡 What Inspired Us The idea for Signapse was born out of a simple observation: millions of people around the world who are deaf or hard of hearing face communication challenges in everyday life. Whether it’s at schools, hospitals, public offices, or workplaces, the lack of accessible communication tools often leaves them behind. We were driven by the vision of a world where technology bridges this gap and empowers everyone to communicate freely.
🛠️ How We Built It Our tech stack combined computer vision, deep learning, and natural language processing to build a comprehensive communication assistant:
Model Training: We trained a custom gesture recognition model using YOLOv5 and MediaPipe for real-time hand detection.
Translation Engine: For sign-to-text, we used sequential recognition models on labeled gesture datasets. For text-to-sign, we implemented a video-based sign language rendering engine.
Speech Integration: Integrated gTTS to convert recognized signs into natural speech.
Frontend & Interface: Developed a user-friendly GUI using Tkinter and OpenCV for live video processing and feedback.
Voice Input Mode: Enabled voice-to-sign capability, translating spoken input into sign visuals using a database of sign video snippets.
🌐 YouTube & Google Meet Extension: We built a browser extension that provides real-time subtitle-to-3D sign language conversion during video calls and streaming, making platforms like YouTube and Google Meet more accessible.
🕶️ AR Glasses Integration: We also prototyped AR glasses that recognize hand signs and render 3D animated sign language avatars in real time—enabling a futuristic, hands-free communication solution for accessibility on the go.
🧠 What We Learned Real-time gesture detection is complex and requires careful optimization to balance speed and accuracy.
Creating smooth and synchronized sign animations for full sentences is much harder than detecting isolated signs.
Accessibility solutions must be robust, reliable, and work well across diverse conditions like lighting and background noise.
Integrating AI with hardware (like AR glasses) introduced new dimensions of design and debugging.
Team collaboration and version control were essential in managing parallel streams of development (models, extensions, hardware integration).
⚙️ Challenges We Faced Model Accuracy: Fine-tuning the model to differentiate between similar signs was a major hurdle.
Synchronization: Ensuring that sign language video and 3D avatars align precisely with real-time inputs like speech or gesture.
Dataset Limitations: Limited availability of large, diverse sign language datasets for full-sentence detection and 3D model training.
Real-Time Performance: Running all systems (vision, voice, UI, and rendering) smoothly on non-GPU devices required deep optimization.
Hardware Integration: Calibrating camera input and latency in the AR glasses while maintaining real-time performance.
🌍 Impact & Vision We believe Signapse is more than a tool—it’s a leap forward in building a more inclusive world. Our vision is to evolve it into a wearable or plug-in solution that empowers millions to communicate effortlessly, whether through desktop, mobile, or AR-based platforms.
We aim to bring sign language access to every corner of the digital world, from video conferences and education to public services and beyond.
Log in or sign up for Devpost to join the conversation.