SignBridge AI

An AI-powered inclusive classroom platform bridging communication for Deaf students with real-time sign recognition, live captions, and smart lesson summaries.

Comment

Inspiration

Deaf and hard-of-hearing students in traditional classrooms often face significant communication barriers. Many teachers do not know sign language, and students struggle to follow fast-paced lectures while attempting to take comprehensive notes at the same time. We were inspired to build SignBridge AI to break down these barriers and ensure every student has equal access to classroom education through accessible, real-time AI technology.

What it does

SignBridge AI is an inclusive, two-way interactive classroom assistant:

  • Student → Teacher (Gesture Translation): Uses computer vision to recognize student hand gestures in real-time via camera, converting sign language into text and audible speech so the teacher can immediately hear the student's questions or responses.
  • Teacher → Student (Live Subtitles): Converts the teacher's spoken lecture into continuous, real-time visual subtitles for deaf students.
  • After Class (AI Lesson Summaries): Automatically synthesizes the entire classroom lecture transcript into a structured, easy-to-read study guide (Main Topics, Key Concepts, Vocabulary, and Homework) powered by Groq Llama 3.3.
  • Dual Role Dashboards: Tailored interfaces for both Teachers (session management & student gesture feeds) and Students (live subtitles, gesture controls & AI summaries).

How we built it

  • Frontend UI/UX: Built with HTML5, Tailwind CSS, JavaScript (ES6+), and Lucide Icons for a clean, accessible, high-contrast user experience.
  • Computer Vision: Integrated Google MediaPipe Hands to detect 21 hand joints in real time, translating complex gestures with high accuracy directly in the browser.
  • Speech Engine: Built with the Web Speech API for live speech-to-text captions and Web Speech Synthesis for real-time text-to-speech voice playback.
  • AI Summary Proxy: Developed a secure Node.js/Express server that proxies transcript payloads to Groq API (llama-3.3-70b-versatile), keeping API keys safe and private.

Challenges we ran into

  • Real-Time Latency: Ensuring camera gesture recognition ran smoothly at 30+ FPS without lagging the live subtitle stream.
  • API Key Security: Moving AI API interactions to a secure backend server environment to follow production best practices.
  • Inclusivity & UX: Designing an interface that is extremely easy to navigate for both teachers and students without steep learning curves.

Accomplishments that we're proud of

  • Successfully creating a functional two-way bridge where a deaf student can "speak" via gestures and hear real-time voice output.
  • Achieving fast, structured AI lesson summaries using Llama 3.3 that transform raw audio transcriptions into digestible study notes.
  • Maintaining zero external dependencies for camera translation, allowing it to run smoothly on standard webcams and laptop cameras.

What we learned

  • How to harness high-speed browser-based computer vision (MediaPipe) alongside natural language processing models.
  • The importance of designing accessible, empathetic technology specifically tailored for inclusive education and real community impact.

What's next for SignBridge AI

  • Expanding gesture support to full 3D sign language dictionaries (ASL / BISINDO).
  • Adding multi-language subtitle translations for non-native speaking students.
  • Integrating offline AI models for environments with limited internet connection.

Built With

Share this project:

Updates