SignBridge

Website home page
Real life trial
Poster

SignBridge was inspired by the communication barriers Deaf ASL users may face in healthcare settings when providers do not understand American Sign Language. In these situations, even expressing a basic need can become difficult. We wanted to explore how AI could support simple, accessible communication while keeping the user in control of every message. SignBridge is an AI-powered web application that recognizes a limited set of static ASL fingerspelling handshapes. The application displays the predicted letter and confidence score, then asks the user to confirm or correct it. Confirmed letters can be combined into short text and read aloud. Healthcare providers can also type a response that appears in large, accessible text. The prototype does not translate complete ASL conversations and is not intended to replace qualified interpreters for diagnoses, treatment decisions, informed consent, or emergencies. We first designed the interface and user journey in Figma. We then built the frontend using React, Vite, and TypeScript, and developed the backend using Python and FastAPI. MediaPipe is used to detect 21 hand landmarks, while a K-Nearest Neighbors classifier predicts supported static ASL letters and returns a confidence score. The frontend and backend communicate through API requests. We used Replit for development and integration, and NotebookLM to organize our research and project content. Our main challenges were connecting two separate frontend and backend projects, configuring API requests and CORS, managing camera permissions, and handling browser compatibility issues with MediaPipe and WebGL. Since some browsers could not process the landmarks reliably, we explored server-side image processing as a fallback. Another challenge was defining a realistic and responsible project scope. ASL includes movement, facial expressions, body position, and its own grammar, so we limited our prototype to isolated static fingerspelling handshapes instead of claiming to translate complete ASL. We learned how computer vision, hand-landmark tracking, machine learning, and APIs can be combined in one accessible application. We also learned that building responsible assistive technology requires more than technical accuracy. Privacy, accessibility, user confirmation, clear limitations, and collaboration with the Deaf community are equally important. We would like to expand the training dataset with guidance from Deaf ASL users, qualified interpreters, and healthcare professionals. Future versions could improve accuracy and recognize movement, facial expressions, and body position. We would also strengthen privacy, accessibility, and independent testing before considering use in real healthcare environments.