SignAid

Inspiration

We decided to build this project after reflecting on how some of our friends and family members are deaf or hard of hearing, and we noticed how communication breaks down quickly if the people around them can't use sign language.

In fast paced emergency situations like being in an ambulance, at an ER front desk, or at an accident scene, every second matter, and a patient who communicates in ASL can have a hard time getting their needs and issues across to someone that can help them. We realized how workarounds like using an ASL-to-English chart, passing notes back and forth, or waiting for a translator can waste precious time that matters in these emergency situations,

So we decided to create an application that allows a computer to bridge those extremely crucial first minutes, letting the patient and medic understand each other effectively and fast.

What it does

In SignAid, the patient signs, the camera recognizes ASL hand shapes in real time, and reads the patient's answer aloud to the medic as a natural sentence.

If needed, the medic can also transcribe their questions and shown as text for the patient. Our application also asks the right questions without the need for a medic's input, asking basic questions to help create a clinical handoff summary that the ER staff or medic can read to help them effectively.

Also, the entire application is hands-free for patient, with signs like DONE, CLEAR, or DELETE allowing the patient to send and correct answers without touching the device.

How we built it

A web app in HTML, CSS and JavaScript
We built SignAid as a web app with the Next.js framework, allowing it to run in any modern browser with nothing to install
Google’s MediaPipe Hand Landmarker runs as WebAssembly directly in the browser with GPU acceleration, finding 21 landmarks on each hand in every webcam frame. The video never leaves the device.
The Web Speech API handles speech-to-text for the medic and text-to-speech for the patient’s answers.
A Python FastAPI server receives hand signs about 15 times a second over a WebSocket and replies with the recognized sign.
Answer modes limit which signs are compared at a certain moment which helps prevent symbol confusion
We recorded thousands of samples from four signers and built an evaluation script that tests each person against everyone else's samples
The Gemini integration helped turn the signed answers into natural sentences, it picks follow up questions, and it writes a structured handoff summary.

AI Tools we Used:

Claude (Claude Code, by Anthropic) helped us plan the architecture, write and review much of the code and tests, debug issues, and draft documentation.
Google Gemini runs inside SignAid itself, through the Gemini API, for the language features above.

Challenges we ran into

In the initial stages of training our model, we found that our model often mixed up signs with similar hand shapes but different meanings. After training with more data from different signers, we improved the model's accuracy.
We were relatively new to ASL, so it took some time for us to implement it as accurately as possible. We had to narrow down our ASL guide to the emergency essentials.
We had some difficulty navigating some of the development tools and setup, as we sometimes added our code to the wrong file and struggled to understand how tools like GitHub work initially. However, after some time, we grew more comfortable with using these tools.

Accomplishments that we're proud of

We effectively utilized MediaPipe's hand landmark detection system to tackle the specific problem that we identified
We utilized Gemini to generate questions for the patient to answer in the case that there isn't a medical professional present
All video processing is done on a single device, allowing for full privacy
Our user interface is clean and accessible, allowing any patient to pick up how it works quickly
This was our first official hackathon, and we are proud to have made a polished, working product

What we learned

We learned a large amount about AI and Machine Learning. Claude code assisted us with coding the backend, and brought our designs to life in the frontend. Regarding machine learning, we now understand the importance of good test data and how much of a difference lighting and positioning make on test data for a machine learning algorithm. We also learned some important signs in ASL, a topic all of us were unfamiliar with before this project.

What's next for SignAid

We learned a large amount about AI and Machine Learning. Claude code assisted us with coding the backend, and brought our designs to life in the frontend. Regarding machine learning, we now understand the importance of good test data and how much of a difference lighting and positioning make on test data for a machine learning algorithm. We also learned some important signs in ASL, a topic all of us were unfamiliar with before this project.