Inspiration

Imagine walking into an emergency room in pain, and nobody there speaks your language. For a lot of patients with hearing disabilities, that is just a normal ER visit.

A 2025 study in Academic Emergency Medicine looked at Deaf and Hard of Hearing patients. Only about 29% of them could communicate on their own with ER staff. Deaf ASL users also stay around 30 minutes longer in the ER than other patients who are just as sick. Hospitals do have video interpreting. But it takes time to set up, and in one survey about 59% of Deaf patients were not satisfied with it.

We kept thinking about those first few minutes, before an interpreter shows up. We didn't want to replace interpreters. We wanted something that works right away, while everyone waits.

What it does

SignWave works in both directions.

When the doctor talks, the patient sees big live captions. Above them, Shasta signs the doctor's words, one word at a time. The word being signed lights up in the caption. If a caption is wrong, the doctor can click it and fix it, or just type.

When the patient signs, the app reads one sign at a time from the webcam. They can edit it or start over. Nothing goes to the doctor until the patient taps Confirm & send. On the doctor's screen, the message shows up marked as confirmed by the patient. The doctor can also play it out loud in a natural voice.

How we built it

The backend is Python with FastAPI. It owns the camera and the mic, and it pushes everything to two web pages over websockets. The frontend is React, with Three.js for the 3D cougar.

For the doctor's speech, we run ElevenLabs scribe-v2. Silero VAD sits in front of it, so ElevenLabs only hears speech. We also give the model a short list of ER words so it spells drug names correctly.

For the patient's signing, MediaPipe tracks the hands, face and body. Each sign goes into the model from Microsoft's ASL Citizen project. Currently, it knows 2,731 signs. We narrowed it down to 129 signs that matter in an ER: yes, no, the pain scale, body parts, symptoms, family and medicine. That made it a lot more accurate.

Shasta was the fun part. We didn't animate anything by hand, and nothing is generated. For each word, we pull a real video of a Deaf signer from ASL Citizen. MediaPipe tracks 178 points on the body, face and hands in every frame. Then we pose the cougar directly on those points. Each finger is a small cylinder between two tracked joints, so the handshape is the signer's own. The jaw, eyebrows and eyelids follow the signer's face too. We blend each sign into the next so the motion flows, and we cache the results so common words play right away. Words that aren't in the dataset, like most drug names, come from the open sign.mt project, which usually fingerspells them.

Challenges we ran into

Full sentence translation was our big heartbreak. We got a sentence level research model running live, and it did fine on news clips. Then we gave it videos of Deaf signers signing STOMACH HURT, and it said "He was heartbroken." It had barely seen any medical signing. Even when we gave it a list of ER phrases to choose from, it picked the right one first only 53% of the time. So we switched it off and went with one sign at a time, which gets it right 86% of the time. Shipping the less flashy version was the right call.

Turning the doctor's words into signing was slow at first. Four new words took over 30 seconds. Part of that was a bug. Our clip quality check counted the resting frames before and after each sign, so almost every clip failed and we processed two more. We fixed the check, fetched signs in parallel, and started fetching while the doctor was still talking. Now four new words take about 4.5 seconds, and words we've seen before start right away.

The room itself was a challenge. It's loud and full of people talking. We found that the voice detector treats faint speech from across the room as speech. Captions stayed open for several seconds and picked up other people's words. So we added a loudness gate. Quiet sound from farther away now counts as silence, and a caption closes about half a second after the doctor stops.

Accomplishments that we're proud of

On ER signs, our model's first guess is right 86% of the time, and the right sign is in its top three 98.5% of the time.

On the speech side, ElevenLabs Scribe got a 0.9% word error rate on our ER sentences, and it got every medical term right.

We're proud of Shasta too. Every sign it makes is a real Deaf signer's recorded motion. We track that motion with MediaPipe and pose the cougar on it, with no hand animation at all. Meanwhile, the camera side keeps up at 30 frames per second on our device.

What we learned

General video AI can't read ASL yet. Published results put it around 1 BLEU on ASL, which is close to nothing. Models trained on Deaf signers are still the way to go.

A small system that is honest beats a big one that guesses. In an ER, 129 words that work are more useful than thousands that don't.

Sign language is much more than hands. Facial expressions and word order carry a lot of the grammar. Our cougar signs word by word in English order, and we learned how much that leaves out.

And the person who knows what they meant should be the one who checks it.

What's next for SignWave

First, we want to put it in front of Deaf users and ASL interpreters and listen to what they tell us.

After that, we want to test speech recognition with real voices and real ER noise. We want to train the sentence model on ER phrases so we can turn it back on. We also want to add fingerspelling for names and drug names, and teach the cougar ASL word order and facial grammar. Longer term, we'd love to pilot SignWave in a real emergency department with a Deaf advisory group.

SignWave is a research prototype. It is not a certified medical interpreter, and a qualified interpreter is still the right choice for care. We see it as something that helps until that interpreter arrives.