Inspiration

Seven in ten deaf people and people with hearing loss have never been asked how they need to communicate in NHS care (RNID and SignHealth, Still Ignored, 2025). So when communication breaks down in a GP appointment, the usual fix is to hand the patient a pad of paper or a phone and ask them to type.

That workaround puts the burden on the patient. It slows the consultation, and for many sign language users, written English is a second language, not their first. It turns a conversation into a form, and the patient stops being a full participant in their own care.

We wanted to flip the model: let the patient sign in their own language and be heard.

What it does

SignBridge lets a deaf patient sign to their GP and be heard in their own consultation.

  • Sign: the patient signs to the webcam. Both hands are tracked and matched to a sign.
  • Write: the signs become one plain sentence on screen, using the doctor’s last question as context.
  • Approve: the patient confirms with a thumbs-up, and only then is that exact sentence spoken aloud.
  • Response: the doctor speaks, and the patient reads the reply in plain words, with any medical jargon explained.
  • Keep: at the end, the patient gets a plain-words visit summary they can download and show at their next appointment. Nothing is stored unless they choose to.

How we built it

  • Tracking: MediaPipe hand and pose landmarks from a webcam, classified by a small model we trained.
  • Training data: we recorded every sign ourselves, on our own webcam.
  • Sentences: the recognised signs and the doctor’s last question go to our language model, which writes one short first-person sentence and never adds symptoms or times that weren’t signed. We built its training data with -Adaption: 66 hand-written examples, grown to 1,074 using only our signs, then improved by Adaption (data quality D → C). We fine-tuned a small open model (Qwen 3.5 4B) with Adaption’s AutoScientist, and it was preferred over the base model 64.6% of the time. Claude is the live fallback.
  • Voice: ElevenLabs speaks the approved sentence, and Maya from Anam speaks it lip-synced.
  • Doctor’s side: ElevenLabs Scribe transcribes the doctor live, and Claude rewrites it in plain words and explains jargon (“analgesia” → “painkillers, like paracetamol”).
  • Interface: a React and TypeScript web app with a laptop view and a phone view, a tabletop mode that faces the doctor, live sign chips, word-by-word captions, and a visit summary the patient can keep.

Challenges we ran into

Public sign datasets didn’t fit, so we recorded our own.

Fingerspelling sets (ASL Alphabet, Sign MNIST): these are still images of letters, not moving word signs like PAIN. Word-level sets (WLASL): about 2,000 words as videos, but they’re large and messy, and many source links are reportedly dead. Every clip would also need running through our tracker first. Google’s Kaggle sign dataset: a general 250-sign vocabulary, not built around medical consultations. Different cameras: a model trained on other people’s phone videos tends to do worse on a laptop webcam than one trained on takes from that same webcam.

So we recorded every sign ourselves, on one camera, and kept the vocabulary small enough to be reliable.

Generated training data drifted. Adaption’s generic augmentation invented signs our recogniser can’t read and wrote 150-word explanations instead of one sentence. Inspecting the data caught it, so we constrained generation to our vocabulary and the patient’s voice before training.

Safety. A wrong sentence spoken aloud in a clinic is worse than none. Nothing is spoken until the patient approves it, and anything the recogniser isn’t sure about is held back.

Accomplishments that we're proud of

A full consultation loop working end to end in a weekend: sign, approve, speak, and a plain-language rewrite of the doctor’s reply. The patient stays in control at every step, and keeps their own record of the visit.

What we learned

  • A small closed vocabulary plus context from the doctor’s question is more reliable than open-ended recognition for a high-stakes setting.
  • Recording on the same camera you demo on matters more than the amount of data.
  • Checking generated data matters as much as the model: our first augmented set would have taught the model to ramble.
  • The approval step is the feature that makes it trustworthy.

What's next for SignBridge

  • BSL, which is what most UK sign language users use (ASL is the prototype).
  • A larger vocabulary, recorded with Deaf signers.
  • Testing with Deaf patients and GPs before anything is used in a real consultation.

Built With

Share this project:

Updates

Submission history