Signapse

Sign it. Point to it. Be understood. Everything ASL users need at the hospital, in one place

Comment

Signapse

Sign it. Point to it. Be understood.

Signapse is an all-in-one communication platform that helps Deaf and ASL-using patients communicate with healthcare professionals, from explaining their symptoms to understanding a doctor's response and calling their care team.

Live at signapse.health

Inspiration

Imagine arriving at an emergency room in pain, unable to communicate your symptoms clearly while waiting for an interpreter. Every minute matters, but being understood shouldn't depend on how quickly someone can arrive to interpret.

For Deaf and ASL-using patients, communication involves more than translating signs into words. In American Sign Language (ASL), facial expressions and head movements carry grammatical meaning. Raised eyebrows can indicate a question, a headshake can express negation, and a furrowed brow can intensify a message. Losing these signals can mean losing important context.

We wanted to build a tool that considers the whole message: the hands, the face, and the intended tone. More importantly, we wanted to support communication throughout the healthcare experience, not just at the first point of contact.

That's why we built Signapse.

What it does

Signapse brings patient-to-doctor communication, accessible medical responses, and follow-up calls into one platform.

1. Understands the hands and the face

Signapse tracks hand shapes alongside facial expressions and head movements to identify grammatical signals such as questions, negation, and emphasis. When a message depends on a facial marker, Signapse flags it for confirmation rather than silently assuming its meaning.

2. Puts patients in control

After signing, patients see Signapse's interpretation in large, easy-to-read text. They can review, edit, send, or redo their message before it reaches medical staff.

Nothing is sent until the patient presses Send. This gives patients an opportunity to catch misunderstandings and remain in control of their communication.

3. Preserves meaning through speech

Signapse transforms recognized signs into a natural-language clinical sentence while accounting for facial grammar.

For example, a sequence such as "PAIN CHEST TIGHT SINCE MORNING" can become a clearer clinical sentence for medical staff to review.

Its tone-aware text-to-speech adjusts pitch, pace, and volume based on the identified grammatical signals, helping distinguish questions, intensified messages, negation, and neutral statements. The goal is to communicate more than the words alone.

4. Gives medical staff a clearer clinical picture

On the clinician's screen, Signapse organizes the patient's message into a structured summary with fields such as complaint, onset, and location.

It also includes:

  • Urgency indicators and an alert chime to draw attention to potentially urgent complaints.
  • An interactive body map for indicating the location of pain.
  • A 0–10 pain scale to help patients communicate symptom severity.

This gives medical staff a more organized starting point for reviewing the patient's concerns.

5. Makes the doctor's response accessible

Communication goes both ways. Doctors can type or speak a response, which Signapse presents to patients using concise text, recognizable icons, and large YES and NO buttons.

Whether the doctor is discussing an X-ray, medication, drinking water, or waiting, patients have a more accessible way to understand the next steps and communicate their answers. Responses are also recorded in the doctor's log.

6. Brings accessible communication to phone calls

Communication shouldn't stop when a patient leaves the hospital. Signapse extends its translation workflow to calls with a patient's care team.

During a supported call, patients can sign to their camera while Signapse translates their message into speech for the doctor. The doctor's replies appear on the patient's screen as text and icons, helping both sides communicate when signing and spoken conversation would otherwise be difficult to coordinate.

7. Keeps a reviewable conversation record

Signapse saves conversation exchanges and pain readings to Tiger Data. Medical staff can review the conversation, verify individual lines, and draft a chart note.

Uncertain lines are labelled for review, helping keep clinicians involved in checking what was understood before information is used in documentation.

8. Prioritizes patient privacy

Signapse processes camera frames and hand and facial landmarks on the device. Only text and numerical information are sent to the backend, rather than patient video.

A live counter displays the number of video frames sent, showing 0 in the current implementation.

How we built it

Signapse combines computer vision, sign recognition, language processing, speech synthesis, and a real-time patient–clinician interface.

  • Computer vision: Python, MediaPipe, and OpenCV track hand and facial landmarks, including eyebrow position, squint, and head movement. FastAPI exposes the facial-grammar processing service.
  • Sign recognition: PyTorch and TensorFlow support the sign-recognition pipeline.
  • Language processing: The Google Gemini API helps transform recognized sign sequences and grammar flags into readable clinical sentences and tone labels.
  • Speech synthesis: The Web Speech API generates spoken output, with pitch, pace, and volume adjusted to reflect the identified tone. The Web Audio API provides the alert chime.
  • Frontend: React 18, TypeScript, and Vite power the patient and clinician interfaces, supported by custom CSS design tokens and an interactive SVG body map.
  • Backend: Node.js, Hono, Zod, and WebSockets connect the application's components and synchronize communication between interfaces.
  • Data storage: Tiger Data stores conversation records and pain readings.
  • Hosting and deployment: Signapse is live at signapse.health, with the frontend hosted on Cloudflare Workers. The domain is registered through GoDaddy, with DNS managed through Cloudflare, and the API is hosted on Render.
  • Development workflow: We organized the project as an npm workspaces monorepo and collaborated using Git and GitHub.

Challenges we ran into

Recognizing continuous signing. Sign language recognition is complex, especially when signs flow together naturally. We had to work within a limited vocabulary and focus on building a usable end-to-end workflow.

Interpreting facial grammar. Eyebrow positions and head movements vary between individuals. Distinguishing a meaningful grammatical signal from someone's natural resting expression is a challenge that requires careful calibration.

Preserving tone in synthesized speech. Browser-based voices have limitations, so translating grammatical signals into appropriate pitch, pace, and volume required balancing expressiveness with intelligibility.

Connecting different technologies in real time. Integrating Python-based vision processing with a Node.js backend and synchronized frontend interfaces introduced challenges around communication and responsiveness.

Handling real-world camera conditions. Poor lighting and hands crossing in front of the face can make visual tracking less reliable, highlighting the need for further testing across different environments and signers.

Accomplishments that we're proud of

  • Looking beyond hand shapes: We incorporated facial grammar into the communication pipeline, helping preserve distinctions such as questions, negation, and emphasis.
  • Building the complete communication loop: Signapse connects signing, patient review, clinical summaries, accessible doctor responses, follow-up calls, and reviewable conversation records.
  • Putting patients in control: Patients can review and correct their messages before sending them.
  • Designing around privacy: Camera processing happens on the device, with only text and numerical data sent to the backend.
  • Keeping people involved: Uncertain interpretations are flagged for review instead of being treated as unquestionable results.
  • Shipping it live: Signapse is deployed and publicly reachable at signapse.health, with its API running on Render.

What we learned

Building Signapse reinforced that accessible communication is about more than converting one form of language into another. In ASL, facial expressions and movement can change the meaning of a message. In healthcare, missing that context can matter.

We also learned that an effective assistive tool needs to consider the entire experience: what a patient communicates, what a clinician understands, how the clinician responds, and how that conversation is documented.

Finally, uncertainty should be made visible. Rather than presenting every interpretation as definitive, systems like Signapse should make it easier for patients and qualified healthcare professionals to verify meaning.

What's next for Signapse

We want to continue developing Signapse with Deaf community members, ASL users, interpreters, and healthcare professionals.

Our next steps include:

  • Expanding the recognition vocabulary and training across a wider range of signers.
  • Testing usability and interpretation quality with Deaf users and clinicians.
  • Improving tone-aware speech synthesis for clearer, more natural communication.
  • Exploring clinician dictation and integration with electronic health records.
  • Investigating support for additional sign languages.

Our goal is not to replace human interpreters. It is to give Deaf and ASL-using patients another way to communicate especially in emergencies, support healthcare professionals in understanding their needs, and help qualified interpreters remain part of the care process when needed.

With Signapse, patients can sign it, point to it, and be understood.

Built With

Share this project:

Updates

Submission history