Inspiration

In Indonesia, millions of individuals have hearing impairments, yet less than 1% of hospital Emergency Departments (IGD) have professional sign language interpreters on standby 24/7. When a patient arrives with acute pain or critical conditions, communicating through handwritten notes or phone typing is painfully slow, exhausting, and prone to severe misdiagnosis.

We asked ourselves: What if a patient knows exactly what they are feeling, but cannot convey it during the most critical triage minutes? This urgency inspired us to build SignMedic—a fast, safe, and privacy-preserving triage communication bridge that empowers patients and healthcare staff to communicate seamlessly from the very first minute of arrival.

What it does

SignMedic is an accessible, multimodal emergency triage communication web platform featuring:

  1. On-Device BISINDO Gesture Recognition: Real-time gesture classification for clinical signs running 100% client-side in the browser using MediaPipe and TensorFlow.js (<150ms inference latency). No video or facial data ever leaves the device.
  2. Fail-Safe Visual Triage Board: 28 interactive clinical triage cards (symptoms, pain scale 1–10, emergency alerts) designed to operate smoothly even with poor lighting, low camera resolution, or without internet.
  3. Medical Staff Speech-to-Text (STT): Automatically transcribes spoken questions from doctors and nurses into large, high-contrast text on the patient's screen.
  4. P2P Multi-Device Synchronization: Instantly pairs the patient's smartphone with the hospital triage desk monitor via WebRTC (QR code / 4-digit PIN), enabling real-time two-way triage without cloud intermediaries.

How we built it

  • Machine Learning & Feature Pipeline:
    • Trained on the open-source WL-BISINDO dataset (Glenn Leonali, Kaggle) and enriched with newly recorded clinical emergency signs (Cough, Fever, Cold, and Two-Handed Shortness of Breath) to address the extreme scarcity of Indonesian medical sign data.
    • Implemented Feature Specification v2.0.0, featuring Active-Hand First Tracking and Idle Table-Hand Suppression to automatically ignore passive hands resting on hospital tables.
    • Symmetrical inference architecture guaranteeing 100% mirror-invariance for left-handed patients and front/rear cameras.
  • Frontend & Web Architecture:
    • Modern web stack built with React 18, TypeScript, Tailwind CSS, and Vite.
    • Zero-latency multi-device communication powered by WebRTC via peerjs.
    • Thoroughly tested with 37/37 automated unit and integration tests in Vitest.

Challenges we ran into

  1. Severe Scarcity of Medical Sign Datasets: Existing BISINDO datasets focus on daily greetings and general vocabulary. To make SignMedic viable in an emergency room, we had to self-record and augment clinical emergency signs while enforcing strict validation protocols to avoid data leakage.
  2. Clinical Setting Noise (Resting Hands): Patients in triage often rest one arm on the table or their lap while gesturing with the other. Without filtering, standard trackers confuse the gesture as two-handed. We designed a temporal path-length and height scoring algorithm to suppress stationary hands automatically.
  3. Smartphone Camera Orientation & Inversion: Switching between front and rear cameras caused coordinate mirroring discrepancies. We engineered dynamic mirroring logic so video preview and skeleton drawing remain intuitive in both selfie and back-camera modes.
  4. Clinical Safety over Guesswork: In an ER, a wrong prediction can be dangerous. We implemented strict Out-of-Distribution (OOD) rejection (_OTHER_ class) and a 0.85 confidence threshold to prompt the user to the visual board whenever the AI is not absolutely confident.

Accomplishments that we're proud of

  • Rigorous Empirical ML Validation: Evaluated using a strict Clean Nested Leave-One-Subject-Out (LOSO) protocol across 5 independent signers:
    • 88.04% overall accuracy on completely unseen patients.
    • 99.68% clinical precision at the 0.85 threshold (307 out of 308 accepted predictions accurate).
    • 97.73% rejection rate for non-medical random gestures into _OTHER_.
  • True Privacy by Design: 100% in-browser processing with zero patient video stored or uploaded to servers.
  • Resilient Fallback Design: If lighting fails or camera access is denied, the patient is never locked out—the visual board provides complete independent triage communication.

What we learned

  • Designing for emergency healthcare requires prioritizing predictability and fail-safes over raw feature quantity.
  • An AI model in triage must know its own limits: saying "Gesture uncertain, please tap a visual card" is vastly safer than an unconfident guess.
  • Balancing accessible mobile UI (3:4 portrait view) with large medical dashboard desktop layouts across WebRTC.

What's next for SignMedic

  • Dataset Expansion: Partnering with medical professionals and healthcare organizations to expand medical sign vocabulary (e.g., specific pain locations, allergies, injury descriptions).
  • Hospital EHR / SIMRS Integration: Exporting standardized triage summary records directly into hospital management systems (via FHIR protocols).
  • Field Pilot Testing: Running real-world usability studies in partner hospital emergency rooms across Indonesia.

Built With

Share this project:

Updates

Submission history