Mediq

Touchless clinical workstation co-assistant: navigate patient records with mid-air gestures and verified voice, backed by dual-AI checks to prevent medical errors.

Comment

Inspiration

In surgical operating rooms, trauma bays, and sterile cleanrooms, clinicians face a dangerous operational dilemma:

  1. The Sterility Dilemma: Consulting an Electronic Health Record (EHR), inspecting a CT scan, or verifying vitals forces medical staff to break scrub, deglove, and touch contaminated keyboards/screens—or yell to unverified assistants.
  2. The Ambient Voice Flaw: Standard voice assistants fail open. Anyone walking into the room—a visitor, an orderly, or ambient hospital noise—can trigger record modifications.

We asked: What if clinical workstations were completely touchless, operated through sterile air gestures and fail-closed biometric voice verification?

We engineered Mediq: the first zero-contamination clinical workstation co-pilot powered by on-device biometric defense, sterile 3D computer vision, and dual-model AI consensus.

What It Does

Mediq transforms standard hospital hardware into an air-gapped, touchless clinical command centre:

  • Fail-Closed On-Device Voice Biometrics: Spoken commands ("Open patient 104") are intercepted by an on-device WeSpeaker ResNet34 ONNX model in WebAssembly. It evaluates acoustic consistency and extracts 256-D speaker embeddings in under 40ms. If cosine similarity drops below threshold ($\cos(\theta) < 0.50$), it fails closed—access is denied, and no audio ever leaves the browser.
  • Sterile 3D Air Gestures: MediaPipe Vision running in WASM tracks 21 hand landmarks at 30+ FPS. Gloved clinicians can pinch-drag floating lab reports, two-hand zoom CT scans, and scroll vitals in mid-air—zero physical glass touched.
  • Dual-Model Clinical Consensus:
    • Drafting Engine: Open-weight Hermes 3 synthesizes doctor dictations into structured SOAP notes.
    • Safety Auditing Engine: Google Gemini independently stress-tests prescriptions against EHR history.
  • Lethal Allergy Intercept: Proactively catches dangerous contraindications (e.g., cross-reactive aminopenicillins for penicillin-allergic patients) before orders reach the pharmacy.
  • Deterministic Air-Gapped Fallback: Hardcoded clinical rule engines ensure contraindication alerts never fail even during hospital internet outages.

How We Built It

  • Frontend & UX: Built with React 19, TypeScript, and Tailwind CSS, bundled with Vite for sub-millisecond hot reloads.
  • Edge Biometrics & Vision: ONNX Runtime Web (WASM) executes WeSpeaker ResNet34 models directly in the client. MediaPipe Tasks Vision delivers zero-latency 3D hand tracking in the browser.
  • State Synchronization: Full-duplex WebSockets for instant real-time telemetry between the local workstation and background services.
  • AI & Auditing: Multi-model pipeline combining DigitalOcean GenAI (Hermes 3) for structured drafting and Google Gemini API for adversarial safety auditing.
  • Verification & Testing: Rigorous test harness featuring 101+ automated tests across Vitest (unit/integration) and Playwright (end-to-end browser flows).

Challenges We Faced

  • Real-Time WASM Concurrency: Running 256-dimensional acoustic embedding extraction alongside 21-point hand tracking in browser threads at 30+ FPS required careful memory management to eliminate frame drops.
  • Acoustic Edge Cases: Distinguishing hoarse clinician voices and surgical mask attenuation from background OR alarms while enforcing an absolute fail-closed posture.
  • Eliminating Confirmation Bias: Ensuring the auditing model evaluated notes with zero shared context from the drafting LLM, guaranteeing true adversarial safety.

Accomplishments That We're Proud Of

  • 🥈 Awarded 2nd Place at Hacktoberfest Hack Day Guelph × Google Developer Groups (GDG).
  • Achieved sub-40ms on-device speaker verification with zero audio transmission to cloud servers.
  • Authored a 101-test automated suite (Vitest + Playwright) enforcing zero hallucinated medical dosages.
  • Packaged the entire architecture into a reproducible, single-command docker compose up deployment.

What We Learned

  • Deepened our mastery of WebAssembly, PCM audio streaming via the Web Audio API, and client-side ONNX runtime execution.
  • The imperative need for adversarial multi-model consensus when deploying generative AI into mission-critical clinical environments.
  • Designing interfaces for extreme physical environments where zero touch is a biological necessity.

What's Next for Mediq

  • Direct integration with hospital HL7 / FHIR data standards (Epic Systems, Cerner).
  • Multi-microphone spatial array beamforming to isolate lead surgeon audio in noisy trauma rooms.
  • Native packaging for sterile hospital cart displays and medical-grade embedded units.

Built With

Share this project:

Updates