Witness

Inspiration

Clinical trials are moving into people's homes. Participants now give themselves injections, use inhalers and collect samples without a nurse in the room, and many of them are older adults who live alone. The study team usually only finds out something went wrong after it happened, or never. We wanted a calm coach for the participant and a trustworthy witness for the study team: something that guides each step, notices when an important one is skipped, and says "stop, call your nurse" instead of guessing.

What it does

Witness turns a phone into a step-by-step coach for at-home trial procedures.

For the participant (opened from a link, nothing to install):

A short practice run checks that the camera can see their hands.
The procedure appears one step at a time in large text, with a voice coach reading each step aloud.
The phone camera checks each step as it happens. Timed steps like "press and hold for 10 seconds" show a countdown ring that only runs while the step is actually visible.
If a critical step is skipped or a hold ends early, Witness stops the procedure and shows "Stop. Call your study nurse" with a one-tap call button.
Pause and Call my nurse are always on screen, and every voice command ("pause", "continue", "repeat", "done", "call my nurse") also has a button.

For the study team:

Paste a procedure in plain text and Witness compiles it into structured steps. A coordinator reviews, edits and publishes it; nothing goes live automatically.
A live dashboard shows every session. Each session has a time-stamped timeline: when every step was confirmed, how (camera, timed hold, spoken, tapped) and the evidence frame behind it.
Critical misses, skipped steps and nurse calls become flags and (simulated) escalation alerts that a person marks as reviewed.

Safety by design: Witness flags and records, and a human makes every medical decision. It never gives dosing advice. Participants only hear lines the study team reviewed, never model-written text. When it isn't sure, it stops instead of saying "carry on".

How we built it

Next.js 16 (App Router), TypeScript in strict mode, Tailwind CSS 4 and Zod validation everywhere.
A pure step engine at the core: no I/O, no clock, fully deterministic. A step counts as confirmed only after $k = 3$ consecutive matching frames with confidence $\ge 0.7$. Skips need the same evidence, so one bad frame can never advance or stop a procedure. Beyond 40+ scenario tests, property-based tests (fast-check) throw thousands of random input sequences at it and check rules like "a critical step is never silently skipped" and "completed means every critical step was confirmed".
Gemini vision: about 1 frame per second goes to a server route that asks Gemini 3.5 Flash-Lite which step the frame shows. The model can only pick from the current step plus a few ahead, or "none". Results come back in about 1 second.
Gemini compiler: Gemini 3.8 Flash with structured output compiles protocol text into steps. A deterministic cleanup pass and one automatic repair round-trip handle malformed output, and a lint step flags any dose language for human review.
Voice: browser speech by default, plus Gemini Live via short-lived ephemeral tokens. Live is used only to recognize commands through function calling, and anything that moves the procedure forward must also pass our own parser.
Supabase (Postgres + Storage) with row-level security that denies all browser access by default, an append-only event log enforced by a database trigger, immutable published protocols, and evidence frames in a private bucket served through short-lived signed URLs.
Deployed on Netlify. Tested with Vitest (288 unit tests) and Playwright end-to-end flows with axe accessibility checks in light and dark mode at phone width.
We used AI coding assistants (Claude Code and Codex) heavily, with our own architecture, review and testing driving the work.

Challenges we ran into

Speed versus accuracy. Our first vision model took 7–20 seconds per frame, or returned "high demand" errors. Measuring alternatives led us to Flash-Lite at about 0.8 seconds per frame, fast enough for real-time coaching.
Frames arrive out of order. With model latency longer than the gap between frames, results come back late and shuffled. Sequence numbers, single-flight requests and stale-frame dropping keep the engine honest.
"Hold for 10 seconds" isn't a single frame. We had to model holds as durations that only count while the step stays visible, with early release on a critical step treated as a critical miss.
Voice that stays in its lane. Live testing showed short phrases like "I'm done" were never picked up until we tuned speech detection. The model also sometimes tried to talk despite instructions, so we discard its audio entirely and keep every spoken line app-authored.
Designing for older adults. Large type, buttons at least 64px tall, one step per screen, WCAG AA contrast, and no screen that depends on an animation finishing.

Accomplishments that we're proud of

A safety core we trust: the "never silently skip a critical step" rule is proven by randomized tests, not just examples.
The full loop works end to end on a live deployment: coordinator creates a link → participant is coached → a critical miss stops the procedure → the study team reviews the timeline with evidence.
A calm, accessible interface that holds up under automated accessibility checks.
A protocol compiler that turns plain text into reviewable steps, so Witness generalizes beyond a single procedure.

What we learned

In healthcare, "stop and ask a human" is a feature, and building the uncertainty paths took as much work as the happy path.
Measure models before committing to them; the "best" model on paper was the wrong one for real-time video.
Keeping AI in a narrow, checkable role (classify a frame, recognize a command) makes it far easier to make safe.

What's next for Witness

Usability sessions with older adults on real phones, plus accuracy evaluation of camera recognition on recorded procedures.
Real SMS and phone escalation to nurses and consented trusted contacts (simulated today).
The path to real patients: a clinical validation study, IRB review, HIPAA compliance and BAAs with vendors, audit logging, and an FDA device-software review.
More procedures, multiple languages, and a native app for more reliable camera and voice.

Hackathon prototype. Synthetic participants only; the demo procedure uses a prop and is not medical guidance.