Bridge

Inspiration

Some people with cerebral palsy understand every word said to them but cannot speak back. Watching how conversations work for them today - spelling letter by letter, or pointing at the same static phrase board regardless of the question - made the gap obvious: the bottleneck isn't understanding, it's the interface. Modern AI can listen to a question and instantly propose relevant answers, so why should someone answer "Where does it hurt?" with a board designed for "yes/no"? Bridge was born from that mismatch: use AI to generate the choices, but never the choice.

What it does

Bridge is a real-time conversation aid. A clinician speaks naturally; Bridge transcribes the question over a live OpenAI Realtime voice session, then GPT-5.6 analyses it and generates 4–6 tailored response cards - yes/no, pain scale, feelings, locations, or open answers, always including "I'm not sure" and "ask me differently." The patient selects a card by eye gaze, single switch, or keyboard, and Bridge speaks their answer aloud instantly. Session memory lets follow-up questions build on earlier turns, and clinicians can optionally load patient context (interests, vocabulary, visit history) so cards feel personal. The core rule never bends: AI proposes, only the patient chooses.

How we built it

React frontend with WebRTC audio into OpenAI's Realtime API (gpt-realtime-2.1) for low-latency listening and speech, with semantic voice activity detection set to transcribe-only so the model never answers on the patient's behalf. Card generation runs as a parallel path: GPT-5.6 via the Responses API with structured output, wrapped in a bounded plan → validate → revise agent loop - the model picks an AAC strategy, deterministic guardrails check card count, label length, uniqueness, and uncertainty coverage, and the model gets exactly one revision if a check fails. An Express server keeps the API key off the browser. The whole thing was designed, implemented, and debugged in Codex, which worked across the React/WebRTC/Express/SDK stack in one continuous feedback loop.

Challenges we ran into

Keeping the voice model silent was surprisingly hard - Realtime sessions want to reply, so we had to disable automatic responses and trigger speech only from an explicit card selection. Separating fast audio from slower reasoning took real architecture work: card planning can't block the live session, and stale card requests had to be aborted so an old answer never overwrites a new question. Repeated-card behaviour on follow-up questions led us to design bounded session memory that treats past selections as historical statements, not assumed preferences. And plenty of unglamorous debugging: Windows PowerShell execution policies, port conflicts, key loading, and network failures, diagnosed layer by layer.

Accomplishments that we're proud of

A genuinely agentic but bounded system - the model adapts its strategy per question, yet deterministic gates and patient selection keep it safe. Sub-conversational latency: cards appear while the question still hangs in the air. Failing honestly: if generation fails, Bridge never substitutes generic cards that could be mistaken for the patient's words. And accessibility built in from the start - switch and keyboard navigation, stable card layout for eye-gaze targeting, ARIA live regions - not bolted on afterwards.

What we learned

The hardest design problems weren't technical, they were ethical-made-technical: what does the AI not get to do? Every important decision - no inferred intent, no automatic replies, opt-in context, session-only memory - was about constraining capability, not adding it. We also learned that agentic AI works best when bounded: a plan-validate-revise loop with one retry beats an open-ended agent for anything safety-critical. And Codex taught us how much faster iteration gets when the development agent can see the whole stack, from a UI symptom to the API boundary causing it.

What's next for BRIDGE

Working with speech-language pathologists and AAC users to validate vocabulary, scanning timing, and consent flows with real eye-gaze and switch hardware. Configurable auto-scan for single-switch users. Session renewal handling for conversations beyond 60 minutes. Proper clinical data foundations - authentication, encryption, audit logs, and FHIR integration - so patient context can move from a demo JSON file to real, governed records. The prototype proves the interaction; the next step is proving it in a clinic

Built With

• cerebralpalsy
• codex
• gpt-5.6
• gpt-realtime-2.1
• text-to-speech