CLARA

Inspiration

Before a doctor's visit, most of us do the same thing: try to remember when the pain started, which medication we're on, whether that surgery was 2018 or 2019, and then forget half of it in the exam room. Patient portals hold the records, but they're hard to use, especially for older, unwell or anxious patients.

We wanted something that felt less like a form and more like talking to a calm clinician who has already read your chart. That became CLARA, a voice-first health information assistant with a 3D avatar companion. She listens, asks one question at a time, can look at a photo or listen to a cough, flags emergencies, and helps you arrive at your appointment prepared. She never diagnoses.

What it does

Talk or type. Patients speak with a 3D avatar who lip-syncs in real time, or type if they prefer. Captions are on by default.
She knows your history. Encrypted records are compressed into a short briefing with personal details removed, so CLARA can say "Your record shows a left ACL reconstruction in 2019. That was your left knee, so this is likely separate."
Photo check. Patients can show CLARA a rash, mole or wound and get a plain-language description of what it could be consistent with, how urgent it seems, and which kind of clinician to see.
Sound check. Patients can record or upload a cough, their breathing or their voice. CLARA describes what she hears (a dry cough, a hoarse voice, a wheeze) as another clue for the visit.
Safety first. Emergency red flags (chest pain, stroke signs, trouble breathing, self-harm language) are detected in code, independently of any model, and the patient is told to call 911 or 988.
A real ending. The conversation closes with a "What to raise at your visit" summary and nearby clinicians from the national provider registry.
Accessible by default. Atkinson Hyperlegible type, text size up to 150%, high contrast, reduced motion, 44px touch targets, keyboard-navigable tabs and live screen-reader regions.

How we built it

App: Next.js 16 (App Router), React 19 and Tailwind v4, deployed on Vercel straight from GitHub.
Voice: ElevenLabs Conversational AI with signed-URL sessions and about 117 client tools that let the agent operate the site. Destructive actions (deleting records, photos, notes or the session) are deliberately kept out of the voice path.
Avatar: a rigged 3D model rendered with React Three Fiber. Her mouth moves with the audio's volume and spectrum, and she drags to any screen corner as a companion on other pages.
Photo check: Gemini reads the image and returns structured, schema-validated findings.
Sound check: ElevenLabs Scribe transcribes any words and tags non-speech sounds. Gemini then listens to the audio itself, with that transcript as context.
CLARA's words: for both checks, Claude turns the structured findings into a short, warm reply. Every reply must pass our diagnosis audit, and must mention 911 if the case is an emergency. Otherwise we show a plain message built straight from the findings.
Text fallback: a deterministic responder for when voice isn't available. It remembers what the patient already said (onset, trend, severity, what helps, other symptoms), so it never asks twice.

Identity without PII in storage. Patients sign in with name, date of birth and a passkey (Touch ID or Windows Hello) or a password. Their storage ID is derived, so no personal information appears in any key:

$$\text{patientId} = \operatorname{HMAC\text{-}SHA256}\big(K,\ \text{lower}(\text{name}) \,|\, \text{dob}\big)$$

Encryption before storage. Records and transcripts are encrypted in the app with AES-256-GCM. The object's own storage key is bound in as associated data, so a ciphertext can't be swapped between patients:

$$c = \operatorname{AES\text{-}GCM}_{k}\big(m;\ \text{AAD} = \text{objectKey}\big)$$

Finding care nearby. Live NPI Registry results are cached in Snowflake and ranked with ST_DISTANCE. Without Snowflake, we rank in memory with the haversine distance:

$$d = 2r \arcsin!\sqrt{\sin^2!\tfrac{\Delta\varphi}{2} + \cos\varphi_1 \cos\varphi_2 \sin^2!\tfrac{\Delta\lambda}{2}}$$

Small, consistent audio. Every recording or upload is mixed to mono and resampled in the browser to 16 kHz WAV, using linear interpolation:

$$y[n] = (1-t)\,x[\lfloor s n \rfloor] + t\,x[\lfloor s n \rfloor + 1], \qquad s = \tfrac{f_{\text{in}}}{16000},\ t = s n - \lfloor s n \rfloor$$

so a 30-second clip is about $30 \times 16000 \times 2 \approx 0.96$ MB, whatever device recorded it.

Challenges we ran into

Hearing a cough properly. Our voice provider transcribes speech beautifully, but it doesn't judge whether a voice is hoarse or a breath has a wheeze. We split the job: ElevenLabs Scribe transcribes and tags sounds, Gemini listens for voice and breathing qualities, and Claude explains the result. We also had to turn off the browser's noise suppression and echo cancellation, which are designed to erase exactly the coughs and breath sounds we wanted to keep.
Making CLARA listen. Our first text responder asked for the pain score the patient had just given, and confused a right knee with a left-knee surgery. We rebuilt it to track what the patient has said, repeat it back ("So far I have: about 5 out of 10, starting four days ago, a little worse"), and mention record facts only when they relate.
Ending the conversation. Early on, every follow-up after the summary got the same paragraph back. Now CLARA acknowledges new details ("Noted: numbness. I'll include that in your visit summary"), and the next-step buttons sit right under her latest reply.
Safe without being cold. A disclaimer under every message made CLARA feel defensive. We kept one persistent disclaimer bar and moved safety into the logic itself: emergency detection, a diagnosis audit on every reply, and no auto-dialing.
The avatar. With our first model, posing her arms flung strands of hair sideways, because the hair was weighted to the arm bones. We reshaped the hair at load time before eventually switching to a cleaner model, which needed a new loading image and updated tests.
Designing for worried people on phones. We ran repeated structured design critiques (heuristic scoring plus automated accessibility and layout checks) and rebuilt the consultation page around CLARA herself. The phone layout, a pinned reply box and a designed ending all came out of that process.
Three people, many branches. We merged parallel work on voice tools, maps, security and the avatar without losing anyone's changes.

What we learned

Trust in a health product comes from behaviour, not visuals. Repeating back what a patient said did more than any colour choice.
Critical safety checks shouldn't depend on a model following instructions. Enforcing them in code made the AI parts much easier to reason about.
Using several AI services is easiest when each does one job it's good at: Gemini perceives, Claude explains, ElevenLabs speaks, and our code decides what's safe to show.
Accessibility (text scaling, contrast, reduced motion, captions) is far cheaper to build in from day one than to add later.