Clara

Inspiration

Being understood at the doctor's office is easy to take for granted. Millions of patients don't speak their doctor's language, and more than 25 million people in the U.S. have limited English proficiency. For them, a routine visit can mean describing symptoms with gestures or through their child, nodding along to a diagnosis they didn't follow, and going home with discharge papers they can't read.

Professional interpreters are the best option, but they often aren't available on short notice, in small clinics, or for languages like Nepali, Somali, or Burmese. I also noticed that most translation tools assume one side speaks English. Doctors practice in Spanish, French, Arabic, and many other languages, and their patients often speak something else entirely. I wanted to build something that works between any two languages and needs nothing more than the phone the patient already has.

What it does

Clara lets a doctor and a patient talk by voice, each in their own language. The doctor picks their language and the patient's, and Clara shows a QR code. The patient scans it with their phone camera, with nothing to install, and the whole app appears in their language, down to every button and label.

Either person taps the mic and speaks. Clara transcribes the speech, translates it, and reads it aloud in a natural voice on the other person's device. Both people can see the original and the translation, and live status updates like "Patient is speaking…" and "Translating…" keep the conversation moving.

Clara supports 24 languages and translates directly between any two of them, with no English in between. A Spanish-speaking doctor can treat a Nepali-speaking patient just as easily as an English-speaking one can.

Real conversations are messy, so Clara is built for that. Patients mix languages ("मलाई headache छ"), use slang, and get misheard by speech recognition. Clara works out what was actually said before translating, and it's instructed never to invent symptoms or soften what the patient said.

After the visit, the doctor can write or dictate a summary, medication instructions, or care instructions, and the patient receives them in their own language to read, listen to, print, or reply to. Uploaded PDFs come back as the same document, with the same logo, tables, colors, and fonts, and only the text translated.

How we built it

The backend runs on Node.js and Express and uses Server-Sent Events to keep both devices in sync in real time. Multer handles uploads, and the qrcode package generates visit invites. A self-signed certificate enables local HTTPS so phones on the same Wi-Fi can use the microphone.

Translation runs on the Claude API. Claude Sonnet 5.5 interprets patient speech, since that's where reconstructing mixed-language or misheard speech matters most. Claude Haiku 5.5 translates doctor messages, documents, PDFs, and both interfaces. Speech-to-text uses Groq's Whisper large-v3, with ElevenLabs Scribe as an option. ElevenLabs multilingual voices read translations aloud, with a basic fallback voice if ElevenLabs isn't available.

The PDF translator is written in Python with PyMuPDF. It extracts text spans with their positions and styling, removes the original text, and typesets the translation back into the same space, using Google Noto fonts for each script and fontTools for font subsetting. The frontend is plain HTML, CSS, and JavaScript with no build step, using the Web Audio API and MediaRecorder for voice capture.

Challenges we ran into

Keeping layouts intact in PDF translation was the hardest part. Translated text is often longer than the original, and scripts like Devanagari, Arabic, Amharic, and Burmese take up space differently than English. I had to find the free space around each text box, avoid images and other content on the page, and ask the model to keep its wording short enough to fit, so nothing overlaps or runs off the page.

Scripts caused another problem. The model sometimes answered in the wrong writing system, such as Urdu written in Devanagari instead of Perso-Arabic. I added a check that detects the wrong script and retries the translation with a correction.

The patient interpretation also needed careful rules. People make small talk, complain, joke, and talk about family members. Clara has to interpret each message on its own terms without reshaping it to fit earlier symptoms or turning a comment about someone else into the patient's own complaint.

Accomplishments that we're proud of

I'm proud that Clara works between any pair of 24 languages, including ones that interpretation tools often skip, like Nepali, Somali, Haitian Creole, Amharic, and Burmese. I'm also proud that the patient side needs nothing but a phone camera: no account, no download, and an interface in their own language from the first screen.

The layout-preserving PDF translation is the feature I'm happiest with. Getting a discharge summary back with the same logo, tables, and formatting, only in the patient's language, makes it something they can actually use at home.

What we learned

I learned that in medical translation, accuracy matters more than fluency. A translation that reads smoothly but changes a dose or softens a symptom is worse than a plain one. Clara's translation rules say to never add, remove, or change medical information, drug names, doses, numbers, dates, or times.

I also learned to be honest about what a prototype can do. Clara's translations come from AI and can contain mistakes. It can help when no interpreter is available, but critical medical decisions should still be confirmed with a qualified medical interpreter.

What's next for Clara

The biggest step is training my own machine learning model for higher accuracy. General-purpose speech and translation models aren't built for medical conversations, and they struggle most where Clara matters most: less common languages, mixed-language speech, accents, and medical terms like drug names and dosages. I want to fine-tune a model on medical dialogue in the languages Clara supports, with a focus on lower-resource languages like Nepali, Somali, and Burmese, so it hears and translates symptoms, medications, and instructions more reliably. I'd measure it against professional interpreters to see where it improves on the current pipeline and where it still needs work.

Beyond that, I want to deploy Clara so clinics can use it without running a local server, add visit history so patients can return to their documents later, and look into the privacy and compliance requirements needed for real patient data.

Most of all, I want to test Clara in real clinics with the patients and doctors it's built for, and learn where it helps and where it still falls short.

Built With

• claude
• css
• elevenlabs
• express.js
• groq
• javascript
• node.js
• pymupdf
• python
• whisper