MedSpeak

Inspiration

Healthcare communication has two major problems that go largely unsolved: patients forget what their doctor said, and language barriers prevent effective care. We've all been in a rushed appointment, nodding along, only to leave unsure of our diagnosis or what medication to take. For non-English speaking patients, this problem is even more serious — critical medical information gets lost in translation.

MedSpeak was built to solve both problems.

What it does

MedSpeak is a three-feature AI medical assistant:

1. Visit Recap — Record or type what was said during a doctor's appointment. Our AI scribe automatically extracts diagnosis, observations, medications, instructions, red flags, follow-up timeline, and generates questions for the next visit — all translated into English, Arabic, Spanish, and French.

2. Live Translation — A real-time push-to-talk translator for doctor-patient conversations. The doctor speaks English, the patient hears Arabic (or any supported language). The patient responds in their language and the doctor receives the English translation instantly with voice playback.

3. Health Reference — An interactive health metrics dashboard covering 17 medical indicators including blood sugar, blood pressure, BMI, thyroid, vitamin D, uric acid, and more. Users slide to their value and instantly see color-coded normal ranges. Includes a bilingual wellness self-assessment checklist.

How we built it

• Frontend: React + TypeScript + Tailwind CSS, deployed on Miaoda platform
• Backend: Supabase Edge Functions (Deno) for secure API key management
• AI: OpenAI GPT-4o for medical note processing and translation, Whisper for audio transcription
• Voice: Web Speech API for real-time recording, SpeechSynthesis for audio playback
• Storage: localStorage for visit history, Supabase for edge function hosting

Challenges we faced

The biggest challenge was real-time bilingual speech recognition. The Web Speech API only listens in one language at a time, so we couldn't truly auto-detect Arabic vs English simultaneously. We went through multiple architectures — from SpeechRecognition language-switching to MediaRecorder + Whisper — before settling on a push-to-talk approach that is reliable and clinically practical.

Mobile voice recording was another challenge — Android Chrome's speech service triggers system notification sounds on every restart, and word duplication was common. We solved this with a transcript merging algorithm that detects overlapping text and prevents repetition.

The AI prompt engineering for Visit Recap required significant iteration to prevent over-diagnosis, correctly classify scheduling information as follow-up timelines, and handle conversations where no doctor label was present.

What we learned

• Prompt engineering for medical AI requires strict rules to prevent hallucination
• Browser speech APIs have significant cross-platform inconsistencies
• Post-processing AI output in the frontend is often more reliable than relying on prompt changes alone when backend deployment is restricted
• Building for multilingual users requires thinking about RTL layouts, font rendering, and cultural context from day one

What's next

• Android and iOS native apps using React Native
• OpenAI Realtime API for truly hands-free continuous live translation
• User authentication and cloud-synced visit history
• Integration with hospital EHR systems

Built With

• deno
• lucide
• openai-gpt-4o
• openai-whisper
• react
• shadcn/ui
• supabase
• supabase-edge-functions
• tailwind-css
• typescript
• vite
• web-speech-api