Inspiration

A lot of our family & friends are in pre-med and they shared how expensive it was just to prepare for mock clinical trials. Practicing real clinical conversations often requires paid actors, limited lab time, and institutional resources, making it hard to get enough repetition. We wanted to build a system that makes this kind of training affordable, accessible, and available anytime.

What it does

Our app turns clinical case practice into a fast, competitive experience. Users interact with AI-powered patients through chat or voice, just like a real standardized patient encounter. As they ask questions, the system tracks discovered symptoms, clinical reasoning, and efficiency (time, questions asked, etc.).

Each case ends with a structured debrief that compares the user’s diagnosis and approach, highlighting missed clues, unnecessary questions, and overall performance. Think of it like LeetCode, but for clinical reasoning and patient behavior interaction. Every result is computed through a scoring algorithm we designed using real OSCE-style evaluation guidelines.

How we built it

We used the Synthea repository to generate realistic patient profiles and medical conditions, giving us structured, high-quality clinical data to simulate real cases. We mapped these cases to standardized patient (SP) rubric items so we could evaluate users on communication and clinical reasoning.

For interaction, we used Gemini to power free-form patient responses, allowing users to have natural, open-ended conversations instead of selecting predefined options. We integrated ElevenLabs to enable real-time voice conversations, making the experience feel closer to an actual patient encounter.

Challenges we ran into

One of the biggest challenges was controlling the AI so it behaved like a real patient, only revealing information when asked, without giving away the diagnosis too easily. We had to ensure the model “knew” the underlying condition, but expressed it realistically as patients don’t state medical terms, they describe how they feel.

Accomplishments that we're proud of

We’re proud that we built a fully interactive conversation pipeline which is low latency and feels like a real conversation during mock trials. We are also proud of the Scoring Algorithm and State management we built for the mock trials.

What we learned

What's next for PatientZero

We want to push the 3D experience beyond visuals by adding animations and interactivity, making the patient feel more alive and responsive during the interview. This includes things like subtle facial expressions, body language, and reactions to questions to better simulate real clinical encounters.

On the educational side, we plan to introduce more advanced case formats, including doctor-to-student scenarios where users must ask targeted, technical questions and diagnose based on concise clinical responses. These would feel more like the fast-paced diagnostic challenges seen in short-form medical content, helping students sharpen high-level reasoning and decision-making under pressure.

Built With

  • elevenlabs
  • gemini
  • k2
  • next.js
  • react-three-fiber
  • sonarqube
  • supabase
  • synthea
  • vercel
Share this project:

Updates