Gemini Clinical Synthesizer | Healthcare

The Clinical Synthesizer uses multimodal AI to turn chaotic handwritten documents into one unified, standards-compliant record with accurate SNOMED/RxNorm coding and automating medical record keeping.

Comment

Project Track: Healthcare

Inspiration

Doctors and medical staff spend countless hours manually pulling data from handwritten or scanned notes. Each chart holds vital information, but it’s buried in messy formats. We wanted to build a tool that could read clinical documents, extract the key information into standardized codes (SNOMED CT & RxNorm), and let people update it through natural conversation.

That idea became Gemini Clinical Synthesizer — an app that turns unstructured notes into structured patient records and lets you manage them just by chatting.

What it does

Gemini Clinical Synthesizer automatically:

  1. Performs OCR and understanding on uploaded medical notes using Google Gemini.
  2. Generates a unified JSON record with standardized problems (SNOMED CT) and medications (RxNorm).
  3. Displays an interactive dashboard showing all current conditions and medications.
  4. Includes a conversational assistant that can add, remove, or update items (e.g., “Add 10 mg Prednisone”).
  5. Saves everything to Firestore so changes persist instantly across sessions.

The assistant even regenerates the Quick Summary automatically whenever the data changes.

How we built it

  • Frontend: Streamlit (Python) for a lightweight, reactive UI.
  • AI Backend: Google Gemini 2.5 Flash with structured JSON response schemas.
  • Database: Google Cloud Firestore for patient-level persistence.
  • Schema Validation: Pydantic models enforcing correct SNOMED/RxNorm format.
  • Real-Time Updates: Session-state caching and dynamic reruns keep chat, data, and tables in sync.

Challenges we ran into

  • Keeping chat history and Firestore data synchronized through Streamlit’s rerun model.
  • Persisting data in Firestore
  • Making the input bar “sticky” at the bottom while maintaining scrollable chat.
  • Reducing Gemini and Firestore load times through caching and lazy initialization.

Accomplishments that we're proud of

  • Built a fully functional LLM-powered clinical assistant that updates structured data in real time.
  • Designed a UI that feels natural to use, combining conversational AI with a cleanly formatted medical dashboard.
  • Automatically regenerates a Quick Summary using Gemini every time the record changes.
  • Learned how to integrate multiple Google Cloud services seamlessly in one app.

What we learned

We learned how to integrate LLMs safely with structured data, how to manage stateful UIs in Streamlit, and how important prompt design is when controlling model output formats.
We also gained experience in maintaining data integrity while allowing natural language edits — a key challenge for healthcare-grade AI tools.

What's next for Gemini Clinical Synthesizer

  • Add authentication and per-user access control for realistic hospital use.
  • Expand record fields to include labs, allergies, and vitals.
  • Deploy a shareable web demo so clinicians and students can try it on sample notes.

Built With

  • gemini-2.5-flash
  • google-cloud-firestore
  • google-genai-sdk
  • pandas
  • pil-(pillow)
  • pydantic
  • python
  • streamlit
Share this project:

Updates