Med-Gemini: Multi-Agent Clinical Operating System

A multi-agent AI workforce powered by Gemini 3 that automates medical triage, scan analysis, and clinical documentation to empower doctors and save lives.

Comment

Inspiration

In many developing nations, the healthcare system is at a breaking point. With doctor-to-patient ratios as low as (1:10,000), physicians are forced to see over 100 patients a day. This leads to a dangerous cycle:

Administrative Burnout: Doctors spend 40% of their time on paperwork.

Cognitive Overload: Human error increases when processing massive amounts of unstructured medical history in seconds. Delayed Intervention: Critical cases wait too long in queues while low-urgency cases are processed. I built Med-Gemini to be an "Intelligent Force Multiplier." I wanted to create a system that acts as a super-powered Chief Resident—handling the heavy lifting of data synthesis, history analysis, and documentation so doctors can focus on the one thing AI can't do: the human connection of healing.

How I Built It

Med-Gemini is a Multi-Agent Clinical Operating System built on a coordinated fleet of specialized AI agents.

The AI Workforce (Powered by Gemini 3)

The Orchestrator (Gemini 3 Pro): The "Chief Resident." It coordinates the sub-agents, manages state handoffs, and synthesizes final reports.

History Agent (Gemini 3 Pro): Leveraging Gemini 3's million-token context window, this agent ingests years of raw EHR data, scanned PDFs, and unstructured notes to create a concise clinical timeline. Scan Agent (Multimodal): An AI radiologist that analyzes DICOM and image files (X-Rays, CTs) to identify abnormalities with visual heatmaps.

Diagnosis Agent (Gemini 3 Pro): Uses clinical reasoning to generate differential diagnoses (DDx) and calculate urgency levels.

Coding Agent (Gemini 3 Flash): Designed for pure speed and accuracy, it generates ICD-10 and CPT codes for billing and records in milliseconds.

The Tech Stack

Framework: Next.js 16 (App Router) for a sleek, "boxy" modern UI. Orchestration: Inngest for event-driven, durable AI workflows. Database: PostgreSQL with Prisma ORM. Storage: AWS S3 for secure medical image handling. Auth: Clerk for HIPAA-ready authentication workflows.

Challenges I Faced

Unstructured Data: Medical notes are notoriously messy. Engineering the History Agent to accurately distinguish between a patient's self-reported "history of" versus a confirmed clinical diagnosis required rigorous prompt refinement.

Agent Handoffs: Ensuring the Diagnosis Agent received the synthesized output from both the Scan and History agents without losing context was a challenge. I solved this by using Inngest to create a linear, step-based workflow where each agent builds on the "Knowledge Graph" of the previous one.

Latency vs. Accuracy: We initially used Pro for everything, but it was slow for simple tasks. By moving the medical coding task to Gemini 3 Flash, we reduced documentation latency significantly without sacrificing billing accuracy.

What I Learned

The most profound realization was the power of Long Context. Traditionally, RAG (Retrieval-Augmented Generation) was needed to handle medical histories. With Gemini 3, I learned that stuffing the entire patient history into the prompt actually leads to better reasoning and fewer "hallucinations" because the model has the full clinical picture simultaneously.

I also learned that Agency is the future of medical UX. Doctors don't want a chatbot; they want a system that does the work—a Clinical OS that triages and documents while they treat.

Impact & Future

Med-Gemini isn't just a prototype; it's a blueprint for equitable healthcare. By reducing administrative time by 70%, we can effectively triple the capacity of a single doctor in a high-load environment.

Next steps involve integrating real-world wearable data for continuous monitoring and expanding the Scan Agent to support 3D MRI reconstruction.

Share this project:

Updates