Agent Doctors
overview
Full Idea

Inspiration

Most medical AI today is a chatbot. It has no ward, no handoffs, and no senior physician watching. Real medicine doesn't work that way. A patient is triaged by a nurse, examined by a doctor, and checked by a pharmacist, lab scientist and radiologist. The attending physician then signs off and takes responsibility.

I wanted to build that whole loop, not one smart model. Each role would be a specialised agent. Each claim would be grounded in a textbook. A human doctor would stay in charge, and that doctor's judgment would teach the system.

The slogan became: an autonomous virtual hospital that thinks, deliberates, and learns from its physicians.

What it does

Intake and triage. A synthetic patient arrives and gets an ESI acuity level (1–5). A MedGemma router sends the case to one of 7 departments (Internal Medicine, Cardiology, Neurology, Surgery, Emergency, Obstetrics, Gynecology).
Multi-agent deliberation. Doctor, Nurse, Pharmacist, Lab, Radiology and Patient agents move through a 4-stage encounter state machine:

INTAKE_TRIAGE → EXAMINATION_WORKUP → INTERVENTION_TREATMENT → DISPOSITION_REVIEW

Specialists change from Standby to Ready when a handoff needs them. The pharmacist checks drug–allergy conflicts, contraindications and renal dosing.

RAG grounding. About 47,800 chunks from Harrison's, Braunwald's, Williams, Schwartz's, Adams & Victor's and Tintinalli's give every claim a citation (book, chapter, page).
Clinical report. The agents' discussion becomes a structured assessment with evidence snippets.
Physician sign-off. An attending reviews, grades and signs in one click. Each rejection becomes a preference triplet $(x, y_w, y_l)$.
On-device learning. Those triplets train a LoRA adapter on MedGemma 4B, on a laptop RTX 4060.

How I built it

Model: MedGemma 4B, 4-bit NF4 quantised (bfloat16 compute), about 4.15 GB of 8 GB VRAM, plus a DPO LoRA adapter.
Backend: FastAPI, Pydantic v2, WebSocket event bus, SQLite (7 tables).
Knowledge: Textbook PDFs parsed into chapters and chunks, 768-d embeddings, cosine top-$k$ retrieval.
Learning: Reward model, preference-pair generator, policy evaluator, TRL DPOTrainer + PEFT.
Frontend: React 19, TypeScript, Vite, Tailwind 4, animated SVG particle "deliberation" view.
Quality: 67 automated pytest tests.

Retrieval. For a query embedding $q$ and chunk embedding $d_i$, the system ranks by cosine similarity:

$$\text{sim}(q, d_i) = \frac{q \cdot d_i}{\lVert q \rVert , \lVert d_i \rVert}$$

It returns the top $k$ chunks as citations.

Clinical reward. Physician reviews are scored on a scale from $-5$ to $+10$. Diagnostic accuracy adds $+3$ and guideline adherence adds $+2$. A missed contraindication costs $-4$ and a dangerous intervention costs $-5$. Rejected cases become rejected answers and the physician's correction becomes the chosen answer.

Alignment. I used Direct Preference Optimization, so no separate reward network is needed at training time:

$$\mathcal{L}{\text{DPO}}(\pi\theta;\pi_{\text{ref}})=-\mathbb{E}{(x,y_w,y_l)\sim\mathcal{D}}\left[\log\sigma!\left(\beta\log\frac{\pi\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)}-\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\right)\right]$$

On the project's benchmark vignettes, the DPO-aligned policy scored +107.6% higher in clinical score than the base model. That figure comes from a small synthetic benchmark, not a clinical trial.

I kept one rule throughout: each layer has one job.

RAG supplies the medical knowledge.
MedGemma generates.
Verification checks.
Physician feedback teaches.
Fine-tuning improves format and reasoning style. It is not for memorising books.

What I learned

Separation of concerns is a safety feature. If the model, the knowledge and the checking each have one job, a failure is easy to trace.
Preference data beats instruction data for clinical judgment. A doctor saying "this answer is worse than that one, and here is why" is a much richer signal than another prompt–answer pair.
Quantisation makes this practical. 4-bit NF4 plus LoRA let me train and serve a 4B medical model on an 8 GB laptop GPU. No cloud bill and no patient data leaves the machine.
A demo is a product too. Judges and mobile users can't install a GPU stack. Graceful offline fallbacks mattered as much as the model.
Honesty about scope. The audit showed that a planned external-API layer (RxNorm, openFDA, PubMed) did not exist yet, so I documented it instead of claiming it.

Challenges

8 GB of VRAM. Fitting MedGemma, the LoRA adapter and inference together meant 4-bit quantisation and careful lazy loading at server start. A deterministic heuristic engine takes over when CUDA is unavailable.
Ingesting about 600 MB of textbook PDFs. Parsing noisy layouts into clean chapter, heading and page metadata was slow. It was also the only way to produce citations a doctor could check.
Making agents deliberate, not chat. A state machine, an event bus and WebSocket streaming turn agent turns into something you can watch live in the UI.
Static hosting broke the live demo. On GitHub Pages, absolute paths like /case-intake.html escaped the repo subpath and returned 404. I fixed it with relative ./ paths and base: './' in Vite. With no Python backend there, I added offline fallback datasets so every screen still works.
Mentor feedback. The repo needed a visible open-source license and a clickable hosted demo. I added Apache 2.0, set the repository homepage, and added a submission-links table.
Showing the video on GitHub. GitHub strips tags and blocks unmuted autoplay in READMEs. I used an optimised looping GIF that links to the full HD video.
Safety. The data is synthetic only, with no real PHI. There are no hardcoded secrets, and a physician always makes the final decision.

What's next

Add the real external knowledge connectors, add a verification layer that fails closed on any claim without evidence_ids, and move from the 2D patient-journey view to a 3D hospital. The system is a research prototype, not a medical device. Sign-off stays with human doctors.