Diagnostic Follow-Up Guardian

Gemini-powered clinical agent fleets autonomously detect missed diagnostic follow-ups, gather evidence, recommend next steps, and coordinate care—with clinicians retaining final approval.

Comment

Inspiration

A diagnostic finding is only valuable when it leads to appropriate follow-up care. In real clinical workflows, important recommendations can become buried in reports, delayed by fragmented data, or lost during handoffs between radiology, primary care, specialists, and scheduling teams.

I built Diagnostic Follow-Up Guardian to explore a practical question:

What if autonomous AI agents could continuously watch for unresolved diagnostic findings, assemble the relevant evidence, recommend the next action, and coordinate follow-up—without taking final authority away from clinicians?

The goal is not to replace clinical judgment. Guardian reduces the administrative and cognitive burden surrounding follow-up while keeping a qualified human in control of consequential decisions.

What it does

Diagnostic Follow-Up Guardian is an event-driven, multi-agent clinical workflow deployed on Google Cloud. It monitors diagnostic results and coordinates the work required to move a patient from a newly identified finding to an approved follow-up plan.

Guardian can:

  • Detect diagnostic findings that may require follow-up.
  • Prioritize cases by urgency, finding, and workflow status.
  • Retrieve diagnostic imaging and longitudinal clinical context.
  • Use Gemini to synthesize evidence and generate a grounded recommendation.
  • Link recommendations to their supporting clinical evidence.
  • Create a draft follow-up plan for clinician review.
  • Coordinate approved actions through asynchronous workflows.
  • Track scheduled, overdue, rejected, and completed cases.
  • Preserve human approval for consequential clinical decisions.
  • Maintain audit records and distributed traces for governance.
  • Surface fleet health, workflow activity, and provider throughput in an operational workspace.

The clinician remains the final decision-maker. Recommendations can be approved, rejected, or revised, and the workflow records who made the decision and why.

Agent architecture

Guardian uses specialized agents rather than relying on one general-purpose assistant:

  • Intake Agent: Receives finalized diagnostic-result events and creates or updates cases.
  • Context Agent: Retrieves relevant longitudinal patient information from FHIR.
  • Evidence Agent: connects diagnostic findings with source imaging and supporting evidence.
  • Recommendation Agent: Uses Gemini to produce a structured, evidence-grounded follow-up recommendation.
  • Coordinator Agent: Advances approved cases through the follow-up workflow.
  • Monitor Agent: Detects stalled, overdue, or incomplete cases.
  • Governance Agent: Maintains approval history, audit evidence, and workflow traceability.
  • Orchestrator Agent: Coordinates agent execution, state transitions, retries, and handoffs.

For the operational experience, these agents are organized into coordinated fleets covering population surveillance, clinical reasoning, and follow-up coordination. This allows the system to process individual cases while also monitoring the health of the broader patient cohort.

How I built it

The solution combines Gemini with managed Google Cloud services:

  1. A finalized diagnostic result produces an event.
  2. Pub/Sub delivers the event to the Guardian workflow.
  3. The orchestrator invokes the appropriate specialized agents.
  4. Agents retrieve clinical context from the Cloud Healthcare API.
  5. Source DICOM imaging remains linked to the case.
  6. Gemini synthesizes the available context into a structured recommendation.
  7. Case state, recommendation versions, and approval records are persisted in Firestore.
  8. The clinician reviews the recommendation in the Guardian web application.
  9. An approval or rejection event advances the workflow.
  10. Cloud Trace and structured logs provide end-to-end observability.

The backend is packaged in containers and deployed to Cloud Run. The clinical workspace provides cohort prioritization, case review, source-evidence access, approval controls, fleet monitoring, and governance views.

Data sources

Guardian demonstrates multimodal clinical follow-up using:

  • LIDC-IDRI: De-identified thoracic CT imaging.
  • Synthea: Synthetic longitudinal patient and encounter data.
  • DICOM: Source diagnostic imaging.
  • FHIR: Patient, condition, observation, encounter, and related clinical context.
  • Synthetic workflow events: Used to demonstrate autonomous case intake, agent coordination, approval, and monitoring safely.

Twenty LIDC-IDRI imaging studies were paired with twenty synthetic FHIR patient records. This created a reproducible demonstration cohort without exposing real patient information.

Safety and governance

Healthcare automation requires stronger safeguards than a conventional AI assistant. Guardian therefore separates autonomous workflow coordination from clinical authority.

The agents may collect context, evaluate workflow state, draft recommendations, and identify overdue work. However, a clinician must approve consequential follow-up actions. The system also preserves recommendation versions, evidence references, reviewer decisions, timestamps, agent activity, and case-state transitions.

Each agent uses a dedicated Google Cloud service identity with least-privilege access. Permissions are scoped according to the agent’s responsibility—for example, an evidence agent may view imaging while a coordination agent can update workflow state.

This design supports:

  • Human-in-the-loop approval.
  • Least-privilege agent identities.
  • Evidence-linked recommendations.
  • Persistent workflow state.
  • Auditable agent handoffs.
  • Versioned recommendations.
  • Failure handling and dead-letter events.
  • End-to-end logs and traces.
  • Separation of clinical evidence from generated guidance.

Challenges I faced

Pairing multimodal healthcare data

LIDC-IDRI provides de-identified imaging, while Synthea provides synthetic clinical histories. These sources do not describe the same real people, so I had to create an explicit, reproducible demonstration pairing process. I built scripts to select imaging series, extract eligible synthetic patients, create paired cases, import the data, and verify that every case retained unique imaging and FHIR references.

Making the system genuinely agentic

A button that calls a model is not an autonomous agent workflow. Guardian needed to respond to events, delegate work among specialized agents, persist state, survive asynchronous handoffs, and continue until human approval was required. Pub/Sub, Firestore, Cloud Run, and explicit case-state transitions were essential to making that behavior visible and testable.

Maintaining clinical safety

The system needed to automate the surrounding work without implying that Gemini independently practices medicine. I addressed this through evidence grounding, structured outputs, constrained agent responsibilities, recommendation versioning, and mandatory clinician approval.

Managing cloud permissions

Cloud Healthcare API, DICOM, FHIR, Pub/Sub, Firestore, Vertex AI, and Cloud Run each require different access patterns. Establishing separate identities for eight services and applying least-privilege permissions was more complex than using one broad service account, but it produced a safer enterprise architecture.

Closing the workflow loop

Generating a recommendation was only the beginning. The application also needed confirmation messages, rejection reasons, durable approval records, scheduling states, overdue monitoring, and traceable transitions. Designing those downstream states was critical to making Guardian a workflow rather than a chatbot.

What I learned

The most important lesson was that enterprise agentic AI is primarily a systems-design challenge. Model quality matters, but trustworthy autonomy also depends on identity, state, events, tool boundaries, observability, retry behavior, and clearly defined human checkpoints.

I also learned that specialized agents are most useful when each one has a narrow responsibility and limited access. This improves explainability and makes failures easier to isolate.

Finally, multimodal healthcare demonstrations require rigorous validation. It was not enough to upload images and patient records. I verified that all twenty cases had unique studies, series, previews, linked FHIR context, and usable workflow records before treating the cohort as complete.

Accomplishments

  • Built an event-driven healthcare agent system with eight specialized agents.
  • Deployed the orchestrator as a containerized Cloud Run service.
  • Integrated Gemini through Vertex AI.
  • Connected DICOM and FHIR data through the Cloud Healthcare API.
  • Created and validated a twenty-case multimodal demonstration cohort.
  • Implemented persistent case state and approval records in Firestore.
  • Added Pub/Sub topics for workflow, approval, result, and dead-letter events.
  • Applied dedicated least-privilege identities across the agent architecture.
  • Built clinician, fleet-monitoring, and governance experiences.
  • Preserved human review for consequential clinical actions.
  • Added structured logging and distributed tracing for agent workflows.

What’s next

The next phase is to expand Guardian from a validated demonstration into a configurable enterprise follow-up platform. Planned improvements include additional diagnostic domains, organization-specific follow-up policies, richer scheduling integrations, expanded Model Armor protections, longer-term agent memory, enhanced operational analytics, and prospective evaluation with clinical and compliance stakeholders.

Diagnostic Follow-Up Guardian demonstrates how autonomous agent fleets can reduce missed follow-up risk while preserving the evidence, accountability, and human judgment that healthcare demands.

Built With

Share this project:

Updates