The Gate for Clinical Agents

Inspiration

A healthcare agent that can read patient records, message patients and order refills is the highest-stakes prompt-injection target there is. Clinical paperwork (referral letters, lab reports, insurance forms) is exactly how indirect injection arrives: untrusted text an agent reads and then acts on.

What it does

The prompt-injection gatekeeper pattern sits at the tool boundary: every tool call the agent proposes (read a record, message a patient, order a refill) passes seven typed yes/no questions first. Thresholds turn the answers into PROCEED (execute and audit), CONFIRM (hold for a clinician) or DENY (block and record a security event). Anything risky or ambiguous holds for a clinician instead of executing.

How we built it

Tools are wrapped in a GatedTool that is the only route from the LLM's tool choice to a real function, and the gate fails CLOSED to CONFIRM on any decider failure. A red-team suite of nine attack classes (obvious injection, indirect injection through documents, malicious tool arguments, unauthorized and premature actions, and more) runs live against the decider and writes evaluation_results.json with detection rate, false positives and negatives, and latency: about 115 ms per typed question, and at 0.9+ confidence it is right roughly 95% of the time, so low confidence degrades to a clinician confirmation instead of a guess.

The clinical build

The clinical-agent framing (record reads, patient messaging, refill orders), the clinician-confirmation flow and the demo showing an indirect injection arriving through a patient document are the build of this hackathon; the gate itself and its red-team suite are shipped and tested in the repo.

Challenges

Keeping every figure honest: all numbers come from the repo's README and evaluation results, nothing invented.

Built With

• python