Aegis — CI/CD for Regulated AI

Aegis — CI/CD for Regulated AI. Generate. Validate. Govern. UiPath AgentHack 2026.
The problem: AI fabricates a penicillin allergy. Source note says NKDA. One hallucination can kill a patient.
UiPath Architecture: Summarizer + SENTINEL Validator in Agent Builder. Orchestrated by Maestro. HITL via Action Center.
7 test scenarios. 54 flags detected. Zero false positives. Every flag traced to exact source evidence.

Inspiration

Every week, hospitals deploy new AI tools to summarize patient records at scale. The efficiency gains are real. So are the risks. Large language models hallucinate — they fabricate allergies that don't exist, invent diagnoses, omit critical lab values. In most domains, that's a bug. In clinical settings, it's a patient safety event.

The question that started Aegis: if AI is reading medical records, who is auditing the AI? CI/CD pipelines exist to catch bugs before they reach production. Regulated AI needs the same infrastructure — not as an afterthought, but as a first-class architectural requirement.

What it does

Aegis is a CI/CD framework for regulated AI. It intercepts AI-generated medical summaries before they reach clinicians and runs them through a 4-layer validation pipeline called SENTINEL:

L1 — Claim Extraction: decomposes the summary into atomic verifiable claims with IDs and types.

L2 — Hallucination Detection: verifies each claim has supporting evidence in the source note.

L3 — Contradiction Detection: finds source text that directly opposes summary claims. Runs in parallel with L2.

L4 — Critical Completeness: independent scan for clinically critical items — K⁺, SpO2, ECG changes, disposition — that must appear in any valid summary.

When SENTINEL returns FAIL, a Correction Agent proposes a surgical fix. If severity is CRITICAL or HIGH, the case escalates to UiPath Action Center for physician review. Corrections are proposals — never final. Human approval is required before any clinical record is modified.

Seven test scenarios executed. 54 flags detected. Zero false positives. Five auto-corrected. Two escalated to human review.

How we built it

Built on UiPath Automation Cloud with two deployed agents in Agent Builder: Solution 7 (MedicalRecordsSummarizer) and Solution 6 (SENTINEL Validator). The full pipeline is orchestrated by UiPath Maestro — Solution 8 — with a HITL gateway routing critical cases to Action Center.

The validation engine runs at temperature=0 (ADR-007) with a 4-layer sequential/parallel architecture. Escalation logic lives in a single documented decision: escalateToHuman = hasCritical || hasHigh (ADR-008). Every architectural decision is logged in docs/DECISION_LOG.md.

A Scenario Generator (src/core/scenario-generator.js) creates new test cases from natural language clinical requirements — demonstrating test design as an agentic capability.

Built with Claude Code as the primary coding agent. Claude Code scaffolded the 4-layer pipeline, generated clinical test scenarios, and built the dashboard from scratch. Full evidence documented in docs/claude-code-evidence.md.

Challenges we ran into

The propose-and-escalate tension: Early architecture made the Correction Agent skip when escalate_to_human was true. That's wrong — a system that only escalates without proposing a fix leaves the physician with no starting point. The correct behavior: always propose the correction, always escalate when severity is CRITICAL or HIGH. The physician approves, rejects, or modifies the proposal.

LLM variance at temperature=0: Even with temperature=0, different runs of the same scenario produce slightly different flag counts. ADR-007 documents this explicitly — temperature=0 reduces but does not eliminate variance. The dashboard shows real exported results, not idealized outputs.

Maestro BPMN publish blocked: Solution 8 is fully modeled and validated with zero issues in Studio Web. Publish is blocked by a known UiPath staging bug (June 2026, documented in the community forum). Individual agents are deployed and active. This is documented honestly in all project materials.

JSON truncation on complex cases: TC-005 with 9-10 flags caused output truncation at 1024 tokens. Resolved by increasing max_tokens to 2048 and adding explicit JSON completion instructions to the system prompt.

Accomplishments that we're proud of

7/7 correct verdicts across all test scenarios. Zero false positives on TC-001 (the happy path clean summary). The system correctly identified an internal contradiction within a single source note (TC-006) — not a fabrication vs. source mismatch, but two sections of the same record conflicting with each other. That's the most sophisticated detection in the suite.

The Scenario Generator produced TC-007 — a pediatric asthma discharge hallucination — entirely from natural language requirements. TC-007 was then run E2E in UiPath Agent Builder and returned PASS/NONE on a correct summary, demonstrating zero false positives in production.

Every result is traceable. Every flag has exact source evidence. Every export is a structured JSON audit record. No vaporware.

What we learned

Regulated AI doesn't need more capable models — it needs better governance infrastructure. The failure mode in TC-002 (fabricated penicillin allergy) isn't a model limitation. It's an architectural gap: there was no layer between the AI output and the clinician. Aegis closes that gap.

The "propose + escalate" pattern is the right abstraction for human-in-the-loop AI in high-stakes domains. Auto-correction alone is insufficient. Pure escalation without a proposed fix wastes physician time. The system that works proposes a correction and requires human sign-off — every time, for every critical case.

What's next for Aegis

Expand beyond medical summaries to any high-stakes AI output domain — legal documents, financial reports, regulatory filings. The 4-layer architecture is domain-agnostic; only the L4 critical completeness rules change per vertical.

Integration with HL7 FHIR R4 export formats for direct EHR compatibility. Audit Logger with real-time session metrics and trend analysis. Multi-tenant deployment for hospital networks with per-department escalation routing.

The addressable market is any organization deploying AI in a regulated environment. That's not a niche — it's the next five years of enterprise AI adoption.