Project Story → "About the project"
Inspiration
Every engineering team lives with a quiet, expensive lie: the green CI badge.
When an automated test breaks, the failure is ambiguous. It might be a real product bug the test correctly caught — or it might be a fragile test that broke for a dumb reason: a renamed field, a moved selector, a brittle wait. Telling those two apart is manual, tedious, and relentless, and it repeats every single release. So teams do the human thing: under deadline pressure, they "just make CI green." They loosen the assertion. They add the retry. They skip the flaky one. And every time they do, there's a chance they just masked a real defect that the test was doing its job by catching.
We wanted to attack that exact moment of triage — and we wanted an agent we could trust to do it, because an agent that heals tests carelessly is far more dangerous than no agent at all. That single constraint became our north star:
Never let a self-heal mask a real defect. Under ambiguity, escalate — don't patch.
The problem only gets worse as workflows become AI-infused. You can't
assert output == expected against a non-deterministic agent or LLM step, so those steps
tend to go untested entirely. We realized the most interesting thing we could build for
UiPath Test Cloud was an agent that tests other agents.
What it does
Sentinel turns the entire test lifecycle into a team of 16 governed, specialized agents, orchestrated by the UiPath Platform, with a human in the loop at every decision that matters:
- Generates tests from a requirement (Jira / PRD / natural language) into a structured Test Cloud plan — happy path, negative, edge — human-approved before anything is registered.
- Selects the minimal, risk-ordered subset of tests to run for a given code change (AST change-impact + risk scoring) — and never skips a test guarding a security/auth behavior the change touches.
- Self-heals or escalates — the centerpiece. A failed test is classified into a
calibrated verdict:
product_bug | test_fault | flaky | environment. It proposes a minimal patch only fortest_fault, routes it to a human, and escalates real bugs instead of patching them. - Verifies the heal — after an approved patch, it re-runs in Test Cloud: green → close; still red → revert + escalate. The loop is closed; the heal has to actually work.
- Validates AI steps semantically — an LLM-as-judge gate (faithfulness, no-PII, on-task, format) so non-deterministic agent outputs can be tested at all.
- Measures suite quality — flaky detection, requirements traceability/coverage gaps, mutation scoring (does the suite actually catch anything?), API contract checks.
The submission's master command, sentinel mission run, strings the whole governed flow
end-to-end on UiPath: connect → author tests into Test Cloud → risk-select →
heal-vs-escalate → an AI Trust Gate that scans a third-party AI output for injection,
validates it semantically, checks for drift, and deterministically BLOCKs it before the
flow ever trusts it → policy gate → a go/no-go release-readiness report — every step
audited, costed, and resumable.
How we built it
A clean monolith, layered. One installable package, sentinel/, with strictly
one-directional imports (cli/api → orchestration → agents → core ← integrations). No
distributed-systems overhead — easy to reason about, demo, and ship.
- Agents subclass a
BaseAgentand return validated Pydantic models, never free-form text. The orchestrator routes on structured data, deterministically. - No agent calls another agent. They communicate over an in-process event bus
(
RunStarted,HealProposed,HealApproved,BugEscalated,RunCompleted, …); a pipeline composes the stages. - Governance is interception. A policy engine subscribes to gated events and routes them to UiPath Action Center for human approval — auto vs. human is decided by risk × confidence, not vibes.
- Everything is audited (hash-chained, append-only), costed (per-run token/USD budgets with model-tier routing), and resumable (run state is checkpointed after each stage, so an approval wait or a crash resumes exactly where it stopped).
- Provider-agnostic LLM access through a single wrapper — runs live on OpenAI
gpt-4oor Anthropic Claude, auto-selected by which key is set. - UiPath is the control plane — Test Cloud (design/execute/results), Maestro / Orchestrator (the agent spine + jobs), Action Center (HITL gates), UiPath for Coding Agents (powers the repair agent), Agent Builder (the low-code generator), and an external LangChain/CrewAI-style validator (native + external agents, the bonus).
The discipline that makes it trustworthy is deterministic, not model goodwill. The cardinal rule is enforced at each boundary by code: grounding checks, PII-leak detection, drift gates, default-deny on access, an always-merged accessibility floor, and a heal-approval gate where completed-without-action defaults to rejected — so an unattended approval can never silently mask a bug.
Challenges we ran into
- Trust over cleverness. The hard part of a self-healing agent isn't writing the patch — it's refusing to. We tuned every ambiguous case toward escalation (precision over recall on healing) and backed the verdict with empirical flaky-detection so "flaky" isn't a convenient excuse to mask a failure.
- Testing the non-deterministic. You can't assert equality against an agent, so we built semantic gates with calibrated-confidence prompting + structured outputs + audit replay, and a deterministic allow/BLOCK decision on top so the gate itself is reproducible even when the thing it judges isn't.
- Migrating a verified MVP into the monolith without breaking it. We ported the flat MVP modules into the layered package behind preserved behavior and a green test suite, removing the old code only after the port passed.
- Real platform, real edges. We proved live UiPath connectivity (OAuth +
Orchestrator/Test Cloud reads on staging — 4/5 surfaces reachable), and discovered the
hackathon tenant simply doesn't have the Action Center
actionsservice provisioned (a clean 404). So we built graceful degradation throughout: every integration is best-effort, the offline path is fully functional, and nothing fabricates a result it couldn't actually get.
What we learned
- An agent you can't trust is worse than no agent. The most valuable thing we built wasn't the healing — it was the governed refusal to heal.
- Determinism around non-determinism. Wrap probabilistic reasoning in deterministic gates, structured schemas, and an immutable audit trail, and you get a system you can actually reason about and put in front of a release.
- Structured-by-default scales. Forcing every agent to return validated models made a 16-agent system composable instead of a folder of scripts duct-taped to a parser.
What's next
The full feature catalogue (F-001…F-036, four phases) is implemented and verified offline — 392 tests green. The roadmap continues with persistent cross-run memory that makes verdicts smarter over time, AST/call-graph change-impact, visual self-healing locators, exploratory test discovery, a prompt-injection red-team agent, and a packaged UiPath marketplace solution. The one remaining milestone is full platform-shipping on a UiPath tenant with Test Manager + Action Center enabled end-to-end.
Log in or sign up for Devpost to join the conversation.