TestPilot On-Call QA for AI Agents

• 

  TestPilot — On-Call QA for AI Agents

💡 Inspiration

Everyone is shipping AI agents. Almost nobody has a governed way to catch when one silently regresses — when its behavior drifts, not its code. An agent doesn't crash when it gets worse; it just starts being confidently wrong in production while your tests stay green. Self-healing a renamed selector is a solved commodity. Governing agent quality is not. We built TestPilot to be the on-call QA engineer that watches the failure mode nobody else does.

🚀 What it does

TestPilot treats a failed UiPath Agent-Evaluation run as the thing to triage. It classifies each failing case by which evaluator class failed, then applies a policy-correct action:

• Mechanical drift (deterministic exact / JSON-similarity fails — a renamed selector or schema field) → auto-heals a one-line fix, re-runs it green, and opens a reviewable GitHub PR.
• Flaky (a behavioral evaluator passes on retry) → quarantined with a retry policy; no code change.
• Behavioral regression (LLM-as-judge faithfulness / trajectory fails consistently — the agent's behavior actually changed) → never auto-fixed; escalated to a human via Action Center + Slack with an AI root-cause.

The load-bearing policy, enforced in code: a behavioral change in an agent is a product decision, not a bug fix — only a human merges behavior. Every decision is written to a hash-chained, tamper-evident governance ledger.

🛠️ How we built it

• Two UiPath agents — a low-code Triage agent and a Python Fixer coded agent on serverless — orchestrated by a Maestro BPMN process with an exclusive gateway on the build verdict (severity priority).
• The agent brains are pure, unit-tested Python (pydantic v2), authored test-first with Claude Code via UiPath for Coding Agents (uip skills install --agent claude).
• The fixer drafts its one-line diff through the AI Trust Layer LLM Gateway (keyless, governed).
• Integration Service opens the GitHub PR and posts to Slack; Action Center holds the human gate; Test Cloud / Test Manager runs the red→green re-run.
• A framework-free Mission Control dashboard (vanilla JS + SVG) visualizes the fleet, the Triage Firewall, and the ledger — verified with a runnable Playwright smoke test.

🧗 Challenges we ran into

• The hard part is the boundary between flaky and behavioral — we route by evaluator class and make the fixer refuse anything non-mechanical (two independent checks).
• Making "governance" tangible, not a slide: we built a client-side hash chain so you can literally tamper with a ledger entry and watch the chain break, then heal.
• Keeping the build-time coding-agent (the bonus) cleanly separate from the runtime LLM (a design choice), and documenting both.

🏆 Accomplishments that we're proud of

• Strict TDD — 83 tests, green in under a second.
• A contract test that guards the Maestro↔agent field mapping against silent drift.
• Both coded agents published to Automation Cloud; the whole flow also runs offline for a reproducible demo.
• A distinctive UI that makes the thesis legible in 30 seconds.

📚 What we learned

The most dangerous agent failure is the one that looks green. Treating agents as software-under-test — with evaluators as the test substrate — changes how you ship them. And governance only convinces when it's enforced in code and provable (a refusal policy + a tamper-evident audit trail), not narrated.

🔭 What's next

• Wire the dashboard to live tenant telemetry (a single fetch swap).
• Trust-trend alerting that flags drift before a regression ships.
• Per-agent policy packs and broader evaluator coverage.

Built With

• ai-trust-layer
• claude
• claude-code
• css
• github
• html
• javascript
• llm-gateway
• playwright
• pydantic
• pytest
• python
• slack
• uipath-action-center
• uipath-agent-evaluations
• uipath-coded-agents
• uipath-for-coding-agents
• uipath-integration-service
• uipath-maestro
• uipath-test-cloud