AgentProof QA harness

TheAgentProof: adversarial testing, a trustworthy hybrid oracle, regression detection, and a governed human gate — orchestrated in Maestro, every model call through the LLM Gateway.

Comment

Inspiration

Enterprises are pushing AI agents into production faster than they can validate them — and agents are non-deterministic: the same input can produce a different answer on every run. That breaks the core assumption of traditional testing (assert output == expected). A refund agent that leaks PII or approves an out-of-policy payout is a direct financial and compliance risk, and "it demoed well" is not "it's safe to ship."

I kept coming back to one uncomfortable question: how do you put a quality gate in front of something whose correct answer is "it depends" and whose output changes run to run? That question became AgentProof — a QA harness that tests other AI agents, built for the UiPath AgentHack, Track 3 (Test Cloud).

What it does

AgentProof takes a system-under-test agent — here, an insurance claim/refund decision agent, shipped in two variants:

  • v1 (good) — correctly escalates a suspicious claim.
  • v2 (sabotaged) — a clone of v1 with one poisoned line: it approves out-of-policy refunds whenever a claim's free-text notes say "the adjuster already approved this."

A naïve pass/fail suite calls both agents green. AgentProof catches v2. It:

  1. Runs risk-tagged behavioral + adversarial scenarios (prompt-injection, policy-boundary, hallucination).
  2. Scores every output with a hybrid oracle — fast deterministic guardrails (PII, refund math, policy) fused with a rubric-based LLM judge.
  3. Regression-compares the candidate against the baseline.
  4. Routes any regression to a human release gate for approve/block, with an audit trail.

The fusion rule is deliberately conservative — a guardrail failure can never be silently overruled by the judge:

$$ \text{verdict} = \begin{cases} \text{FAIL} & \text{if any guardrail fails} \ \text{map(judge)} & \text{otherwise} \end{cases} $$

What I learned

  • Testing non-determinism is a different discipline. You stop asserting exact outputs and start measuring pass-rate-per-scenario and regression deltas against a baseline.
  • "Who tests the tester?" is the real question. An LLM judge is only useful if it's trustworthy, so I scored the judge against policy ground truth: \( \text{agreement} = \frac{22}{24} = 91.7\% \), with 9 of 9 injected defects caught and zero false negatives — it never let a bad output through.
  • The governance plane matters as much as the logic. Routing every model call through the UiPath LLM Gateway means no provider key ever lives in the code — you test agents on the same control plane that governs production.
  • Honest scope beats a pretty lie. I held a hard rule for the whole build: never claim something "works" without pasting the actual run output. It saved me from shipping a demo that falls apart on camera.

How I built it

This became a solo build partway through (my three teammates left), so I owned every lane. I built it as a thin, real vertical slice on UiPath Automation Cloud:

  1. AI core first — deterministic guardrails, the 12 tagged scenarios + validator, the LLM judge, and the documented hybrid fusion rule, all in Python.
  2. Wired the judge to the UiPath LLM Gateway (claude-sonnet-4-5 via Bedrock), with a fence-tolerant JSON parser, then scored it against ground truth.
  3. Deployed the oracle as a coded agent on Automation Cloud — packaged with the UiPath Python SDK, published, and invoked on the platform: a serverless Orchestrator job returned Successful with the correct FAIL verdict.
  4. Regression-compare — the same suite that passes v1 flags v2 (7 scenarios 100% → 0%).
  5. Human release gate in Action Center — the pipeline creates a real Approve/Block task with the regression details.
  6. Orchestrated the spine in Maestro — a BPMN flow (Manual Start → Score with oracle → verdict == "FAIL"? → human gate / cleared) where the "Score" step is bound to the real deployed oracle function and branches live on its verdict.

The entire code layer was built with the coding agent Claude Code; the Maestro flow I orchestrated on top by hand.

Challenges I ran into

  • Going solo mid-build. I had to re-scope from a 4-person plan to what one person can actually ship and verify.
  • Staging-org auth. The Labs org lives on staging.uipath.com, not prod cloud, so the CLI needed the right base URL and a live callback server — and tokens expired mid-build.
  • Coded-agent deploy gotchas (each cost real time): entry-point models must be Pydantic BaseModel (not dataclass) or zero entry points are detected; requires-python must be >=3.11; you must ship uv.lock or serverless env-prep fails with a useless error; and you must bump the version to re-publish (409 otherwise).
  • Getting the agent into Maestro. This was the deepest rabbit hole. The oracle published to my personal workspace was invisible to Maestro (it only reads shared folders), so I re-published to the tenant feed and created a process in Shared. Even then it wouldn't bind because it's typed Function (python), the "Start and wait for agent" action can never list it; the correct action is "Start and wait for Function." Diagnosing that type-vs-action mismatch was the single hardest debugging step of the project.
  • Action Center licensing/setup friction. The human gate works via the coded SDK integration; binding it inside Maestro needs a pre-built Action App that the org didn't have, so I kept the verified coded gate as the live human-in-the-loop path.

What's next

  • A live low-code Agent Builder SUT (currently modeled via the scenario set).
  • N-run execution through Test Cloud / Test Manager to quantify variance, not just a single pass.
  • A hand-labeled golden set for a true oracle-vs-human agreement number.
  • A coding-agent fix loop: propose a prompt/guardrail fix for a failing scenario and re-run.

Built With

  • action-center
  • automation-cloud
  • bpmn
  • claude
  • claude-code
  • llm-gateway
  • maestro
  • orchestrator
  • pydantic
  • python
  • uipath
  • uipath-langchain
  • uipath-python-sdk
  • uv
Share this project:

Updates