Agentic Test Swarm

Inspiration

Enterprise teams are starting to let AI agents call APIs, automate workflows, and talk to other agents. That creates a new testing problem: before an AI worker touches production data, how do we prove what it is allowed to do, what it must refuse, and when a human must step in?

We built Agentic Test Swarm around that question. Instead of treating testing as a final checklist, we made testing itself agentic: agents attack the workflow, design tests, select high-risk coverage, analyze failures, recommend repair, re-test, and only then produce a restricted runtime permit.

What it does

Agentic Test Swarm is a UiPath-orchestrated testing swarm for AI-infused enterprise workflows.

In our demo, a Marketing Outreach Agent asks a Customer Data Agent to export raw VIP customer emails. The request contains an executive-override style prompt injection. The swarm turns that risk into certification tests, catches the raw PII export failure, analyzes the failure, proposes a guardrail repair, runs targeted re-tests, and produces a restricted runtime permit.

The final runtime behavior is:

Aggregate marketing insights are allowed.
Raw customer email export is blocked.
High-risk PII access requires human approval.
Unauthorized agent-to-agent PII access returns deny_and_suspend.

The permit is not the product. The product is the UiPath-governed testing loop that creates evidence before an AI workflow is trusted.

How we built it

UiPath is the orchestration and governance layer.

We used:

UiPath Test Cloud / Test Manager to store certification test cases, requirement coverage, and execution evidence.
UiPath Action Center to keep a human accountable for restricted permit approval.
UiPath API Workflow as the runtime proxy for agent-to-agent tool calls.
UiPath Studio Web / Maestro-style workflow modeling for the certification lifecycle.
UiPath-native Agent Builder / Studio Web agent specs for Policy Miner, Red-Team Agent, Test Designer Agent, and Failure Analyst Agent.
Coding-agent evidence for adversarial test generation, repair candidate design, and API Workflow payload scaffolding.
A deterministic FastAPI worker deployed on Vercel for repeatable demo execution and evidence APIs.

The key design choice is split trust:

AI agents propose attacks and repairs.
Deterministic oracles define expected outcomes.
UiPath records evidence, gates approval, and enforces runtime decisions.

Challenges we ran into

The biggest challenge was avoiding a Python-only demo. It was easy to build a local AI testing loop, but the hackathon asks for UiPath to be the execution and governance layer.

We solved that by making UiPath surfaces visible and meaningful:

Test Manager has a live requirement PVACPOV91:151 linked to six certification tests.
Test Cloud evidence shows the passing certification set.
Action Center holds the human approval task.
API Workflow enforces the runtime decision.
The Vercel worker acts as a called worker, not the system of record.

Another challenge was showing AI testing clearly without overclaiming. We separated before-repair failure evidence from after-repair Test Manager evidence, so judges can see what is live in UiPath and what is deterministic replay evidence.

Accomplishments that we're proud of

Built a public live demo: https://permitops-uipath.vercel.app/
Created a Test Manager project with six certification tests.
Created live requirement PVACPOV91:151 and linked it to all six tests.
Added incident-memory coverage with TC-006 for a new phone-number export risk.
Preserved an Action Center approval task for the final human gate.
Implemented API Workflow-style runtime enforcement returning deny_and_suspend.
Added a Test Cloud traceability pack: Requirement -> Test Case -> Execution -> Defect -> Runtime Permit.
Added before/after repair evidence so the failure-analysis story is auditable.
Built an MIT-licensed public GitHub repository with tests, docs, deck, and evidence files.
Verified the local test suite with 80 passed.

What we learned

The hard part is not generating a test case. The hard part is making AI-generated testing evidence governable.

External LLMs and coding agents are powerful for generating attacks and repairs, but enterprises need a control plane that records evidence, enforces approval, and makes runtime decisions auditable. UiPath fits that role well: Test Cloud stores evidence, Action Center keeps human accountability, and API Workflow turns the result into an enforceable production control.

What's next for Agentic Test Swarm

Next, we would make the remaining Test Cloud-native loop fully live:

Attach captured AI traces and hash evidence directly to Test Manager results.
Configure a real Test Manager Create Defect webhook for failed AI-workflow tests.
Turn Obsolete Test Scout into a live stale-test classifier.
Use Testing Process Governance to sign off certification tests.
Expand from customer-data access to invoice, refund, procurement, and support AI workflows.

The long-term vision is simple: every enterprise AI worker should be attacked, repaired, re-tested, approved, and monitored before it gets production privileges.