Inspiration
Enterprise GenAI apps are hard to test with classic QA because outputs are nondeterministic and failures are semantic: prompt injection, hallucination, policy bypass, and data leakage. AegisQA makes adversarial AI testing repeatable and governed inside UiPath Test Cloud.
What it does
AegisQA generates adversarial test cases, executes them through UiPath Studio Web, evaluates AI responses, and syncs findings into UiPath Test Manager.
It tests for prompt injection, jailbreak resistance, system prompt leakage, synthetic PII leakage, policy bypass, hallucination, and tone/safety compliance.
How we built it
We built a Python FastAPI backend with attacker and evaluator agents, a SQLite state store, and a Streamlit Control Center.
UiPath Studio Web triggers the backend through an API Workflow. AegisQA processes the run, scores responses, and creates Test Manager artifacts including Test Cases, Test Sets, Test Case Logs, and failed-result evidence.
Challenges we ran into
The hardest part was integrating external agentic findings into UiPath Test Manager’s execution model. Some third-party synthetic executions remain marked as Running, so AegisQA exposes that honestly as synced_execution_open while preserving the failed-result evidence.
Accomplishments that we're proud of
- Built a working Track 3 solution on UiPath Automation Cloud.
- Triggered the QA loop from Studio Web.
- Synced AegisQA findings into Test Manager.
- Added mock mode, Gemini support, and documented coding-agent assistance.
- Validated the project with 23 passing tests.
What we learned
Agentic QA needs more than prompt generation. It needs orchestration, semantic evaluation, evidence capture, and governance. UiPath works well as the control plane for that loop.
What's next for Project AegisQA
Next we want to connect real enterprise AI apps, expand the evaluator rubric, add richer Test Manager dashboards, and support risk-based test selection.
Log in or sign up for Devpost to join the conversation.