NovaFabric Flaky Detective

An agentic testing assistant that flags flaky and regression-prone AI-agent test runs, then replays and diffs captured runs to pinpoint the root cause. Built on UiPath, powered by NovaFabric.

Comment

Inspiration

AI agents are non-deterministic: give an LLM the same input twice and you can get two different runs. That makes testing agentic automations painful — when a test fails, you can't tell whether the code actually broke (a real regression) or the agent just behaved differently this time (flakiness). Teams end up ignoring failing tests and miss real bugs. We wanted an agent that answers one question well: "Is this failure real, or just flaky?"

What it does

NovaFabric Flaky Detective is an agentic testing assistant on the UiPath platform. When an AI-agent test run looks inconsistent, it:

  1. Replays a previously captured baseline run,
  2. Diffs the baseline against the current run (model calls, tool calls, outputs),
  3. Classifies the difference as a real regression vs non-deterministic flakiness, and
  4. Names the exact differing step and suggests remediation.

The key idea: UiPath's built-in agent evaluations score ONE run against an expectation. Flaky Detective compares TWO real runs to separate flakiness from a genuine regression — so they're complementary. Evals gate a single run; Flaky Detective triages why two runs diverged.

How we built it

  • UiPath Agent Builder (low-code) is the reasoning, orchestration and governance layer. The agent takes the tester's question in plain English and explains the verdict.
  • The agent calls one tool, analyzeRuns, a custom UiPath Integration Service connector built from our API's OpenAPI spec.
  • A FastAPI service exposes POST /analyze, which uses NovaFabric to capture and replay two agent runs as structured "capsules" and diff them. The verdict itself is deterministic; the agent narrates it and escalates real regressions to a human.
  • Deployed as a Docker image on Render; the connector and agent run in UiPath Automation Cloud.

Challenges we ran into

  • Reading NovaFabric's real diff shape: the signal is nested under sections with per-pair flags (request_changed / response_changed / result_changed). We rewrote the classifier against an actual compare() run instead of guessing the keys.
  • A live agent-under-test produced empty traces that conflated flaky with regression, so we generate synthetic baseline/flaky/regression capsules deterministically — no API keys needed.
  • Agent Builder has no raw "HTTP endpoint" tool, so we built a custom Integration Service connector from the OpenAPI spec.
  • Render free-tier cold starts — we warm /health before each run.

What we learned

The useful signal in agent testing isn't "did it pass" — it's where two real runs diverged. A deterministic classifier plus an LLM narrator keeps the verdict trustworthy while making it human-readable and actionable. Keeping the scope tight — one job done reliably — made the whole pipeline (UiPath → connector → FastAPI → NovaFabric → verdict) provable end to end.

What's next

  • Plug verdicts into UiPath Test Manager (regression = FAILED, flaky = WARN).
  • A live capture mode that runs the test fresh and diffs it against a stored baseline, for CI.
  • Confidence calibration and a richer per-step "differing step" drill-down.

https://novafabric.ai/ https://github.com/MSKazemi/novafabric_flaky_detective https://pypi.org/project/novafabric/ https://flaky-detective.onrender.com/fixtures https://hub.docker.com/r/kazemi/flaky-detective https://cloud.uipath.com/novafabricagenthack/studio_/designer/c4d004df-1422-4ed6-87c5-b83d33951159?solutionId=2414bd3b-8bab-48b8-e47a-08ded1e3ee41 https://flaky-detective.onrender.com/docs

Built With

  • fastapi
  • novafabric
  • python
  • uipath
  • uipath-agent-builder
  • uipath-automation-cloud
Share this project:

Updates