Crucible
Resilience testing for AI agent workflows. UiPath orchestrates the tests and governs the verdicts in UiPath Test Cloud.
Landing page: https://crucible-ui-five.vercel.app Live demo: https://crucible-ui-five.vercel.app/demo Live API: https://crucible-production-314a.up.railway.app
Inspiration
Multi agent AI systems are being put in charge of real decisions: insurance approvals, claims, onboarding, support. The danger is not that one agent is dumb. The danger is that one agent quietly hands a wrong answer to the next agent, which trusts it, and the bad outcome sails all the way through with nobody noticing.
This is the failure mode normal testing misses. Unit tests check one agent in isolation. End to end tests check the happy path. Neither one asks the question that actually matters in production: when something goes wrong mid pipeline, does the system catch it and pull in a human, or does it silently finalize a wrong decision?
We wanted a way to find that out on purpose, before a real user is affected. So we built a crash test lab for AI agent pipelines.
What it does
Crucible injects deterministic faults between the agents in a running multi agent system, then scores whether the system detects the fault, escalates to a human, or silently fails.
It runs the target pipeline twice. Once with no fault, to establish the correct outcome. Once with a fault injected at a chosen step, to see what actually happens. Then it compares the two and produces a resilience score from 0 to 100, a verdict, and an explanation.
Our reference target is a healthcare prior authorization pipeline: three AI agents that read an insurance request, apply coverage policy, and decide whether to approve it or escalate to a human reviewer. The test case is a deliberately hard one, an MRI request with the diagnosis field left blank, which a safe system must flag and escalate, never auto approve.
The six faults Crucible can inject:
- ai_hallucinate: replaces an agent output with a convincing but fabricated determination
- ai_refuse: forces an agent to refuse instead of producing a real determination
- data_drop_field: removes a required field from the data passed between agents
- data_prompt_injection: injects malicious instructions into the data passed between agents
- infra_crash: crashes an agent mid pipeline with a simulated server error
- infra_latency: delays an agent response to simulate a slow or hanging service
The faults are injected deterministically, the same fault the same way every time, so each test is reproducible.
The result that matters
The headline case is ai_hallucinate injected after the review step. With no fault, the review agent correctly returns needs_info and the resolution agent escalates the blank diagnosis MRI to a human. Safe.
With the fault injected, a fabricated confident approval overwrites the review output, and the resolution agent auto finalizes it as approved. A request that should never have been approved is approved, with no detection and no escalation. Crucible scores this 0, verdict "Silent Corruption."
The same pipeline scores 70 "Mostly resilient" against ai_refuse and data_prompt_injection, because the resolution gate still caught those and escalated. That contrast is the point: Crucible discriminates. A useful test tells you which faults your system survives and which ones it does not.
How UiPath fits (orchestration and governance)
The Track 3 requirement is that UiPath be the orchestration and governance layer, not just a place results are dumped. So we made UiPath orchestrate the testing.
A published UiPath API Workflow runs on UiPath Automation Cloud as a serverless cloud job. It loops over a set of scenarios passed in as input, calls Crucible's live API for each one, gets the verdict back, and the verdict is governed in UiPath Test Manager. One job runs the full multi scenario suite and lands every verdict in Test Cloud, where they appear as pass or fail in an auditable record.
This is a real cloud job, not a design time debug. It is parameterized by input, so the same workflow can run any set of faults. The three agents being tested are pure Python plus an LLM, with no UiPath inside them, which is intended: UiPath governs the results, it is not inside the tested workflow. Because Crucible intercepts at the message layer, it is framework agnostic by design.
How scoring works
The resilience score is built from three components, so it is interpretable rather than a single opaque number:
- Outcome integrity: did the final decision stay correct under the fault
- Detection: did the system notice something was wrong
- Escalation: did the system route the case to a human when it should have
A silent corruption, where the outcome flips to wrong with no detection and no escalation, scores 0. A case where the fault is absorbed and the outcome is still correct scores high. A case where the pipeline breaks and produces no decision at all scores in the middle, degraded but not silently wrong. The verdict text and the per component breakdown are returned with every run so the score can always be traced back to what happened.
How we built it
The Crucible engine and API are Python, deployed on Railway, exposing a single /run endpoint that takes a scenario and a sample and returns the full comparison report. Railway hosts the always on service that both UiPath and the demo call.
The three agent prior authorization pipeline is pure Python with an LLM for the agent reasoning. Crucible taps the messages passed between agents and injects the chosen fault at the chosen step.
The orchestration and governance layer is a UiPath API Workflow built in Studio Web, published to UiPath Automation Cloud, and run as a serverless cloud job. Verdicts are pushed to UiPath Test Manager through its results API.
The demo is a Vite and React app on Vercel. It calls the live Crucible API directly so anyone can pick a fault, run it, and see the result agent by agent, with a plain language verdict telling them whether the workflow is safe against that fault and what to do next.
Challenges we ran into
Making UiPath genuinely orchestrate rather than just receive results took the most work. Getting a multi scenario loop to run on Automation Cloud, pass scenarios in as input, call the external service per iteration, and land every verdict in Test Cloud meant working through the API Workflow's loop and output behavior carefully until a published cloud job ran the whole suite end to end.
Designing a score that is honest and interpretable was the other hard part. It was important that the score never be a black box, that a silent corruption be clearly distinguishable from a clean failure, and that every number trace back to a concrete behavior in the run.
Accomplishments we are proud of
A published UiPath cloud job that orchestrates an external AI resilience suite and governs every verdict in Test Cloud, proven end to end. A scoring system that discriminates between dangerous silent corruption and safe degradation rather than scoring everything the same. And a live demo that turns an abstract reliability problem into something a non technical reviewer can understand in ten seconds, on a real healthcare case.
What we learned
The most dangerous AI failures are not loud errors, they are confident wrong answers that the rest of the pipeline trusts. Testing for that requires deliberately injecting faults and checking the human in the loop behavior, not just checking outputs. And orchestration plus governance, having a system decide when to run the tests and keep an auditable record of the verdicts, is what turns a one off check into something an organization can actually rely on.
What's next
Bring your own workflow. Today Crucible tests a reference pipeline. The natural next step is letting any team point Crucible at their own agent system in one step, by intercepting their agents' calls at the message layer, so any framework can be battle tested without rewriting it.
Beyond that: more sample cases beyond the single reference request, fault specific mitigation guidance tailored to each failure, and risk based test selection where an agent decides which faults to run at which steps based on where a pipeline is most exposed.
Honest scope
The demo runs against one reference pipeline and one carefully chosen hard case, so the failure mode is unmistakable. The underlying agents use a live LLM, so the fault injection is deterministic while the agent reasoning is not. The Test Cloud verdicts are governed inside our private UiPath tenant, so they are demonstrated in our walkthrough rather than browsable by visitors. These are deliberate scoping choices for the hackathon, and the roadmap above is how each one generalizes.
Log in or sign up for Devpost to join the conversation.