Inspiration
Most code gets merged without tests because writing them takes time nobody has. And when tests do run, results end up scattered across CI logs instead of somewhere governed and traceable.
During my coding projects, I keep running into the same problem where writing tests always felt like something we'd "do later." When deadlines got tight, tests were skipped, and even when they were run, the results ended up buried in CI logs where nobody looked at them again. I wanted testing to become part of an actual feedback loop instead of just another checkbox. My idea was to build a system that could automatically generate tests, verify them, execute them in UiPath Test Cloud, and then use the results to help determine whether code is actually ready to merge. Rather than simply collecting test results, I wanted to make them actionable—turning automated testing into something developers can trust when making decisions.
What it does
When a developer opens a GitHub pull request, Crucible automatically analyzes the code to understand what the change is intended to accomplish. It then generates a suite of tests and sends them to a second AI agent that acts as a strict reviewer. Instead of giving vague suggestions, the reviewer identifies concrete issues and sends the tests back for revision. This feedback loop repeats for up to five iterations until the tests meet production-quality standards. Once the tests are approved, Crucible performs static security analysis on both the pull request code and the generated tests. It then executes the tests, uploads the results to UiPath Test Cloud, and retrieves the pass rate. Finally, Crucible combines all of this information into a single confidence score: 30% based on how many review iterations were required, 30% from the security scan, and 40% from the actual Test Cloud results. The score is posted directly on the pull request along with a merge recommendation. A score above 0.85 indicates the change is ready to merge. A score below 0.65, or the presence of any critical security finding, blocks the merge and automatically opens a Maestro case for human review. To improve traceability, every generated test is linked back to the requirement it validates. Crucible also detects tests that rely on timing, system clocks, or random values, flagging them before they can introduce flaky behavior into the CI pipeline.
How we built it
The system starts with a FastAPI Webhook server, which receives pull request events and immediately places them into a background queue so it never blocks while waiting for AI responses. Three Claude agents then process the request in sequence. The first agent uses a fast, low-cost model to extract software requirements from the pull request. The second generates the corresponding test cases, and the third acts as a skeptical reviewer, returning only APPROVED or a specific list of issues—never vague feedback. Before any tests are executed, Semgrep performs static security analysis on both the generated tests and the pull request source code. A caching layer avoids unnecessary scans by skipping files that haven't changed since the previous iteration. Once the tests pass validation, they are uploaded to UiPath Test Cloud through its REST API. To keep the architecture modular, I built the Test Cloud client as a standalone component and validated it independently before integrating it into the pipeline. Throughout the process, a SQLite database stores the pipeline state, allowing a live Streamlit dashboard to display the progress of every stage in real time. The entire system was developed using Claude Code, which significantly accelerated implementation while allowing us to focus on the overall architecture and workflow rather than boilerplate code. Built entirely with Claude Code.
Challenges we ran into
GitHub's API wasn't what I expected. The webhook payload includes a field called changed_files, but it only contains the number of modified files—not the files themselves. I had to make an additional API request to retrieve the actual file list before the pipeline could analyze the PR.
The UiPath Test Cloud integration was more involved than I anticipated. There isn't a single endpoint for uploading results; recording one test execution requires three dependent API calls, each using IDs returned by the previous request. Using the wrong ID produces cryptic errors, so I spent considerable time validating the workflow against the live API.
Authentication was another challenge - although I was issued an API token, the endpoints I needed actually authenticated using browser session cookies. I solved this by reading Chrome's stored cookies during local development while supporting an environment-variable fallback for deployed environments.
I also ran into an environment issue with Semgrep. It had been installed into a different Python environment than the server, so subprocess calls couldn't locate the executable. I fixed this by implementing a resolver that first searches the system PATH and then falls back to the directory of the active Python interpreter.
Finally, I realized I was scanning the wrong files. Initially, my security pipeline analyzed only the AI-generated tests, but well-written tests rarely contain meaningful vulnerabilities. The real security risks exist in the application code itself, so I expanded the scan to include the pull request source code. After that change, the security findings matched my expectations.
Accomplishments that we're proud of
I successfully built a complete end-to-end feedback loop with UiPath Test Cloud. Test results are uploaded automatically; the pass rate is retrieved from the platform, and the real execution data contributes 40% to the merge confidence score. I'm also proud that the entire demo pipeline is deterministic and verified end to end. The same inputs consistently produce the same outputs, and my unit tests validate that the live pipeline behaves exactly as expected. I'm most proud of the Writer–Critic architecture. Instead of accepting the first set of AI-generated tests, I use a second AI agent to review them and force revisions. In practice, the Critic consistently catches missing edge cases, weak assertions, and tests that only cover the happy path, leading to noticeably higher-quality test suites than single-pass generation.
What we learned
One of the biggest lessons I learned was to validate external integrations before building around them. On the first day, I wrote a validation script that exercised every API against the live endpoints. It quickly exposed several incorrect assumptions that would have been much harder to debug once the AI pipeline was complete. I also learned that adversarial AI workflows produce better results than relying on a single model. Having one AI generate tests and a second AI act as a strict reviewer consistently led to stronger test suites, even when both agents were powered by the same underlying model. Finally, I learned that a confidence score is only useful if it's grounded in objective evidence. That's why I weighted UiPath Test Cloud results at 40% of the final score—ensuring that real execution results have more influence than the AI's own assessment.
What's next for Untitled
The next step is to make Crucible even more useful as a continuous quality gate throughout the development lifecycle. Regression tracking. Instead of only determining whether the current tests pass, I want Crucible to compare each pull request against previous runs and flag regressions such as reduced test coverage or weaker test quality. AI-powered fix suggestions. Today, Crucible identifies failing tests and security issues. In the future, I want it to automatically analyze those failures and suggest changes to the source code, helping developers resolve issues instead of simply reporting them. Support for more languages. Since Crucible already consumes standard test result formats, the underlying architecture is largely language-agnostic. Expanding language-specific prompting and analysis will allow it to support a much wider range of development stacks. Native UiPath integration. I'd also like to move more of the orchestration into UiPath Agent Builder itself, using Claude agents as external tools. This would make the workflow more seamless while taking better advantage of UiPath's native automation capabilities.
Log in or sign up for Devpost to join the conversation.