Credo

Credo reproduces the same payment action against base and PR: one charge versus two
Requirements from Confluence define payment retries, timeouts, and the one-charge rule
Credo surfaces the blocking regression and test evidence directly inside Jira
Ask for an additional test in plain English and run it against both versions
The regression appears as a failed required Credo check on the pull request

Inspiration

We've been using coding agents a lot lately, and they can open a PR in minutes. The problem is that someone still has to review it, and reviewing a diff only tells you what changed, not what it actually does. We've all approved PRs that looked totally fine and then broke something once they ran (like charging a customer twice). We wanted a way to actually see a PR in action before merging it, so we built Credo. It reads what the Jira ticket asked for, runs the old code and the PR side by side, and lets you watch and play with both live, right inside the ticket. Basically, we wanted to be able to merge with conviction instead of just hoping for the best.

What it does

When someone opens a PR for a Jira ticket, Credo reads the ticket's acceptance criteria, the linked Confluence spec, and the code changes. It then spins up two sandboxes, one running the old code and one running the PR, writes tests for each acceptance criterion, and runs them on both. Anything that behaves differently gets flagged, along with the evidence. All of this shows up in a panel inside the Jira ticket, where you can watch the agent click through the app in real time, replay any test afterwards, jump in and try both versions yourself, make the payment provider slow or fail, or just ask for a new test in plain English, either in the panel or by asking Rovo. If Credo finds a regression, the PR can't be merged until it's fixed. Credo also comments on the ticket and the PR and offers to file a Bug, but anything that changes the ticket still needs a human to click the button. In our demo, a PR that "simplifies payment retries" accidentally lets a customer get charged multiple times for the same order, and Credo catches it every time we've run it.

How we built it

The Jira side is a Forge app: the panel is built with UI Kit, web triggers listen for GitHub webhooks and updates from our runner, and a Rovo agent lets you talk to Credo from chat. The runner lives on an Azure VM. For each PR, it checks out the old and new code into separate Docker sandboxes (each with its own app, database, and fake payment provider), asks Factory Droid to plan the test scenarios, runs them with Playwright, and streams the agent's browser back into Jira so you can watch live. We also pull in Greptile's review so the agent knows what looks risky before it starts testing.

Challenges we ran into

Deployment was a bit of a hassle. Our coding environment couldn't SSH into the VM, so every deploy had to go through GitHub Actions, and debugging meant digging through workflow logs. Getting Jira, Confluence, and Forge all set up on a demo site took way longer than we expected (our first Forge deploy kept failing because Forge's bundler didn't like how our shared code was set up, so we had to change how it gets copied into the app). The AI planner was tricky too, since it would write different tests on different runs, and sometimes it would say a criterion was covered when the test didn't actually prove it. We ended up adding checks so it can't claim something passed without real proof.

Accomplishments that we're proud of

Our favorite moment was when Credo caught a mistake in our own spec. One of our acceptance criteria said a payment timeout should leave the order "pending", and Credo failed it on both the old and new code. Turns out the app had always marked it as failed, so the bug was in our ticket, not the code. That's exactly the kind of thing we built Credo to catch! We're also proud that the whole thing works end to end on a real PR: open the PR, two sandboxes spin up, the double charge shows up 10 out of 10 times in our repeat test, and the results land in Jira, GitHub, and Rovo without anyone touching a terminal.

What we learned

Getting an AI to write tests is easy, but getting it to write tests you can actually trust is a lot harder. A lot of our time went into making Credo honest, like never counting a crash as a pass and showing what actually happened ("2 charges, expected 1") instead of what the test was supposed to check. We also learned a ton about Forge and Rovo, and about running throwaway Docker sandboxes on a VM, way more than we expected to in 36 hours.

What's next for Credo

Something we really wanted to add but didn't have time for is load testing. Right now our demo only fires a handful of payments at once, but imagine the sandbox simulating thousands of users hitting the app like it's production, so you catch the bugs that only show up at scale before your customers do. We'd also like Credo to save the tests it writes back into the repo, so they keep running after the PR is merged. Eventually we want it to work on more than just our checkout demo!