DevSleuthAgent

DevSleuthAgent turns vague Jira bugs into proven, reviewable fixes. GPT-5.6 investigates real code, sandboxes every claim, replays failures, and opens draft GitHub PRs only after validation.

Comment

DevSleuthAgent

Inspiration

Bug reports are usually vague. Someone heard about a problem from someone else, and now an engineer has to sort out whether it's even real. That first stage eats hours. You have to figure out if the bug is real, which version broke it, and how to reproduce it.

I built DevSleuthAgent to speed that up. It takes a Jira bug ticket and turns it into actual proof, then drafts a GitHub PR only if it can show the bug is real and the fix works.

What it does

DevSleuthAgent watches for labelled Jira tickets, maps each one to an approved repo, and pins the exact commit. Then it investigates in the background.

GPT-5.6 reads a bounded slice of the code and writes a regression test using the repo's real APIs. The test runs in a locked down Docker sandbox with no network and read only source. A failure has to repeat across clean replays before it counts as reproduced.

Once the bug is proven, the fix step starts. GPT-5.6 suggests a small code change. Then DevSleuthAgent runs three checks. The test should fail without the change, pass with it, and the full suite should still pass. If all three holds, it opens a draft PR for review and links the evidence back to Jira.

The model only investigates. It doesn't get the final say.

How I built it

It's a Python service built on FastAPI, with background job tracking, an evidence store, a restricted Docker sandbox, Jira webhooks, GitHub draft PR publishing, and a live web UI.

The model calls use the OpenAI Responses API with GPT-5.6 and store: false. The model gets no shell, no Docker, no Jira or GitHub access, and no write access to the repo. All of that sits behind service layers and validation gates.

Challenges

The hardest part was making sure a generated test isn't mistaken for proof. A bad assertion can fail even when the product works fine. So the agent checks why the test failed, replays matching failures in fresh sandboxes, and backs off when the evidence isn't strong enough.

Validating the model's patches safely was the other big one. I added bounded edits, disposable checkouts, before and after checks, full suite runs, and draft only PRs. The agent never merges or deploys.

What I learned

Reliable coding agents need more than good prompts. They need grounded code context, isolated execution, deterministic checks, a clear way to say "not enough evidence", and a human at the final call.

I also learned that a real tool needs the whole workflow, not just a model call. Jira intake, live progress, immutable evidence, fix validation, GitHub review, and a link back to the original ticket are what make it useful.

What's next for DevSleuthAgent

Right now the loop works for one flow. Jira ticket, verified evidence, validated draft PR. The next step is making it hold up for enterprise teams running thousands of repos and millions of events.

  1. Scale the execution platform Move from an in-process job registry to a durable, event-driven setup. Signed Jira and GitHub events go into a queue with idempotency keys, retries, and dead letter handling. Stateless workers scale out, and sandbox workers get scheduled on their own so one slow repo can't block another team.

  2. Enterprise controls. Each org gets an isolated tenant, repo allow-lists, encrypted credentials, role-based access, and an audit trail for every ticket, model call, sandbox run, patch, and publish. Admins set the policies, like which repos can be investigated and whether auto draft PRs are allowed.

  3. A broader intelligence layer. The same evidence engine can spot duplicate incidents, cluster recurring failures, flag flaky tests, and catch regressions after a deploy. Over time it builds a searchable graph linking tickets, commits, tests, sandbox runs, patches, PRs, and incidents. So teams can ask what changed, what was proven, who approved it, and whether it's happened before.

  4. Human control stays the default. Scale shouldn't mean unchecked autonomy. The model proposes, execution verifies, policy governs, and humans own the merge and deploy calls.

The goal isn't to generate more code. It's to give every team a trusted, auditable way to turn shaky bug reports into verified action.

Built With

Share this project:

Updates