Retrace

AI triage for Jira you can replay: every decision is hash-chained, explained and verifiable.

Comment

Inspiration

AI agents are getting good at triaging Jira issues, but teams can't check why an agent chose a priority or an assignee, or whether it would choose the same thing tomorrow. For software teams, that makes AI triage hard to trust and impossible to audit. We wanted triage that you can replay like a test: same inputs, same decision, same hash, with the reasons written out.

What it does

Retrace suggests a priority, labels and an assignee for a Jira issue, and records every step of that decision as a deterministic, sha256 hash-chained trace.

  • Triage + record: a priority badge, labels and an assignee, with plain-language reasons ("crash: the app crashes or freezes (+4)") and a root hash.
  • Determinism: triage an unchanged issue again and you get the same hash. Retrace says "Unchanged" instead of saving a new trace.
  • Replay: re-execute the stored trace from its recorded inputs and check every step hash. The result is Verified, or Mismatch naming the step that diverged.
  • What-if: pretend the issue had other labels, had labels removed, or had a forced priority. Retrace reruns with the same seed and shows a before → after diff. Nothing is saved.
  • Three surfaces, one engine: a Jira issue sidebar panel, a Rovo agent (Retrace Triage), and an MCP server that Rovo Studio agents and outside MCP clients can use, plus an open-source SKILL.md.

Retrace is read-only and advisory. It never edits issues.

How we built it

  • Forge app on Jira Cloud. The UI Kit issue panel is in the issue sidebar (jira:issueContext), and Forge storage (KVS) holds traces. Jira REST is called through asApp. Scopes are just read:jira-work and storage:app, with no egress.
  • Rovo: a rovo:agent with four action modules: find-issues (JQL search, so the agent never guesses keys), triage-issue, replay-trace and what-if-trace. A rovo:skill packages SKILL.md instructions.
  • MCP: a rovo:mcp module exposes the same four actions as tools, at https://mcp.atlassian.com/forge/2b9b66b6-b31e-4dcd-810f-9b73f6cf9015.
  • Deterministic core in plain TypeScript: canonical JSON, a seeded RNG for tie-breaks, a sha256 step chain, replay, and a what-if diff. Time is never hashed, and order-insensitive inputs are sorted.
  • Tests: 57 vitest tests, including golden root-hash fixtures that pin the engine's output and an end-to-end run through the MCP tool handlers.

Challenges we ran into

  • Keeping determinism honest. During development, rewording a single reason string changed every step hash, and old traces stopped verifying. We kept the hashed text fixed and moved friendlier wording into the display layer. Golden-hash tests now fail if hashed output drifts within an engine version.
  • Agents guessing issue keys. Early on, Rovo invented keys and got 404s. We added a find-issues action and prompt rules so the agent only uses real keys.
  • Making the panel visible and robust. We moved to the issue sidebar section, and added loading states, an error boundary, and handling for traces stored in an older format.
  • Making what-if mean what it says. Removing a label now also stops the scorer from re-detecting it in the text, and the result is still reproducible.

Accomplishments that we're proud of

  • A Rovo answer, the issue panel and an MCP client all produce the same trace hash for the same issue.
  • Replay gives a clear Verified or Mismatch result that anyone can check.
  • An explainable scorer whose reasons read like a colleague's notes.
  • A small, read-only permission footprint with no external egress.

What we learned

  • Building Rovo agents, actions, skills and MCP tools on Forge, and how one action can serve all three.
  • Determinism is a product feature: it has to be designed in (fixed seeds, canonical data, no clocks) and then protected with tests.
  • Agents behave far better when their tools return clear errors and their instructions say exactly what to do when something fails.

What's next for Retrace

  • Learning assignee skills from components and history, still deterministically.
  • Optionally applying the suggestion to the issue, with the trace hash attached as evidence.
  • Confluence audit reports of trace history.
  • A Marketplace listing.

Built With

Share this project:

Updates

Submission history