Judge for Jira

Done means the checks passed. Freeze a work item's acceptance checks before work starts; Judge checks every delivery, from people or AI agents, and blocks Done until it passes.

Comment

Inspiration

Since May 2026, Jira lets teams assign work items to AI agents the same way they assign a colleague. Atlassian's own State of Developer Experience 2026 found that 49% of developers spend 4+ hours a week reviewing AI output, correcting prompts and more. The review is the bottleneck, and it is mostly done by eye: someone opens the delivery and checks whether it has what was asked for. Agents also report work as done when it is not. In one public ten-month review of a coding agent's pull requests, the agent wrote that it had fixed the build errors while the build still failed.

Checklist apps can block Done, but a ticked box only proves someone ticked it. Atlassian's own acceptance-criteria check is an AI reading the work and advising a reviewer. I wanted the objective part of "is this done?" to be decided by a fixed program, against checks everyone agreed on before the work started, with a result anyone can re-run.

What it does

  1. Freeze the checks before work starts. In the Judge panel on any work item, write the required text (one item per line, for example "## Summary" or "## How it was tested") and word limits, or write JSON checks for data deliveries (required fields, data shape, exact file hash). Press Freeze checks. Judge stores the checks in app-private storage and posts a comment with the exact checks and their SHA-256, so the goalposts cannot move quietly.
  2. Judge every delivery. A delivery is an attachment, or a comment whose first line starts with "Delivery". A person or an AI agent can hand it in. Within seconds Judge runs the frozen checks with a fixed program (no AI decides the result) and posts PASS or FAIL with a plain reason for each check. Discussion comments are never judged, and neither are comments limited to a role or group.
  3. Block Done until it passes. A workflow validator stops the move to Done and shows why ("The last delivery failed 1 of 2 checks: Required text (missing terms: ## How it was tested)"). Work items without frozen checks are never blocked.
  4. Receipts anyone can re-run. Every receipt carries the hashes of the checks and of the delivery, plus the full receipt data. The panel's Re-check receipt button re-reads the delivery from Jira and runs the checks again. Outside Jira, a small open-source verifier takes the posted checks, the receipt data and the delivered file and prints CONFIRMED with the same receipt hash, or says exactly what differs.
  5. The Judge agent in Rovo drafts checks from a work item's description and explains a failed delivery in plain words, reading the real receipt through its actions. It cannot freeze or change checks.
  6. Outside agents can test their work first (MCP). Through the app's MCP server, any MCP client (a coding agent, for example) can read a work item's checks and run them on a draft before handing it in. It cannot change the checks. An open-source SKILL.md teaches agents the workflow: read the checks, test the draft, hand in only on pass.

How we built it

  • Forge, Node.js 24 runtime, everything on Atlassian's cloud. No outside servers, no egress, no outside AI service. The app is eligible for Runs on Atlassian.
  • Jira: jira:issuePanel (UI Kit) for freezing checks and showing the latest receipt; product event triggers (avi:jira:commented:issue, avi:jira:updated:issue with ignoreSelf) to judge deliveries; Jira REST to read attachments and post receipt comments in ADF; jira:workflowValidator for the Done gate (company-managed and team-managed projects); jira:adminPage as the Get started page shown after install.
  • Storage: Forge key-value storage for the frozen checks and the latest receipt. It stores no account IDs, names or email addresses.
  • Rovo: rovo:agent with two actions (get-judge-status, check-delivery, both read-only) and a reusable rovo:skill that depends on them.
  • MCP: rovo:mcp exposes the same two actions to outside MCP clients once a site admin turns tools on.
  • The checker comes from Judge Protocol, my open-source evaluator for paid AI agent jobs (MIT). For Jira I added comment parsing (ADF to text, one fixed conversion), the freeze record, receipts and their hashes, the re-check path and the offline verifier. 108 automated tests run against in-memory fakes of Jira and storage, plus end-to-end runs on a real site.

Challenges we ran into

  • A Rovo agent from a newly installed app did not appear until the app was reinstalled.
  • One Forge package is CommonJS; under ES modules its default export needed unwrapping, which only showed up at runtime.
  • Jira's change log does not always carry an attachment's file name, so the name now comes from the attachment record itself.
  • Uninstalling an app leaves its workflow rule behind as "Unavailable rule", so the setup steps say how to re-attach it.
  • Making "anyone can re-run it" literally true: every byte that feeds a verdict (the checks, the comment text after a fixed ADF conversion, the attachment bytes) is pinned and hashed, and the receipt hash leaves out the time so a re-run matches.

Accomplishments that we're proud of

  • Real receipts from the live site (KAN-3, and KAN-4 in the video) re-run offline to CONFIRMED with the same hash shown in Jira; the repository's CI re-runs both on every push.
  • An outside MCP client read a work item's checks and tested a draft against them, with nothing posted or stored.
  • Judges install it through the same link I tested: no setup beyond adding one workflow rule.

What we learned

Most of "is this done?" can be decided objectively if the checks are written before the work and never move afterwards. AI is most useful around that core: drafting checks, explaining failures and helping agents fix their work, while the verdict itself stays deterministic.

What's next

  • Checks on code facts from Jira's development information (a linked pull request is merged, builds pass) for coding agents that deliver pull requests.
  • More check kinds for documents and data, such as required headings in order and table columns.
  • A Marketplace listing once the app has run on more teams' work.

Limits

  • Checks cover what a program can decide: required text, length, JSON fields and shape, exact file hash. Anything needing judgment stays with the reviewer.
  • The workflow validator is a Forge Preview feature, and an admin attaches it once per workflow.
  • Rovo features need Rovo on the site; MCP tools need an admin to turn them on.
  • Comments limited to a role or group are not judged.

Source code (MIT): https://github.com/vijaygopalbalasa/judge-for-jira

Built With

Share this project:

Updates

Submission history