ReleaseProof

A Jira issue panel that recomputes ML release evidence, blocks hidden slice regressions and binds approval to exactly the policy and predictions that were reviewed.

Comment

Inspiration

Model releases are often reviewed through one headline number. An overall error metric can improve while one slice of the data, a region, a customer segment or a device type, gets much worse. And once an approval is recorded, nothing ties it to what was actually reviewed: the thresholds can be loosened or the predictions swapped afterwards, and the old "approved" still looks valid.

We wanted the release issue itself to hold the evidence: recompute the metrics from the raw predictions, check every declared slice, and make an approval expire the moment the policy or evidence it covered changes.

What it does

ReleaseProof is an Atlassian Forge app that adds a panel to Jira issues. On a release-review issue:

  • Policy. The team declares the release contract: dataset and model IDs, maximum overall RMSE, maximum overall regression versus the baseline, minimum rows per slice, and per-slice RMSE and regression limits.
  • Evidence. Reviewers import holdout predictions as CSV (id,slice,actual,baseline,candidate, up to 1,000 rows). ReleaseProof recomputes baseline and candidate RMSE overall and per slice. It does not accept a pasted aggregate.
  • Checks. Each threshold becomes a named pass/blocked check. Missing, undersized or undeclared slices block approval, as do failing thresholds. Approval is disabled until every check passes.
  • Evidence-bound approval. Approval records the Jira account, the time, and a SHA-256 fingerprint of the normalized policy and every prediction row. If either changes, the approval is marked stale and the issue shows Revalidation required. Reordering identical rows does not invalidate it; changing a value does.
  • Export. "Export evidence JSON" downloads the complete record: policy, all rows, metrics, checks, fingerprints, the approval with its current status, and the tool's stated limitations.
  • Per-issue, permission-aware. State is stored per Jira issue. Users who can view but not edit the issue get a read-only panel and can still export.

The demo (synthetic data, generated with a fixed seed): a pooled candidate model improves overall holdout RMSE from 16.51 to 7.25, but the rural slice gets worse, from 6.02 to 21.31 (+15.30). ReleaseProof blocks it with two failed checks. A corrected per-slice candidate (overall 1.74, rural 1.68) passes and can be approved. Tightening the overall RMSE limit from 20 to 19 afterwards makes that approval stale.

How we built it

  • Forge app with a jira:issuePanel module, a Custom UI front end (vanilla JavaScript, bundled with esbuild) and a resolver function on the Node.js 22 runtime.
  • Identity and permissions from Forge, not the client. The resolver takes the issue and account from the invocation context. It reads the issue as the user (api.asUser()), and checks authorize().onJiraIssue(id).canEdit() on every call. The service refuses policy changes, imports and approvals with 403 when that check fails.
  • Storage: Forge KVS, keyed by Jira issue ID. Policy, evidence and approval are separate keys. An approval request carries the fingerprint the reviewer saw and is rejected if the state has moved on, and the state is re-read after every write, so a concurrent edit shows up as stale instead of being overwritten.
  • Scopes: read:jira-work, read:jira-user, storage:app. No Jira write scope, no external fetch permission, no AI or inference service. The approver's display name is looked up at read time; only the account ID is stored.
  • Domain logic in one dependency-free module: CSV parsing (BOM, CRLF, quoted fields, reordered headers), strict numeric validation, metrics, checks and canonical SHA-256 fingerprints. The same code runs behind a loopback local server for development. Ten behavioural tests cover hidden slice regressions, support gates, canonical hashing, stale approval, CSV and size boundaries, concurrent edits, read-only denial and the display-name lookup.
  • Demo data: a generator script fits models on 2,000 synthetic training rows and evaluates them on a separate 1,000-row holdout (900 urban, 100 rural), seed 20261029. Formula and coefficients are in demo/recipe.json.

Challenges we ran into

  • A crash that only happened in Jira. The first live run failed with out_namespaceObject is not a constructor. With "type": "module", the Forge bundler hands over @forge/resolver's CommonJS module.exports, so the resolver class lives on .default. Local tests could not catch this; running in a real Jira site did.
  • Getting approval invalidation right. Approval needed to depend on what matters (thresholds, identifiers, row content) and ignore what doesn't (row order, slice order). It also needed to survive concurrent edits without an old approval resurrecting old inputs.
  • Showing who approved without storing personal data. Adding display names needed a new scope, and that made the change a major version that installations must re-approve. We resolve the name at read time and keep only the account ID in storage.

Accomplishments that we're proud of

  • The full flow runs in real Jira Cloud on a second site, installed from the public installation link: blocked regression, passing correction, approval, stale approval after a policy change, evidence export, and empty state on another issue. The demo video is a recording of that site.
  • The read-only path was checked with a second Atlassian account that can browse but not edit: every editing control is disabled, export still works, and the backend refuses writes independently of the UI.
  • Small permission footprint: three scopes, no egress, no inference cost.

What we learned

  • A metric shown in an issue is only useful if you can say which policy and which rows it came from. Binding approval to a fingerprint made "is this approval still valid?" a question the app can answer.
  • Forge's context-derived identity and per-call permission checks make it straightforward to avoid trusting the client.
  • Test in the real product early. The bundler interop bug never appeared locally.

What's next for ReleaseProof

  • A walkthrough with an ML team on a real release review, to learn what evidence they actually attach. Customer demand is not established yet.
  • More metrics beyond RMSE (for example MAE and classification metrics) using the same policy and fingerprint model.
  • Larger evidence sets than 1,000 rows, and a history of past approvals instead of only the current one.

AI use and honesty notes

  • Built with AI coding assistance: Claude (Anthropic) wrote and tested the code, deployment steps, documentation and this demo video under the entrant's direction. The entrant chose the event and approved the scope.
  • The video's narration is a synthetic voice (Kokoro-82M, Apache-2.0), and the video says so. Its footage is a real Jira Cloud session on Nov 5, 2026. Waits are sped up where marked, and the mouse pointer and the browser confirm dialog are redrawn from the session log, because the headless browser draws neither.
  • All model data is synthetic. No customer, user count or production accuracy claim is made.
  • ReleaseProof checks declared thresholds. It is not a fairness guarantee, a statistical significance test, a Jira workflow gate or an immutable audit ledger.

Built With

Share this project:

Updates

Submission history