Config Doctor for Jira

Deterministic, auditable answers to "why can't they see it?" — explained by Rovo.

Comment

Inspiration

"Why can't X see this issue?" is one of the most common tickets a Jira admin gets. Answering it means tracing permission schemes, project roles, groups and issue security levels by hand — often half a day. At the same time configuration sprawl (duplicate fields, dead groups, unused schemes) makes every change risky, and admins can't see what a change will break before making it. Generic AI makes this worse: an LLM asked about permissions will happily guess, and nobody can audit what it did.

What it does

Config Doctor is a Rovo agent plus a Jira admin page whose every answer is computed by deterministic code and cross-checked with Jira's own permission API (ground truth) — 30 of 30 scripted questions answered identically across three runs. It works without Rovo too (admin page), every action lands in an audit log, the same tools are exposed over Rovo MCP, and judges can try it on an empty site with the one-click Demo Lab:

  • Access Explainer — "Why can't Linh see PROJ-123?" returns a verdict, the blocking layer (permission scheme, issue security level or account), the evidence chain (grant → role → group), and fix options that are simulated against the model, each with its blast radius ("adding Linh to staff-hq also grants VEND and two PAY security levels"). Every verdict is cross-checked with Jira's own permission API; if they disagree, Config Doctor says so instead of hiding it.
  • Who can see this? — every viewer and the path that gives them access, after issue security, verified person by person against Jira.
  • Sprawl Audit — an explainable health score (0–100) and ranked findings: near-duplicate fields ("Sprint Goal" / "Sprint goal" / "Sprint Goal (old)"), fields on no screen, empty groups that still hold grants (a silent backdoor), unused schemes, inactive workflows, unused screens. One confirmation turns findings into a Jira cleanup epic.
  • Change Impact Preview — "What happens if I remove group contractors-2024?" → who loses which permission in which project, and who keeps access through another path. Read-only simulation.
  • Proactive watch — weekly scan, trend and "new since last scan".
  • Audit log — every action from chat, MCP, the admin page or the scheduler: who, when, inputs, result.
  • MCP — the same tools are available to any Rovo Studio agent via rovo:mcp.
  • Demo Lab — one click builds a labelled scenario on any site (a restricted issue you can't see, an empty group that still holds a grant, near-duplicate fields); one click removes exactly what it created.

How we built it

Forge (Node 24, TypeScript) with rovo:agent, 9 actions, rovo:mcp, jira:adminPage (UI Kit), async-event consumer and scheduledTrigger, Forge KVS. All reasoning lives in src/core — pure functions with no Forge imports and 100+ unit tests over fixtures (holder types, roles, groups, reporter/assignee, user/group-picker fields, security levels, product access). The agent prompt only routes questions and formats evidence. Every action validates LLM-supplied inputs, identifies the user from the Forge context, checks they are a Jira admin server-side, and writes an audit record. The app is read-only except two opt-in, previewed, token-confirmed writes: the Demo Lab (creates and later deletes only its own recorded objects) and the cleanup plan (creates issues). The scanner is chunked and time-sliced over async events (a full scan of a 25-project demo site takes ~10 s), and snapshots feed the trend chart.

Challenges

  • Jira's own answers are eventually consistent. Right after a group change, permissions/check flipped between yes and no; we now take the majority of up to three checks and show every raw answer.
  • user/permission/search ignores issue security levels — it listed 7 users for an issue only 3 can see, so viewer lists are verified per person with permissions/check.
  • Forge app users cannot run permissions/check for other users (403); the check runs as the invoking admin.
  • Keeping actions under Forge's 55 s limit: cached site index, bounded concurrency and deadlines.
  • Rovo sometimes drops optional inputs — every action works with required inputs plus deterministic context.

Accomplishments

  • 30 / 30 scripted access questions answered identically across three runs, all matching Jira's own check.
  • 132 unit tests over the pure core (≈95% statement coverage); an independent code + security review whose findings were all fixed before submission.
  • A sprawl audit that found every planted problem in our demo site, and a cleanup plan created only after preview → explicit yes → Rovo's own confirmation.
  • One admin page that delivers the core value even on sites where Rovo is not enabled.

What we learned

Agents are most trustworthy when the model never computes facts. Treat the LLM as a router and narrator, make code produce evidence, compare with the platform's ground truth, and log everything.

What's next

Filter/dashboard-sharing analysis for groups, team-managed project access levels, Confluence space permissions, scheduled "permission drift" alerts, and a Marketplace listing.

Built With

Share this project:

Updates

Submission history