Inspiration

Most incidents are preceded by signals that were already written down: an unanswered question on a story, a customer ticket closed as "cannot reproduce", a note on another team's Confluence page. Postmortems start the timeline when the alert fires, so the most important lesson gets lost: the organization knew, but the knowledge never travelled to the team that could act.

Existing tools help after the fact. Post-incident review features build a timeline from the moment the incident opened, and AI assistants surface similar past incidents. We wanted to answer two questions most teams never measure: how many days before the incident did the first signal appear, and which single missing connection would have shortened that time the most?

What it does

Unlike tools that surface similar past incidents or track postmortem actions, Hindsight finds the warning signs your teams had already written in Jira and Confluence before the incident, quotes them verbatim with dates and days-before-incident, shows where each signal stopped between teams, and turns that gap into a permanent route.

  • Looks back in time. Click Analyze on an incident. Hindsight scans Jira, Jira Service Management and Confluence up to 30 days before it. A deterministic filter narrows the field before any model call; on the demo incident, about 50 artifacts are scanned, 5 become candidates, and Forge LLM keeps 4 signals.
  • Quotes every signal. Each signal card shows the signal type, the exact quote, a link to the source, and the number of days before the incident. Quotes are verified against the source text; an invented quote is discarded. In the demo: a Platform page noting the payment provider is retiring TLS 1.1 (12 days before), two customer complaints about failed payments (6 and 5 days before), and an engineer's question about HTTP 426 errors (3 days before).
  • Keeps a human in the loop. Signals are Proposed until a person clicks Confirm (with Undo). Nothing counts in the metrics until it is confirmed.
  • Maps where signals stopped. Trace reach reads Teamwork Graph to see which teams provably touched each signal and draws a reach map. In the demo: first signal 12 days before, awareness gap 12 days, reach 0 of 4. Every signal stopped at Arkana Platform, one missing link from Arkana Payments, the team that owned checkout.
  • Proposes the missing connection. Hindsight proposes up to three routes, ranked by estimated days saved, each with its assumption stated. The top demo route mentions Arkana Payments on payment-related activity in the Platform Confluence space: an estimated ~11 days earlier (up to 12). Approve it and the route becomes Active.
  • Catches the next signal (Sentinel). An Active route watches new Jira issues and comments and new Confluence pages and comments in its source. When a new artifact matches the route's keywords (deterministic, no model call), Hindsight posts one neutral, blameless comment on it: "[Hindsight Sentinel] Heads-up for Arkana Payments", linking the past incident (INC-60). The comment names the team, never a person, and does not repeat the matched keywords; on Jira Service Management projects it is an internal note. In our demo, an admin clicks Post a new provider notice and the heads-up appears on the Platform page within a few minutes, once Confluence search has indexed the new comment (about 70 seconds in our production test). The Routes tab shows "1 alert posted · Matched: payment, gateway" with Useful / Not useful feedback. Admins can Pause and Resume a route. Sentinel ignores its own comments, never alerts twice for the same artifact, and is capped per team per week and per day. Sentinel is live in the production app judges install, and no data leaves Atlassian.
  • Works with agents. The Rovo agent Hindsight Investigator and the reusable skill early-signal-investigation call the same actions as the panel and return the same numbers. Write actions (such as confirming a signal) always show Rovo's review form first.
  • Works over MCP. Read-only tools find-prior-signals, trace-signal-reach and propose-routes are exposed through Atlassian MCP to any MCP client.
  • Writes back to Teamwork Graph. Each analysis writes a "Signal dossier" object through a Forge graph connector, discoverable in Rovo Search.
  • Blameless by design. Reports name teams and roles, never individuals; there are no individual metrics; the app checks the viewer's own permissions before showing a signal.

How Hindsight differs from Ripple (our other Codegeist entry): Hindsight looks backward from an incident to find the early-warning signals and route them to the right team, while Ripple keeps restated facts across Confluence pages in sync with their canonical source and routes stale copies to the page owner.

How we built it

  • Atlassian Forge app for Jira (including Jira Service Management) and Confluence, one installation link. Custom UI with React and @forge/bridge: an issue panel on the incident (tabs Signals, Reach map, Routes; SVG reach map) and an admin Setup page (service mapping, demo data).
  • Forge queues run the pipeline as small, resumable jobs: collect → filter → score → reach → propose, plus schema migration and demo jobs, with per-step progress in the UI. 429 responses from Jira or the LLM are retried through the queue.
  • Forge triggers feed Sentinel: a Jira trigger (issue created, issue commented) with ignoreSelf, and a separate Confluence trigger (page created/updated, comment created). The handler only enqueues; a sentinel queue consumer reads the artifact, matches Active routes and posts the comment, with dedup, weekly/daily caps and retries.
  • Forge SQL stores events, candidates, signals, decisions, reach paths, routes and Sentinel alerts (with feedback). Migrations run in a dedicated consumer.
  • Forge LLM (@forge/llm): the model is picked at runtime with list() (Claude Haiku 4.5 for scoring, Claude Sonnet 4.6 as the strong model) with a fallback chain. We force a tool call, then parse and validate the JSON arguments. Artifact text is treated as untrusted data, never as instructions. The LLM is only called from queue consumers, never from UI resolvers.
  • Teamwork Graph (read) via requestTeamworkGraph as the user: atlassianUserInTeams, atlassianUserAssignedJiraWorkItems and confluencePageContributors to map which teams touched each signal. The app's own account is excluded from reach.
  • Teamwork Graph (write) via a graph:connector module (@forge/teamwork-graph): the scoring consumer writes a Signal dossier with the stored connectionId.
  • Rovo: rovo:agent Hindsight Investigator, rovo:skill early-signal-investigation, and actions find-prior-signals, confirm-signal, trace-signal-reach, propose-routes, create-route.
  • Atlassian MCP: a rovo:mcp module exposing the three GET actions only. Write actions are never exposed over MCP.
  • Runs on Atlassian: no egress, no remote, no external API or LLM. No data leaves Atlassian. forge eligibility reports the app as eligible for Runs on Atlassian in both development and production.
  • TypeScript strict, Vitest unit tests (filter, quote verification, metrics, routes): 575 passing across 30 files.
  • Demo scores are cached per artifact, so re-running Analyze on the demo incident returns identical signals instead of re-scoring.

Challenges we ran into

  • Trusting LLM output. A model can invent a quote or over-flag noise. We filter deterministically first, require every quote to match the source exactly, and require human confirmation. LLM results are still not fully deterministic between runs even at temperature 0 (re-analyzing the demo incident once surfaced a fifth, weaker signal), so analysis results are stored and demo scores are cached per artifact: a repeated Analyze hits the cache and returns the same signals.
  • Time travel for demo data. Jira and Confluence created dates cannot be backdated. Load demo data creates real artifacts now and stores the scenario time ("12 days before the incident") separately. The incident panel shows a "Simulated history" badge on demo incidents, so nobody mistakes the demo for real history.
  • Provable reach, not guesses. We cannot know who read a page, so reach only counts recorded interactions. Teamwork Graph returned no contributors for pages written by the app, so page contributors come from page version editors; the app account itself is detected and excluded.
  • A live alert that cannot loop. Sentinel writes comments into the same spaces it watches. We filter the app's own events in the Jira trigger, skip artifacts written by the app account in the consumer, and deduplicate every alert, so one notice produces exactly one heads-up.
  • A removed endpoint mid-build. Reading Confluence content through the v1 GET /wiki/rest/api/content/{id} endpoint started returning 410: This deprecated endpoint has been removed. We moved Sentinel to the v2 page, space and comment endpoints with a CQL search fallback. A brand-new comment is only readable once it is indexed, so Sentinel re-queues it up to six times with growing delays (60 to 120 seconds, about 10.5 minutes in total) instead of failing. Rate-limit (429) responses are retried at most three times, and permanent errors are skipped without retries.
  • Sites without JSM. Demo setup falls back to a software project when Jira Service Management is not available, and the analysis returns the same four signals.
  • Enabling MCP tools. Enabling tools while the app was connected to both Jira and Confluence returned error -32014. We found a reliable workaround (disconnect Confluence, disable and re-enable tools, reconnect) and documented it for judges.

Accomplishments that we're proud of

  • An end-to-end loop on real Atlassian data: Analyze → 4 quoted signals → Confirm → Trace reach (0 of 4, 12-day awareness gap) → Approve the top route → Sentinel posts a heads-up for the owner team on the next provider notice.
  • The same numbers everywhere: the issue panel, the Rovo agent, and an external MCP client all report 4 confirmed signals, reach 0 of 4 and a 12-day awareness gap for the demo incident.
  • Proven with an external MCP client (MCP Inspector) on the production app: tools/list returns all three tools and tools/call succeeds.
  • Signal dossiers written to Teamwork Graph from a background job and found in Rovo Search.
  • The judge path also works on a site without Jira Service Management.

What we learned

  • The most valuable part of a postmortem happens before the alert: the signals were there, but the path between teams was missing.
  • Constraining the LLM (deterministic filter, forced tool call, verified quotes, human confirmation) matters more than prompt cleverness.
  • "Provable reach" is more honest and more useful than guessing who saw what.
  • Blameless wording is a product feature: describing paths and missing links keeps the focus on the system, not on people.

What's next for Hindsight

  • Weak-signal detection for Sentinel: compare new artifacts with confirmed signal patterns from past incidents and suggest (never alarm) when one looks similar.
  • A backwards timeline view and Hindsight Insights: time-to-first-signal, awareness gap and signal reach per quarter and per service.
  • Assignable agent: assign Hindsight Investigator to an incident and get an "Early signals" briefing written into the PIR page (write-pir-section).
  • Marketplace: a separate listing app, "Hindsight Listing" (Paid via Atlassian, a different Forge app ID from the app judges install), is submitted for Marketplace review (ticket ECOHELP-181403). Next step: complete the review and publish the listing.

Built With

  • atlassian-forge
  • atlassian-mcp
  • claude-haiku-4.5
  • claude-sonnet-4.6
  • confluence
  • custom-ui
  • forge-llm
  • forge-queues
  • forge-sql
  • graph-connector
  • jira
  • jira-service-management
  • react
  • rovo
  • rovo-agent
  • rovo-skill
  • teamwork-graph
  • typescript
  • vitest
Share this project:

Updates

Submission history