Elastic Postmortem AI with Integrity Guard

Inspiration

Incident post-mortems are often written after the fact with weak links to evidence. I wanted a system where every claim in a post-mortem is tied to concrete timeline refs (logs, alerts, changes, chat) and where a second agent independently checks those refs. Elasticsearch already holds the data; Kibana Agent Builder made it possible to put two agents in the loop - one to generate, one to audit - without free-form hallucination.

What it does

Elastic Postmortem AI generates evidence-linked incident post-mortems and audits them for integrity.

• Narrator agent (Kibana Agent Builder): Uses ES|QL to pull the incident timeline from Elasticsearch, then produces a structured post-mortem where every claim cites evidence refs. No claim without a trace.
• Auditor agent: Re-queries Elasticsearch to validate those refs, challenges unsupported claims, and scores overall + decision integrity (e.g. change-window compliance). Flags governance issues like overstrong causality.
• Streamlit UI: Shows trust status (TRUSTED / REVIEW / HIGH RISK), four audit metrics, and formatted Narrator + Auditor output (summary, claims tables, findings) - not raw JSON. Execution trace shows whether each step ran via Agent Builder or the local fallback.
• Graceful fallback: If Agent Builder isn’t configured or a call fails, the app runs a local deterministic pipeline and labels it clearly so judges and users know what they’re seeing.

How I built it

• Backend: Python scripts for ES|QL timeline loading (dynamic time window from the incident doc), narrator and auditor runners, and an Agent Builder HTTP client that POSTs to the Kibana converse API with multiple path fallbacks and per-agent timeouts.
• Agents: Two agents in Kibana Agent Builder, both restricted to ES|QL only (no other tools). Narrator prompt enforces “execute ES|QL first, then output a single JSON object.” Auditor receives the narrator report and re-fetches the timeline to validate. JSON is extracted from the LLM reply with brace-matching and a “largest valid object” heuristic.
• UI: Streamlit app that tries Agent Builder first for Narrator and Auditor, falls back to the local pipeline on failure, and renders both outputs as markdown (summary, impact, claims, findings) with a “Show raw JSON” toggle. Trust banner and execution trace make the data source obvious.
• Data: Synthetic incident (INC-1042) in Elasticsearch; time window is derived from the incident’s created_at / updated_at so the same pipeline would work for other incidents with data.

Challenges I ran into

• Converse API: Early 404s and timeouts. The narrator often took over 60 seconds; the client needed a longer timeout for the narrator and logic to prefer reporting “Read timed out” over later 404s from fallback paths. Path order was adjusted so the default /api/agent_builder/converse is tried first.
• Response shape: Handling 200 with empty or invalid JSON, and empty response.message, so the app cleanly falls back instead of crashing. Making the UI robust to odd claim/finding shapes (e.g. string vs dict) so the local pipeline and Agent Builder output both render.
• Solo scope: Designing two agents, the client, the runners, and the UI alone meant prioritizing: get Agent Builder working end-to-end first, then add formatted output and fallback labeling so the demo reads clearly.

Accomplishments I'm proud of

• Real Agent Builder integration: Both agents run in Kibana; the execution trace and toasts prove it. Screenshots in the repo show the agents and their ES|QL-only tool config.
• Evidence-linking as a contract: Every claim has evidence_refs; the auditor checks them against Elasticsearch. That’s the main differentiator from “one LLM summary.”
• Production-style client: Multiple converse paths, timeout handling, empty-response handling, and clear “demo fallback” labeling so the app works with or without Kibana and stays honest about which path ran.
• Readable UI: Narrator and Auditor as markdown (tables, sections) instead of a JSON dump, plus a single trust banner and four metrics so judges can assess the result in seconds.

What I learned

• Kibana Agent Builder’s converse API, payload shape, and how to constrain agents to a single tool (ES|QL) for deterministic evidence flow.
• Balancing “ship a working demo” with “make it obvious to judges” - e.g. execution trace, formatted output, and docs that show the architecture and the agents.

What's next for Elastic Postmortem AI

• Add a second synthetic incident (e.g. slow degradation, no deploy) to show the pipeline generalizes.
• Optional: one strong visual (e.g. timeline of events with refs, or a simple claims-vs-evidence view) to make the value prop even clearer at a glance.
• Keep the current design: Agent Builder first, local fallback with clear labeling, and evidence-linked output as the core story.

Built With

• elasticsearch
• es|ql
• kibana-agent-builder-(converse-api)
• python
• render
• streamlit