RootLense

Incident Postmortem
Incident Postmortem
Incident Postmortem
Dashboard
RCA

The Problem

When a production system breaks, the hardest part isn't receiving the alert — it's figuring out why it broke.

An on-call engineer may need to jump between logs, Kubernetes events, stack traces, GitHub commits, code diffs, and incident conversations before finding the change that actually caused the failure. What should be a focused investigation can turn into hours of manual correlation.

We wanted to change that.

Inspiration

RootLense was inspired by a simple question:

What if an AI SRE could investigate an incident the way an experienced engineer does — by connecting the symptoms to the code change that caused them?

Instead of treating logs, telemetry, and Git history as separate sources, RootLense brings them together into one investigation workflow.

What RootLense Does

RootLense is an AI-powered incident investigation and remediation agent.

When an incident occurs, RootLense:

Collects incident evidence such as logs, stack traces, Kubernetes events, and incident context.
Retrieves relevant GitHub commits and their actual code diffs.
Uses Gemini to correlate the operational symptoms with code changes.
Identifies the most probable breaking commit and affected file.
Highlights the faulty code change and explains the root cause.
Proposes a remediation and regression test.
Creates a Draft Pull Request for the proposed fix.
Generates an incident postmortem from the investigation.

The goal is to move from:

Alert → Investigation → Root Cause → Remediation → Postmortem

within a single workflow.

How We Built It

RootLense is built around an agent-driven backend and an interactive SRE dashboard.

The system connects incident evidence with Git history and code-level diffs before sending the relevant context to Gemini for root-cause analysis.

A key part of the implementation was making the investigation evidence-driven rather than purely generative. Gemini is given the actual code changes associated with candidate commits, allowing the analysis to connect a production symptom to a concrete change in the codebase.

The remediation workflow keeps a human in the loop: RootLense can prepare a proposed fix and Draft PR, but it does not automatically merge or deploy changes to production.

What We Learned

The biggest lesson was that an AI incident-response system cannot rely on a good-looking explanation alone.

The evidence has to agree.

A useful RCA needs to connect:

incident symptoms → telemetry → deployment → commit → diff → faulty lines → fix

We also learned that giving an AI agent too much unrelated repository context can make its reasoning less reliable. Filtering the candidate Git history to the relevant service dramatically improved the quality and consistency of the investigation.

Challenges

One of our biggest challenges was making the AI analysis consistent with the underlying engineering evidence.

Early iterations could produce plausible explanations even when the available Git commits were unrelated to the incident. We addressed this by creating a deterministic orders-api investigation scenario with a healthy baseline commit and a later breaking commit, then feeding the actual Git diff into the RCA pipeline.

Another challenge was connecting the AI reasoning to an actionable remediation workflow rather than stopping at a text-based root-cause explanation.

We therefore built the investigation around structured outputs containing the suspected commit, affected file, faulty lines, evidence, confidence, and recommended fix.

Why RootLense?

Most incident workflows stop at:

"Something is broken."

RootLense aims to answer the harder questions:

"What changed?"

"Why did that change break production?"

"What should we fix?"

"Can we prepare the fix and document the incident?"

That is the gap RootLense is designed to close.

Impact

RootLense turns fragmented incident investigation into a single, evidence-driven workflow for engineering teams.

Instead of manually jumping between observability tools, Git history, and incident documentation, engineers get a consolidated investigation that connects production symptoms to code-level changes and actionable remediation.

The long-term vision is an AI SRE that helps engineering teams spend less time finding the cause and more time fixing the system.