Incident Root Cause Flow

• 

  Root cause analysis of the demo project in progress.

• 

  Root cause analysis of the demo project -completed.

Inspiration

It’s 2:00 AM. Your phone buzzes because production checkout is failing with 500 errors. The true cost of a critical outage isn't just fixing the bug—it's the hours of forensic archaeology spent digging through dozens of recent merge requests, file diffs, and CI/CD logs trying to locate what broke. We built this project to turn a chaotic, multi-hour manual search into a deterministic, automated, 30-second operation.

What it does

The Incident Root Cause Flow operates as an automated first-responder natively within GitLab. The moment a new Incident work item is created, the flow triggers automatically. It pulls context, evaluates recent changes, and leaves a formatted analysis comment on the incident:

• High Confidence: Applies a likely-root-cause label and tags the exact author of the breaking merge request.
• Low Confidence: Ranks and presents the top 3 suspicious candidates with explicit graph evidence, giving the on-call engineer an instant running start.

How we built it

The system is built entirely on the GitLab Duo Agent Platform utilizing a native AI Catalog flow configuration, eliminating the need for external webhooks or servers.

Instead of relying solely on an LLM to blindly guess where a bug might hide, we use explicit graph traversal via GitLab Orbit. The flow maps the following path: WorkItem (Incident) → Project → MergeRequest → HAS_HEAD_PIPELINE → MergeRequestDiffFile → File → Definition ←[CALLS]← Downstream Callers

By extracting the exact definitions modified in recent MRs and matching them against downstream call graphs, the engine packages this structured graph data and feeds it to the LLM to perform precise logical scoring.

Challenges we ran into

Navigating structural call graphs and resolving multi-step relationships dynamically through the Orbit API required meticulous mapping. We had to ensure that lookback boundaries remained strict to thoroughly investigate downstream function boundaries without hitting timeout limits on the Duo Agent platform.

Accomplishments that we're proud of

We successfully pivoted from an external FastAPI webhook architecture to a 100% native GitLab Duo AI Catalog Flow. In our test suite, the agent successfully traced an off-by-one bug in a central check function (quota.py), calculated its blast radius across independent services, and automatically alerted the right developer without human intervention.

What we learned

Deterministic metadata from a robust knowledge graph like GitLab Orbit supercharges generative AI. Giving an LLM raw text search is messy, but providing it a strictly structured graph relationship map allows it to reason with near-perfect reliability and clear accountability.

What's next for Incident Root Cause Flow

We want to expand the traversal model to dynamically analyze external infrastructure states and automated rollback integrations. If the agent detects high confidence over a critical production bug, it could automatically draft a revert Merge Request and queue it up for the team, moving toward self-healing deployment pipelines.

Built With

• gitlab-duo
• gitlab-orbit
• google-gemini
• knowledge-graph
• yaml