RootCause

Inspiration

Production incidents often leave engineers searching through dashboards, logs, and millions of telemetry events to answer one critical question: what actually caused this?

We wanted to build RootCause to make incident investigation faster and more reliable. Instead of manually writing queries and correlating metrics, a user can simply describe what happened in plain English, and RootCause investigates the underlying playback data to identify the strongest evidence-backed explanation.

Our key principle was simple: AI should help investigate and explain incidents, but it should not hallucinate the root cause.

What it does

RootCause is an AI-powered incident investigation system for streaming platforms.

A user can enter an incident such as:

"Rebuffering spiked last night in the Southwest."

RootCause then performs a multi-stage investigation:

Parses the incident description Resolves the investigation scope Queries millions of playback events Compares the incident window against historical data Confirms whether a meaningful anomaly exists Decomposes the anomaly across multiple dimensions Ranks the strongest contributing signals Drills into interacting factors Identifies a likely culprit segment Generates an evidence-backed explanation and suggested mitigation

The result is presented through an interactive web interface with key metrics, investigation findings, and visual evidence.

How we built it

RootCause is built with:

Python for the diagnostic pipeline FastAPI for the backend API ClickHouse for high-performance analysis of millions of playback events Gemini for natural-language incident interpretation and report generation Chart.js for visualizing investigation evidence HTML/CSS/JavaScript for the interactive frontend

The architecture separates AI interpretation from deterministic evidence analysis.

Gemini helps understand the user's incident description and communicates the final findings. However, anomaly detection and root-cause attribution are performed through deterministic queries and scoring logic against ClickHouse data.

The investigation pipeline follows seven stages:

Parse incident intent Resolve investigation scope Confirm anomaly Decompose across dimensions Rank the strongest excess signal Drill into interacting factors Generate an evidence-backed report Challenges we ran into

One of our biggest challenges was ensuring that the AI did not simply invent a plausible explanation.

We addressed this by designing RootCause around a deterministic investigation pipeline. The LLM does not independently decide the culprit; instead, the underlying data determines whether an anomaly exists and which segments contribute the strongest excess signal.

Another challenge was handling cases where the historical baseline was zero. A zero baseline should not automatically mean there is no anomaly. We added explicit logic to detect newly emerged anomalies when a previously unseen metric suddenly becomes significant.

We also encountered challenges in root-cause ranking. A dimension with a large amount of traffic is not necessarily the actual explanation for an incident. We refined the ranking logic to consider anomalous excess signal and concentration rather than relying only on raw volume.

Accomplishments that we're proud of

We're proud that RootCause can investigate a dataset containing millions of playback events and turn a plain-English incident into an evidence-backed diagnosis.

Some accomplishments include:

Built a deterministic multi-stage investigation pipeline Analyzed over 5 million synthetic playback events Implemented anomaly detection against historical baselines Built multi-dimensional root-cause decomposition Added interacting-factor analysis Handled newly emerged anomalies with zero historical baselines Separated AI narration from evidence-based attribution Built an interactive investigation interface Generated actionable mitigation recommendations

Most importantly, we built RootCause around the idea that the data should determine the root cause—not the language model's guess.

What we learned

Building RootCause taught us that agentic AI systems are most reliable when AI reasoning is combined with deterministic tools and verifiable evidence.

We learned that:

Natural language is an excellent interface for complex investigations LLMs are useful for interpretation and communication Deterministic systems are essential for evidence-based attribution Large-scale analytical databases like ClickHouse make interactive investigation possible Separating reasoning from evidence improves trust in AI-generated conclusions

The biggest lesson was that a useful AI agent should not just provide an answer—it should have a clear, reproducible process for arriving at that answer.

What's next for RootCause

Our next goal is to move RootCause closer to real production incident response workflows.

Future improvements could include:

Integration with real observability platforms Live streaming telemetry ingestion Support for logs, traces, and metrics together Automated incident timeline generation Confidence scoring for root-cause attribution Multi-incident comparison Slack or PagerDuty integration Human approval workflows for remediation Automated but approval-gated actions such as rollback recommendations or traffic rerouting

Ultimately, we envision RootCause as an intelligent investigation layer that helps engineering teams move from "Something is broken" to "Here is the evidence, here is the likely cause, and here is what to do next."