IncidentPilot

IncidentPilot turns noisy production crashes into instant root-cause analysis , built to solve our own late-night on-call headaches from reading logs, cutting troubleshooting time from hour to second

Comment

What it does

IncidentPilot is an AI-powered incident investigation platform designed to help software engineers, DevOps teams, and Site Reliability Engineers (SREs) diagnose production failures faster.

Instead of asking an AI to read thousands of log lines, IncidentPilot first performs deterministic analysis by parsing structured logs, correlating request IDs and correlation IDs, reconstructing incident timelines, extracting stack traces, and identifying affected services. Only the relevant evidence is then sent to an OpenAI-compatible language model, which generates an executive summary, root cause analysis, chain of evidence, confidence score, suggested fixes, recommended next steps, and an incident postmortem.

To demonstrate these capabilities in a realistic and reproducible way, the project includes a deliberately faulty FastAPI e-commerce application that continuously generates production-like traffic and multiple intentionally seeded failures.

How we built it

IncidentPilot consists of two complementary applications.

The first is a FastAPI-based e-commerce backend that acts as a self-contained demonstration environment. It includes common production modules such as authentication, products, checkout, orders, and payments while intentionally introducing realistic failures including malformed payloads, authentication issues, database timeouts, missing configuration, and service outages. Structured logging captures timestamps, request IDs, correlation IDs, latency, stack traces, and other operational metadata.

The second is the IncidentPilot dashboard, built with React, TypeScript, Vite, Recharts, and plain CSS. The dashboard continuously parses structured logs, detects incidents, reconstructs timelines, visualizes evidence, and allows users to investigate incidents interactively.

The AI layer uses the official OpenAI SDK and supports any OpenAI-compatible provider through environment variables, allowing the project to work with both OpenAI models and compatible providers such as Fireworks AI without changing application code.

Throughout development, GPT-5.6 helped refine the overall product architecture, investigation workflow, and implementation strategy, while Codex accelerated implementation by generating backend services, API routes, logging utilities, reusable frontend components, and helping refactor the project into a modular architecture.

Challenges we ran into

One of the biggest design challenges was deciding how much work should be performed by deterministic software versus the language model. Sending an entire production log to an LLM is expensive and often introduces unnecessary context, so I designed a pipeline that extracts only the evidence relevant to a specific incident before invoking AI.

Another challenge was building a demonstration environment that felt realistic without relying on external production infrastructure. Creating a deliberately faulty e-commerce application allowed IncidentPilot to consistently generate reproducible incidents while showcasing the investigation workflow in a controlled environment.

Finally, balancing engineering depth with usability required careful attention. The goal was to create a dashboard that feels like a professional observability tool rather than simply wrapping a chatbot around a log viewer.

Accomplishments that we're proud of

  • Built a complete end-to-end incident investigation platform rather than a simple AI log summarizer.
  • Designed a deterministic investigation pipeline that reconstructs incidents before involving the language model.
  • Created a realistic demo environment capable of generating reproducible production failures.
  • Implemented structured logging, request correlation, evidence reconstruction, and interactive incident visualization.
  • Built the AI layer using the official OpenAI SDK with support for any OpenAI-compatible provider through environment variables.
  • Delivered a modular architecture that cleanly separates log generation, incident detection, deterministic analysis, AI reasoning, and the frontend dashboard.

What we learned

This project reinforced that AI is most effective when combined with deterministic software engineering rather than replacing it. Traditional software techniques such as parsing, correlation, and timeline reconstruction significantly improve the quality and efficiency of AI-generated explanations.

We also learned the importance of designing AI systems around structured evidence instead of raw data. By reducing the amount of information sent to the language model, the application becomes more explainable, more efficient, and easier to extend. Finally, the project demonstrated how Codex and GPT-5.6 can accelerate development while leaving architectural decisions, engineering trade-offs, and product direction in human hands.

What's next for the project

The next step is to evolve IncidentPilot beyond the built-in demonstration environment into a platform capable of analyzing real-world production systems.

Planned improvements include support for uploading arbitrary log files, additional log format parsers, integrations with observability platforms such as Kubernetes, CloudWatch, and Datadog, real-time alerting through Slack and PagerDuty, collaborative investigations for engineering teams, and AI-assisted pull requests that automatically generate proposed fixes after identifying a root cause.

Built With

Share this project:

Updates