AI Incident Triage & Root-Cause Assistant

• 

  This is what the UI looks like.

• 

  analysis - 1

• 

  analysis -2

Inspiration

Production incidents are expensive, stressful, and often slow to diagnose because engineers have to jump between logs, alerts, dashboards, deployment history, and old incident reports. I wanted to build something that could bring all of that context together and help an on-call engineer reach the likely root cause faster without replacing human judgment.

What it does

My AI Incident Triage & Root-Cause Assistant ingests alerts, logs, stack traces, deployment metadata, and service context to help investigate production failures. It classifies the incident, identifies likely affected services, retrieves similar historical incidents, summarizes supporting evidence, and suggests the next debugging steps. It can also generate a concise incident timeline for use during response and postmortems.

How I built it

I built the backend with FastAPI and used a transformer-based model for incident classification and root-cause analysis. I fine-tuned the model using LoRA on incident-style examples containing telemetry, symptoms, and verified causes. I also added a RAG layer over historical incidents using vector embeddings so the system could retrieve similar failures and ground its recommendations in prior evidence.

I built a React dashboard to display incident context, likely causes, supporting logs, and recommended actions. I containerized the application with Docker and designed the architecture so it could integrate with common logging and monitoring tools.

Challenges I ran into

One of the biggest challenges was separating useful signals from noisy logs. An incident can generate thousands of log lines, but only a small portion may be relevant to the actual failure. I also had to reduce the risk of the model making confident but unsupported guesses, so I added evidence requirements and retrieval-based grounding.

Another challenge was creating realistic training data. Real production incident datasets are difficult to obtain, so I used synthetic microservice failures, generated telemetry, and structured incident scenarios for training and evaluation.

Accomplishments that I’m proud of

I’m proud that the project goes beyond simply connecting an LLM to an API. I combined model fine-tuning, retrieval, backend infrastructure, observability, and a usable frontend into one end-to-end system.

I also designed the assistant so that its root-cause recommendations are connected to supporting evidence rather than presented as unexplained model predictions.

What I learned

I learned that model quality alone is not enough to build a useful production AI system. Data quality, retrieval, context selection, evaluation, and guardrails can be just as important as the model itself.

I also learned the importance of measuring AI systems with task-specific metrics such as root-cause accuracy, retrieval quality, grounding, hallucination rate, and latency instead of simply judging whether an answer sounds convincing.

What’s next for AI Incident Triage & Root-Cause Assistant

Next, I want to connect the system to live observability tools such as Datadog, CloudWatch, and Kubernetes. I also want to add continuous learning from resolved incidents, improve root-cause ranking across multiple services, and introduce human feedback so engineers can confirm or reject recommendations.

Longer term, I want the assistant to safely automate low-risk diagnostic actions while keeping any production-changing action behind explicit human approval.

Built With

• amazon-web-services
• cloudwatch
• docker
• faiss
• fastapi
• hugging-face-transformers
• kubernetes
• llama
• lora
• pgvector
• postgresql
• python
• rag
• react
• rest-apis
• typescript