NEXUS — AI Incident Commander

NEXUS is an AI Incident Commander that automates outage triage, investigation, and recovery. Using LangGraph and Splunk, it pinpoints root causes and generates recovery playbooks in seconds.

Comment

Inspiration

Modern operational scale is exponential, but the human capacity to investigate critical outages remains linear. During a major P1 incident, Site Reliability Engineers (SREs) are hit with a deluge of telemetry: thousands of log lines, metric spikes, security events, and active alerts. The first 30 minutes are usually spent trying to correlate data and isolate the root cause, while business losses mount.

We asked ourselves: What if we could deploy an autonomous Incident Command team that works in parallel to investigate, assess, and produce actionable recovery playbooks in seconds instead of hours? This inspired us to build NEXUS, an AI-powered Incident Commander that acts as an automated SRE and Security operations assistant.

What it does

NEXUS connects to your operational data layer (Splunk) and automates the entire incident triage and investigation lifecycle:

  • Ingests Telemetry: Gathers raw service logs, system metrics (CPU, RAM usage), access counts, and security events.
  • Orchestrates Multi-Agent Analysis: Sequentially executes specialized agents—Observability, Security, and Commander—using LangGraph.
  • Correlates Incidents & Threats: Instantly assesses if a technical performance issue (like database connection timeouts) has a security footprint (like a brute-force or DDoS attack).
  • Generates Playbooks: Delivers a structured executive summary, root cause statement, estimated business impact (SLA and revenue), and a step-by-step recovery plan with realistic ETAs and team ownership.

How we built it

NEXUS is designed as a stateful, multi-agent pipeline built on a robust operational data layer:

  • Orchestration: LangGraph (LangChain 0.2) to coordinate a sequential, state-sharing agent pipeline.
  • Backend: FastAPI (Python 3.11) delivering high-throughput simulation endpoints.
  • Frontend: React 18 (Vite + TypeScript + TailwindCSS) featuring a premium, glassmorphism-style dashboard for real-time tracking of the agent workflow.
  • Data Integration: Splunk REST API & Splunk MCP (Model Context Protocol) gateway.
  • Language Models: Groq Cloud (llama-3.3-70b-versatile) and Splunk Hosted Models.

Challenges we ran into

Building a hybrid system connecting local Splunk endpoints with cloud-based LLM APIs brought several challenges:

  • Splunk Auth & Audience Restrictions: Connecting to the local Splunk service REST API frequently threw SSL verification and 401 Unauthorized errors. The model gateway also threw 403 Forbidden (Invalid token audience) due to JWT mismatches. We solved this by developing a custom FallbackLLM wrapper and implementing a hybrid failover architecture in the Splunk client. If credentials fail or Splunk goes offline, the system seamlessly degrades to local databases and backup cloud LLMs.
  • API Deprecations: Mid-hackathon, the primary Groq model we targeted (llama-3.3-70b-specdec) was decommissioned, crashing our backup path. We quickly updated the stack to utilize llama-3.3-70b-versatile to restore complete operational capability.

Accomplishments that we're proud of

  • Zero-Crash Resilience: Built a self-healing LLM and database wrapper that gracefully handles endpoint and token failures, ensuring the application remains fully functional even in offline/hybrid modes.
  • Stateful Agent Handshakes: Successfully used LangGraph to model the exact sequential "handshake" between Observability (technical) and Security (malicious assessment) contexts.
  • Visual Dashboard: Designed an interactive, modern user interface that clearly visualizes the transition of each agent from standby, to running, to complete, along with their findings.

What we learned

  • Graceful Degradation is Non-Negotiable: When building AI integrations for operations, the tool itself cannot become a single point of failure. Designing self-healing wrappers like our LLM fallback layer is critical for real-world resilience.
  • The Power of Stateful Orchestration: Using LangGraph to pass a typed state dict showed us how easily complex, multi-stage human workflows can be mapped into reliable agent chains.
  • Securing Operational Context: Structuring prompts to prevent LLM hallucination during outages is essential. Relying strictly on structured JSON schemas ensured the commander's playbooks remained concise and actionable.

What's next for NEXUS — AI Incident Commander

  • Active Remediation (Looping Actions): Move from read-only command to read-write remediation—allowing agents to run safe, pre-approved commands (like restarting a service, flushing a cache, or blocking an IP on a firewall) directly through Splunk actions.
  • Collaborative Human-in-the-Loop: Implement interactive checkpoints where SREs can approve, reject, or modify steps in the recovery playbook before they are executed.
  • Multi-Data-Source Ingestion: Extend the data client to pull from Datadog, Prometheus, and AWS CloudWatch, unifying all operations metrics under a single NEXUS commander.

Built With

  • axios
  • css3-frameworks:-react-18
  • fastapi
  • groq
  • html5
  • langchain
  • langchain-0.2-data-and-observability:-splunk-enterprise-(rest-api)
  • langgraph
  • langgraph-0.1
  • pydantic
  • python
  • react
  • rest-api-languages:-python-3.11
  • splunk
  • splunk-hosted-models-development-and-build-tools:-vite
  • splunk-model-context-protocol-(mcp)-apis-and-llms:-groq-api-(llama-3.3-70b-versatile)
  • tailwindcss
  • typescript
  • uvicorn
  • vite
Share this project:

Updates