SentinelOps

Autonomous Site Reliability Engineering Agent

Comment

Inspiration

When a service goes down, engineers usually jump between logs, metrics, and dashboards to figure out what happened. This manual context-gathering is highly stressful and drastically increases Mean Time to Resolution (MTTR). I wanted to solve this real-world problem by building an autonomous system that handles the first layer of investigation automatically, allowing engineers to focus on fixing the problem rather than finding it.

What it does

SentinelOps AI is an autonomous Site Reliability Engineering (SRE) assistant. When an incident occurs, it:

  1. Triages the alert by performing a semantic search against similar past incidents from its vector memory.
  2. Diagnoses the issue autonomously by deciding which infrastructure tools to run (e.g., fetching pod logs or checking Kubernetes events).
  3. Resolves the problem by synthesising the findings into a clear diagnosis and a suggested remediation plan. A human operator can then review and execute the plan through a built-in approval gate.

How we built it

The project was built with a robust, modern tech stack:

  • Qwen Cloud: Powered the core reasoning of our autonomous agents, enabling the system to analyse complex logs and accurately deduce root causes.
  • LangChain & LangGraph: Used to orchestrate the multi-agent workflow (Triage Agent, Diagnostic Agent, Resolution Agent).
  • FastAPI: Served as the high-performance backend routing agent communications and tool executions.
  • Gradio: Provided the interactive, enterprise-grade front-end interface.
  • PostgreSQL (pgvector): Handled memory and vector embeddings for semantic search of historical incidents.
  • Docker: Ensured reproducible deployments and isolated environments.

Challenges we ran into

One of the main challenges was giving the agent the ability to execute infrastructure commands safely. If an AI hallucinates a command, it could take down a cluster. We solved this by implementing a strict "Human-in-the-Loop" approval gate. The agent proposes the exact remediation command, but it is physically unable to run it until the operator explicitly approves and triggers it, ensuring maximum safety for production environments.

What's next for SentinelOps AI

In the future, I plan to integrate SentinelOps with PagerDuty, Datadog, and Slack. This will allow the agent to automatically reply to incident Slack threads with its initial diagnosis before the on-call engineer even opens their laptop!

Built With

Share this project:

Updates