OpsPilot

Inspiration I spent two years at Bosch as the primary escalation point for 198,000+ users across 20+ Jira instances and 3 Confluence Data Center clusters. I personally drove root cause analysis on 45+ critical production incidents. I know exactly what it feels like to get paged at 2am, open Splunk half-asleep, cross-reference Grafana dashboards, and try to make a sound remediation decision with your brain running at 40%.

The frustrating part was never the complexity. It was the repetition.

The same database connection pool exhaustion. The same JVM heap spike under load. The same misconfigured load balancer. I had seen every one of these before — and yet every time, we started from scratch. Alert fires. Engineer wakes up. Investigates. Remediates. Writes post-mortem. Closes ticket. Repeat.

That pattern is what OpsPilot is built to break.

Not a research project. Not a demo built around a hypothetical problem. A direct answer to a failure mode I lived through hundreds of times — and one that every engineering team on the planet experiences every week.

What It Does OpsPilot is an autonomous SRE agent powered by Qwen Cloud. It monitors production incidents, diagnoses root causes using AI reasoning, plans remediation with risk scoring, and generates post-mortem reports — with a human-in-the-loop approval gate for any action that cannot be undone.

The full lifecycle it owns:

Detect Accepts an alert, log snippet, and key metrics from any monitoring tool — Grafana, PagerDuty, Splunk, CloudWatch, or a manual paste.
Diagnose Sends structured context to qwen-plus on Qwen Cloud. The agent returns a root cause, a confidence score, and a ranked remediation plan. All in JSON. No freeform text that needs another parse step.
Plan and Execute Each remediation step carries a risk classification: low, medium, or high.

Low-risk steps execute automatically. No human needed. Medium and high-risk steps enter a human approval queue in the dashboard. High-risk and irreversible steps show an explicit warning before the engineer decides.

Report After resolution, a second Qwen call generates a structured post-mortem: what happened, root cause, remediation timeline, and prevention recommendations. What normally takes an engineer 45 minutes to write is done in under 10 seconds.

How We Built It The stack is deliberately production-grade, not demo-grade. AI Layer — Qwen Cloud All inference runs on Alibaba Cloud infrastructure via the Qwen compatible-mode endpoint: python: client = OpenAI( api_key=os.getenv("DASHSCOPE_API_KEY"), base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1", ) qwen-plus for diagnosis and remediation planning Structured JSON output enforced via prompt constraints enable_thinking: False to suppress chain-of-thought and save 300-500 tokens per call max_tokens: 500 hard cap on each inference call Total token cost per full incident workflow: approximately $900$ to $1200$ tokens.

Backend — Python and FastAPI

Single app.py file. Clean REST API with 7 endpoints. No unnecessary abstractions.

The diagnosis logic follows this flow:

alert + logs + metrics | _run_diagnosis() | call_qwen() --> JSON parse --> risk classification | auto_execute low-risk steps | hold medium/high for human approval Frontend — Pure HTML, CSS, JavaScript

No React. No npm. No build step. One HTML file. The entire dashboard — incident list, diagnosis view, remediation plan, approval buttons, post-mortem panel — runs from static/index.html. State is managed in a plain JavaScript object.

This was a deliberate choice. Production tools should be deployable anywhere without a build pipeline.

Challenges We Faced Getting the agent to say "I do not know" gracefully

The hardest problem was not getting Qwen to diagnose incidents. It was getting it to refuse to act when the signal was too weak. Early versions were overconfident. We introduced explicit confidence thresholds and structured the prompt to require the agent to quantify uncertainty before any remediation step was generated.

Risk scoring without ground truth

Defining what is "low risk" versus "high risk" for a general-purpose SRE agent is genuinely hard. A pod restart is low risk in most systems and catastrophic in a stateful database cluster. The agent classifies each step against this framework. The human approval gate is the final safety layer when classification confidence is insufficient.

JSON reliability from the model

Qwen occasionally wraps JSON in markdown code fences or adds explanatory text before the opening brace. We built a dedicated parser that strips fences, handles json prefixes, and falls back gracefully if the parse fails — returning a safe error state rather than crashing the agent loop.

Keeping the human loop meaningful

A human-in-the-loop that nobody actually checks is not a safeguard. It is theater. We designed the approval UI so that an engineer can understand the agent's reasoning in under 30 seconds: risk level, reversibility, exact action, and a plain-English summary. The goal was that an on-call engineer at 3am, half-awake, can make a good decision in one read.

What We Learned The quality of your tool definitions matters more than the quality of your prompts.

We experimented with elaborate prompt engineering but the biggest reliability gains came from making the JSON schema in the prompt precise and explicit — typed fields, clear constraints on allowed values, no ambiguity in the structure. Once the schema was tight, the model outputs became consistent enough to drive a real UI.

We also learned that post-mortems are the best training data. Every human-written post-mortem we used to calibrate our prompts measurably improved the quality of future agent-generated reports. The system gets more useful the more it is run — which is exactly what you want from a production SRE tool.

What Is Next Webhook endpoint to receive alerts directly from Grafana, PagerDuty, and Prometheus Alertmanager — making the ingestion fully automatic Vector memory store using Alibaba Cloud VectorDB so the agent retrieves similar past incidents at diagnosis time PII sanitizer middleware that scrubs customer data from logs before any Qwen API call Jira Service Management integration to auto-create incident tickets and write post-mortems back to Confluence Slack integration so engineers can approve or escalate remediation steps from a thread without opening the dashboard Multi-model routing — use qwen-plus for standard diagnosis and escalate to qwen-max for complex multi-service incidents