-
-
a thumbnail showing the project summary
Inspiration
AIOps Autopilot
An autonomous incident-response agent that diagnoses, fixes, and verifies production failures autonomously, and escalates to a human when it cannot safely act, so it never leaves production broken. Built on Qwen Cloud.
Inspiration
The promise of an agent that auto-fixes production infrastructure is an irresistible demo and a terrifying production idea. The hard part of AIOps is not getting a model to suggest a fix. It is trust: knowing the agent will not confidently apply the wrong change and walk away while the system burns. The goal was an incident autopilot fit to actually run in production, where the defining skill is knowing when not to act.
What it does
AIOps Autopilot takes an ambiguous incident (alert + logs + metrics) and runs it through a six-stage pipeline:
ingest → triage & root-cause → remediation planning → human-in-the-loop gate → execution → verification, recording each outcome to an experience store.
It identifies the root cause, drafts a risk-scored remediation, and then either applies it and verifies the system actually recovered, or, when confidence is low or no safe fix exists, escalates to a human instead of guessing. Every action runs against a sandboxed environment, and any fix that fails verification is automatically rolled back. The result is an agent that fixes what it safely can, contains mistakes through rollback, and hands off the rest.
How it was built
- Models: Qwen 3.7-max (reasoning and planning) and Qwen 3.7-plus, served via Qwen Cloud (Alibaba Cloud Model Studio / DashScope), accessed through the OpenAI-compatible API.
- Pipeline and API: Python with FastAPI, and Pydantic schemas validating every model output with bounded strict-JSON retries.
- Tools via MCP: three Model Context Protocol servers: telemetry (logs and metrics), infra/ops (restart, scale, config, rollback), and knowledge (runbook and past-incident retrieval).
- Safety by construction: action targets are a closed enum injected and validated server-side, never chosen by the model, so the agent cannot address an action outside the sandbox. Only the executor is granted mutating tools, the gate guards destructive actions, and failures auto-roll-back. Result: 0 out-of-sandbox tool calls in the benchmark.
- Evaluation: a fault-injection harness that injects faults with known ground truth, runs both the pipeline and a single-prompt baseline over the same incidents, and verifies remediation by re-checking the live system rather than trusting the model's word. Ground truth never reaches the agent's context.
- Deployment: a containerized backend on Alibaba Cloud ECS, with a live Qwen Cloud self-check that reports host, region, model tiering, tokens, and cost.
Challenges
The honest version, because it is the most interesting part.
The benchmark held the architecture to a fair bar: the single-prompt baseline ran on the same top-tier model with the same retry discipline, so the comparison isolated architecture, not model choice. The mock benchmark predicted a decisive win. Then the first real-model run said the opposite: the single prompt beat the pipeline on remediation, 62.5% versus 25%.
Two discoveries fixed it. First, the loss had briefly stopped reproducing because a measurement patch had masked the bug rather than fixing it, so reverting that patch reproduced the failure exactly. Second, the traces localized the problem precisely. Diagnosis was excellent, but the planner was the bottleneck: running on a cheaper tier, missing an explicit escalate / no-safe-action option (so it fabricated a benign action on unfixable faults), with an action vocabulary that could not express some runbook fixes, and a gate keyed on the wrong confidence signal. The fix addressed the genuine defects: promote the planner to the reasoning tier, add the escalate path, align the vocabulary, and repair the gate. Then one fresh validation, with no tuning to the results.
Lessons learned
- A single frontier-model prompt is a brutally strong baseline. Elaborate orchestration earns the extra cost only on incidents that are unsafe to act on blindly or that need multiple steps. On easy, single-action faults, one good prompt ties it and runs about 5x cheaper, and the report states that plainly.
- For an autonomous remediation system, raw resolution rate is the wrong headline. The question that matters more is whether the system was left broken, which is what the residual-damage metric captures.
- Mock evaluations measure information access, not model intelligence, so the real run is the only verdict.
- Cost-driven model tiering can quietly become the accuracy bottleneck.
Accomplishments (measured on Alibaba Cloud, real models)
After the fix, on an 8-fault suite versus the fair single-prompt baseline:
- Diagnosis: 100% vs 50% root-cause accuracy.
- Remediation: 75% vs 62.5% correct (sandbox-verified).
- Safe outcomes: 87.5% vs 62.5% (resolved, or correctly escalated when unfixable).
- Residual damage: 0% vs 37.5%. The baseline left wrong changes applied to production; the pipeline left none.
- Cost, disclosed: the pipeline runs about 4x the baseline's tokens, and the entire real benchmark cost roughly $0.35, all on free-tier quota.
- A context-summarization design that cut tokens about 44% at real scale.
The durable, honest framing of the value: superior diagnosis, correct human escalation on unfixable faults, multi-step remediation a single prompt structurally cannot do, and zero residual damage, at a known and disclosed cost.
What's next
A reasoning-tier improvement to triage hypothesis ranking (the one fault that still mis-ranks), a richer action vocabulary, more fault classes, and exercising the destructive-action gate path end to end on the live stack.
Log in or sign up for Devpost to join the conversation.