NeuroScale Autopilot

Architecture
Live monitoring overview — incident detected
The Trust Layer decision card, expanded
alibaba-console-ecs-instance

Inspiration

Every incident-response tool I'd used or built before followed the same pattern: detect a problem, then act on it as fast as possible. Speed was the only metric that mattered. But the more production Kubernetes clusters I worked with as a platform engineer, the more I noticed that speed without judgment is how a small incident becomes a large outage. An agent that restarts the wrong deployment, or rolls back the wrong revision, with full confidence and zero hesitation, is not actually safer than a human who is slow but careful.

That observation became the starting point for NeuroScale Autopilot. Instead of asking "how fast can an agent fix this," I asked a different question: "how does an agent prove, before it acts, that acting is actually the right call?" Track 4 of this hackathon is built around autonomous agents that handle ambiguity and use human-in-the-loop checkpoints, which is exactly the problem worth solving here.

Most autonomous agents are built to act as fast as possible. NeuroScale Autopilot is built to act only when it's safe to.

What it does

NeuroScale Autopilot watches a real Kubernetes cluster. When something breaks, it runs a five-agent pipeline:

Detector: reads live cluster events
Analyzer (powered by Qwen): performs root cause analysis with a confidence and risk score
Planner: uses Qwen's embedding model to retrieve the closest matching runbook from a semantic index
Executor: carries out approved remediations with circuit-breaker protection
Escalation agent: prepares a human-readable summary when a decision needs a person

The part I consider the actual product is what sits between the Planner and the Executor: a Trust Layer gate. Before any remediation runs automatically, the system checks three independent signals at once:

Is the analyzer's confidence high?
Does the retrieved runbook's similarity score clear a 0.65 threshold?
Is the risk level low?

All three have to pass for auto-execution. If even one falls short, the system does not guess: it holds the incident, attaches the exact rollback command it would have used, and waits for a human to approve or reject it. A full breakdown of how this scoring works, with real captured examples, is in TRUST_LAYER.md.

In one incident captured directly from the live deployment, Qwen correctly diagnosed a broken container image tag with high confidence in under three seconds, and even noted that the tag looked like a mistake that had been committed by accident. But the matched runbook only scored 0.59 similarity, just under the threshold, so the system held for approval instead of auto-executing a plausible but unverified fix. That single incident is the clearest demonstration of what this project is actually built to do: a confident answer is not, by itself, permission to act.

The system is deployed on a real Alibaba Cloud ECS instance running a real k3s cluster, with a live dashboard and a FastAPI backend exposing eight Model Context Protocol tools so external AI clients can query cluster status, trigger remediations, and approve pending actions. It is reachable right now at http://43.98.177.117:3000, and full verification steps, including raw API responses anyone can reproduce, are documented in PROOF_OF_DEPLOYMENT.md.

How we built it

The pipeline is Python, using FastAPI for the backend and a small React dashboard for the operator view, connected over a WebSocket for live updates. Each of the five agents is its own module with a narrow responsibility, which made it possible to test them independently: the detector watches Kubernetes events and pod states directly through the Kubernetes API, the analyzer and escalation agents call Qwen through an OpenAI-compatible endpoint, and the planner builds a FAISS vector index over a small library of markdown runbooks so it can retrieve the closest match by cosine similarity at incident time.

For the model layer, three different models handle three different jobs, deployed through Alibaba Cloud Model Studio:

Root cause analysis: Qwen-Max
Runbook retrieval: Qwen's text-embedding-v3
Incident summarization: Qwen-Turbo

Using three different models instead of one large model for everything kept latency low on retrieval and let the reasoning-heavy step get the most capable model available.

For the infrastructure, I provisioned a real ECS instance on Alibaba Cloud in the Singapore region, installed a lightweight k3s cluster on it, and deployed a sample checkout service as the target workload the agent monitors. The entire system, including the dashboard and the API, runs in Docker containers on that same instance. This was a deliberate choice: rather than describe the architecture and hope it's believable, I wanted a URL a judge could open right now and see live incidents flow through the real pipeline.

Challenges we ran into

The single hardest problem was not a modeling problem, it was an account configuration problem. My Alibaba Cloud account initially failed to provision an ECS instance in the mainland China region with a real-name authentication error, which I resolved by switching to the Singapore region instead. Later, even after obtaining a valid Qwen API key, every model call failed with an access-denied error. It took careful debugging to discover the real cause: my API key belonged to one Model Studio workspace, while the models I had activated lived in a different workspace under the same account. Generating a new key from inside the correct workspace, and switching the model names to match that workspace's Token Plan naming convention, fixed it completely.

I also found and fixed a real bug in the detector while preparing the live demo. The Kubernetes events watch had no resourceVersion continuation, so every reconnect (which happens roughly every sixty seconds) would re-list recent events instead of resuming from where it left off. In practice, this meant a single ongoing issue could re-trigger a brand new analyze-plan-escalate pipeline run, including a fresh Qwen API call, several times for the same problem. I fixed this by tracking resourceVersion across reconnects and adding a suppression window keyed by incident type, namespace, and resource, then wrote four regression tests specifically covering that behavior so it cannot silently regress.

The last challenge was resisting the urge to add more. It was tempting to keep expanding: more MCP tools, more chaos scenarios, more agents. I made a deliberate choice partway through to stop adding surface area and instead spend the remaining time proving the one workflow that already existed was real, safe, and honestly measured.

Accomplishments that we're proud of

The entire pipeline runs against a genuinely live Kubernetes cluster on Alibaba Cloud, not a local simulation, and every screenshot in the project's documentation was captured from that live instance, including one where I deliberately created a uniquely timestamped broken image tag during testing specifically so the exact string could be cross-checked against the server logs and the raw API response.

Numbers measured directly from that deployment, not estimated:

Metric	Result
Full pipeline latency, from alert to human-ready decision (real Qwen inference included)	Under 5 seconds
Root cause diagnosis accuracy on test incidents	Correct, high confidence, every run
Remediation plans with an explicit rollback command attached	100 percent, zero exceptions
Behavior when Qwen calls failed during infrastructure testing	Escalated to human instead of guessing, every time
Detector regression tests added for the resourceVersion fix	4, all passing alongside the existing 33
Incidents where a confident diagnosis alone was not enough to auto-execute	Every incident tested so far

I also value that the system behaved correctly even when things went wrong on the infrastructure side: during the workspace misconfiguration described above, with every Qwen call failing, the system still made the correct decision by marking confidence low, risk high, and escalating to a human rather than crashing or guessing. A safety system that only behaves safely when everything upstream is working is not actually a safety system, and this one held up when it mattered.

What we learned

The most useful lesson was that a safety gate is only meaningful if it can independently overrule a confident answer. It would have been easy to build a system where a high-confidence root cause analysis just triggers execution. The retrieval-score check exists specifically to catch the case where the model is confident but the system's own supporting evidence, the runbook match, is weak. Watching that gate actually hold in a real incident, on a real diagnosis that turned out to be correct, was the moment the design felt validated rather than theoretical.

I also relearned a lesson about cloud account configuration that is easy to forget: a valid API key and an activated model are two different states, and they have to be verified in the same workspace. Debugging that mismatch was slower than it should have been, and it is now the first thing documented in the project's proof-of-deployment notes so anyone reproducing this setup does not lose the same amount of time.

What's next for NeuroScale Autopilot

With the Trust Layer proven on a real, live incident, the next step is scale: growing the runbook library across more incident classes (network policy violations, storage pressure, cost anomalies) so the same confidence-plus-retrieval-plus-risk gate generalizes across the full range of things that go wrong in production, not just deployment failures.

NeuroScale's next evolution is closing the loop between human judgment and machine judgment. Right now, an approval or rejection lives in the incident history but is never read back before the next near-identical incident. The next milestone makes human verdicts a first-class retrieval signal alongside runbooks, so the Trust Layer weighs prior operational decisions when evaluating similar incidents. Instead of judging every incident in isolation, the system accumulates organizational judgment over time.

The MCP server already exposes NeuroScale's capabilities to external AI clients today. The next milestone is running the same Trust Layer across a fleet of clusters instead of one, proving the same safety guarantees hold at scale, and integrating directly with ArgoCD so the rollback commands the Planner already generates can execute through existing GitOps workflows instead of raw kubectl.

In one sentence: NeuroScale Autopilot is an autonomous Kubernetes SRE that earns the right to automate by proving, incident by incident, that every remediation is safe before it acts. It is running live on Alibaba Cloud for anyone to verify.