Inspiration

I've spent years operating telecom and 5G core networks on Kubernetes, and the 3 a.m. page always goes the same way. Read the alert, check the pods, pull logs from the crashed container, look at what changed, and then decide whether a rollback is safe. Most of that is mechanical, and most alerts aren't worth waking up for. I wanted an agent that does the investigation the way a good SRE would, while a human keeps the final say on any change to production. I also wanted it cheap enough to run on every alert

What it does

Takes alerts straight from Alertmanager and opens an incident for each one.
Triages with Nemotron Nano. Noise is dismissed in about 2 seconds without calling a larger model.
Investigates with Nemotron Super using read-only tools: pod status, current and previous container logs, events, Prometheus metrics, and the deployment's revision history.
Reviews with Nemotron Ultra whenever Super is unsure, the incident spans components, or Super proposes a production change. Ultra confirms or corrects the root cause and the fix.
Waits for one-click human approval, then applies one of four allowed actions: restart, roll back, scale up to 10 replicas, or restore a ConfigMap.
Verifies recovery by watching the rollout until every replica is available. Only then does it mark the incident remediated.
Writes a blameless postmortem with Ultra, packaged as a Nebius Serverless Job, and links it from the incident.
In the OOM demo, the fix was ready about 10 seconds after the incident opened. The agent named the exact change: the memory limit was cut from 128Mi in the latest revision. ## How we built it
Agent: Python and FastAPI, with the OpenAI SDK pointed at Nebius Token Factory. A tiered router sends each call to Nemotron Nano, Super or Ultra.
Tools: the Kubernetes Python client and the Prometheus HTTP API, exposed to Super through native tool calling.
Cluster: k3d (Kubernetes 1.35) on a Windows laptop through WSL2, running Google's Online Boutique, an nginx edge proxy and kube-prometheus-stack with custom alert rules.
Faults: an OOM limit cut, a missing image tag and a broken nginx config, injected by script and reset from known-good manifests.
Safety: least-privilege RBAC scoped to one namespace, an allowlist of four write actions, a shared token on every state-changing call, and human approval by default.
Dashboard: a single-page incident timeline, colored by model tier, with per-tier call, token and latency counts, plus separate time-to-fix and time-to-recover figures

Challenges we ran into

An alert that would never fire. During a stuck rollout the old pod keeps serving traffic, so "replicas unavailable" never triggers for a bad image. I added an image-pull-failure rule.
A reset that undid the fix. kubectl rollout undo after the agent's own rollback reapplied the broken revision. Reset now re-applies the known-good manifests.
Reasoning models and JSON. Smaller models sometimes wrote out their reasoning before the JSON answer. I strip blocks and turn Nano's reasoning off through its prompt.
Measuring the right thing. The first timer counted the time a fix sat waiting for my click. I split it into time to diagnosis and time to recover after approval. 5 . Secrets hygiene. I committed .env once early on. I rotated the secret, deleted and recreated the repo, and put .gitignore and a staged-secrets check into the setup steps.
Serverless access. Token Factory worked from day one, but creating jobs in myAI Cloud project returned PermissionDenied. The postmortem job is packaged and its command is documented; for the demo I ran the same container locally. ## Accomplishments that we're proud of
The diagnosis is correct and specific, naming the revision and the limit that changed, on a real cluster with real alerts.
Ultra's review earns its place: in testing it corrected a proposed restart to the safer rollback.
The agent checks its own work by verifying recovery instead of assuming the fix worked.
The cost structure is visible on screen: most calls stay on Nano and Super

What we learned

Small models are strong at triage and tool use when the prompt narrows their job. The big model belongs at the decision point, not in the loop.
Safety design matters as much as model quality: an allowlist, namespace scope and human approval make it acceptable to point an agent at production.
The order of the investigation matters. Checking recent changes early finds most root causes.

What's next for Self-Healing SRE Agent

Run the agent itself on a Nebius Serverless Endpoint against a managed cluster, and the postmortem job on Nebius Serverless. Remember resolved incidents, so a repeat of a known pattern is fixed by Nano alone. Deduplicate related alerts into one incident. Apply the same pattern to 5G core network functions: eBPF signals in, self-healing actions out.

Built with: python, fastapi, nvidia-nemotron, nebius-token-factory, nebius-serverless, kubernetes, k3d, prometheus, alertmanager, docker, openai-sdk, cloudflare-tunnel, wsl2 Try it out: https://github.com/spodala0391/sre-agent