Architecture: watcher, Laya decision engine, RCA agent and Hindsight memory, with Nemotron tiers on Token Factory.
Incident dashboard: two resolved incidents, 2 fixes stored in Hindsight memory.
Repeat incident routed to Nemotron Nano via Nebius Token Factory; verified fix saved to Hindsight memory.

Inspiration

Every on-call engineer knows the loop: the same incident returns a month later and the team investigates from scratch, because nobody remembers what fixed it last time. Teams that are cutting observability and on-call costs feel this most, and most AI SRE tools also start from zero on every incident. We wanted an agent that remembers, so repeat incidents are cheaper and faster.

What it does

Incident Response Agent is an SRE copilot that watches a microservice stack, opens incidents on its own, finds the root cause, and learns from every fix.

Proactive Watcher detects an SLO breach (for example a 100% error rate on POST /checkout) and opens an incident.
Laya decision engine reads each chat message and decides whether it is the same incident or a new one, and whether the user wants an RCA or a query.
RCA agent recalls past fixes from Hindsight episodic memory (fix:* and failed:* tags), diagnoses, and proposes an action.
Dynamic NVIDIA Nemotron tiers: a new incident uses the largest model (Ultra), a context shift uses Super, and a repeat of a known fix uses the small Nano model. Each choice is recorded as a diagnosis:tier event on the incident timeline.
You mark Fix Worked / Fix Failed, and the outcome is stored for next time.

How we built it

Models: NVIDIA Nemotron (Nano / Super / Ultra) served through Nebius Token Factory (OpenAI-compatible API).
Agents: Google ADK, with LiteLLM as the provider layer (Token Factory, NVIDIA NIM, optional Groq fallback).
Memory: Hindsight for episodic memory. ChromaDB and BM25 for runbook RAG. SQLite for incident storage.
Observability and test bed: OpenTelemetry, Prometheus, Loki, Tempo and Grafana, with the Astronomy Shop microservices and a runtime fault-injection deck.
Hosting: , Docker Compose, Caddy TLS, cost guards (per-hour and per-day caps with a labelled replay fallback).

Benchmark:

Challenges we ran into

Deciding when two chat messages belong to the same incident. Treating a new problem as an old one gives wrong fixes, which is why Laya has a same/different context decision.
Keeping memory honest: we store failed attempts too, so the agent does not repeat a fix that already failed.
Cost control for a public demo, so one visitor can't burn the token budget.
Super and Ultra currently map to the same model in our default config.

Accomplishments that we're proud of

A repeat incident is handled by the small Nano model instead of the full investigation, and you can see the tier decision on the incident timeline.
A hands-on demo with live fault injection that judges can try themselves.
A multi-agent system with tests and CI built mostly in the Submission Period.

What we learned

Memory changes the economics of an AI agent more than a bigger model does.

What's next for Incident Response Agent: an SRE copilot that remembers

Watch more than the checkout service, connect to real observability backends (Prometheus, OpenObserve, Datadog) and ticketing tools, add approval-gated auto-remediation, and share anonymised fix memory across teams.

Pre-existing work and what changed

The project started before the Submission Period as a prototype from the Cloud Native Hackathon in Pune. The agent architecture, Hindsight memory, Nemotron tiering, Token Factory provider, hosted-demo safeguards, benchmark harness and tests were built during the Submission Period. Details are in docs/WHAT_CHANGED.md in the repo.