ARIA - Autonomous Reasoning & Incident Agent

The on-call colleague who never sleeps. ARIA reasons across infrastructure (Dynatrace) and AI quality (Arize) via Gemini to find incidents that live in the gap between them.

Comment

The 3am problem nobody talks about

When you ship AI to production, you end up with two monitoring systems that never talk to each other. One watches your servers - CPU, memory, pods, latency. The other watches your model - hallucination rate, eval scores, token usage. They live in different tools, owned by different teams.

The worst incidents live in the gap between them.

A memory leak on a pod quietly forces the prompt-assembly buffer to evict context. The model starts receiving truncated prompts. With half its context gone, it hallucinates. The infra team sees "memory high." The ML team sees "model degraded." Neither sees the causal chain connecting them because no human is watching both layers at once.

By the time two on-call engineers get on a call and connect the dots, it's already been broken for twenty minutes, and user trust is spent.

ARIA closes that gap. It's one autonomous agent that reasons across both observability layers at the same time, finds the cross-layer causal chain, and resolves it, keeping a human in control of anything risky.

What it does

When an anomaly fires, ARIA:

  1. Pulls live context from both systems - Dynatrace (infrastructure) and Arize Phoenix (model quality), via their MCP servers
  2. Correlates the two signal sets with Gemini, hunting specifically for causal chains that cross the infra/model boundary
  3. Names a single root cause with a confidence score and an explicit causal chain
  4. Proposes the minimum set of fixes - running the safe ones automatically, pausing for a human click on anything that changes model behavior or has blast radius
  5. Writes the incident report, timeline, root cause, actions taken, prevention and closes the loop

The defining moment: ARIA connects memory pressure (Dynatrace) → prompt truncation → hallucination rate climbing (Arize) in seconds. No human would have caught that link that fast.

How we built it

  • Brain: Gemini, orchestrated with Google's Agent Development Kit (ADK) as a three-agent pipeline - a Planner that breaks the investigation into steps, a Reasoner that correlates across both systems, and an Executor that classifies each fix as auto or needs-approval.
  • The key architectural decision: the Reasoner agent carries both partner MCP toolsets at once. A single Gemini reasoning loop can call a Dynatrace tool and an Arize tool in the same turn, which is the only way cross-boundary correlation actually works. Hand the model two siloed agents and it reproduces the exact human blind spot we're trying to kill.
  • Dynatrace MCP for infrastructure: execute_dql, list_problems, get_kubernetes_events, find_entity_by_name.
  • Arize Phoenix MCP for model quality: list-traces, get-spans, get-span-annotations. We seeded a real Phoenix instance with 40 LLM traces (healthy + degraded) and ARIA computes hallucination rate, relevance, and token counts from the live spans.
  • Backend: Python + FastAPI, streaming the reasoning chain to the dashboard over Server-Sent Events with a real human-in-the-loop approval gate (a risky action blocks on an async event until you click approve).
  • Frontend: Next.js + Tailwind ops dashboard - dual signal panels, a live typewriter reasoning chain, a confidence gauge, action cards, and calibration/autonomy controls.
  • Deployed on Render (API) and Vercel (UI).

Challenges we ran into

  • Designing incidents whose root cause is genuinely invisible to either system alone - that's what proves the cross-layer thesis instead of just asserting it.
  • Streaming a reasoning chain that reads naturally on screen while keeping a real approval gate that actually blocks execution until a human decides.
  • Making it bulletproof for a demo without faking it, so we built a dual-mode architecture: live MCP → real Gemini → graceful fallback, so the product never shows a stack trace, and the reasoning is always real.

What we learned

The biggest insight wasn't technical, it was that giving one agent two lenses simultaneously is fundamentally different from running two agents. The correlation only emerges when a single reasoning context holds both signal sets. That's the whole product in one sentence.

What's next

  • Stream Gemini's token-level reasoning directly into the chain
  • Learn remediation playbooks from resolved incidents
  • Expand the cross-layer correlation library (cost spikes, data drift, retrieval failures)
  • Wire execute_action to real remediation tools behind the approval gate

Built With

Share this project:

Updates