Inspiration
Every production AI agent we've worked with shares the same fatal flaw: it's blind to its own behavior. When it makes a wrong decision — a hallucinated schema patch, a missed quarantine, a runaway remediation loop — there is no feedback mechanism that lets the agent see what it just did and why it failed. Teams are left scraping logs in postmortems, manually rewriting prompts, and hoping the next incident goes better.
We kept asking the same question: what if the agent could read its own mind?
Arize Phoenix answered that. The moment we saw that Phoenix exposes an MCP server — meaning an agent can call Phoenix as a tool at runtime — we knew exactly what to build. Not just an agent that gets traced. An agent that reads its own traces, evaluates its own reasoning, and rewrites its own behavior between incidents. That's SENTINEL.
What it does
SENTINEL is an autonomous SRE healing agent that responds to production incidents through a 6-stage remediation pipeline:
| Stage | Tool | Action |
|---|---|---|
| 1 | Dynatrace | Pod topology scan, incident triage |
| 2 | Elastic | Error log search, root cause isolation |
| 3 | GitLab | git blame, hotfix MR creation |
| 4 | MongoDB | Schema patch (collMod) with human approval gate |
| 5 | Fivetran | Downstream connector resync |
| 6 | Arize Phoenix | Full trace ingestion + LLM-as-a-Judge self-eval |
But the real story is what happens at Stage 6 and beyond:
- Every tool call, every LLM step, every decision is captured as an OpenInference span in Phoenix
- Before patching anything, SENTINEL calls
search_spans()to find how similar past incidents were resolved - It calls
run_evaluator()to score its current reasoning against historical quality benchmarks - If its average eval score drops below threshold,
self_improve_prompt()generates an improved system prompt grounded in the agent's own failing trace patterns — no offline retraining, no human prompt engineer needed - High-quality traces are promoted to a
sentinel_golden_tracesPhoenix dataset, becoming the ground truth for future runs
The confidence engine gates every stage behind a deterministic risk score:
\[ \text{confidence} = 0.45 \cdot \text{success_rate} + 0.30 \cdot (1 - \text{drift}) + 0.15 \cdot \text{scale}(n) + 0.10 \cdot \mathbb{1}[\text{has_precedent}] \]
If confidence falls below \(0.40\) (CRITICAL tier), SENTINEL refuses to act and escalates — proving the safety guardrails work and showing judges the one thing no simple automation can do: knowing when not to act.
All of this is visible in real time in the SRE Control Cockpit — a live WebSocket-connected dashboard showing the pipeline, confidence engine, Phoenix trace waterfall, multi-run learning heatmap, and the self-improvement before/after diff.
How we built it
Runtime: Google ADK LlmAgent with Gemini 2.0 Flash as the reasoning core. ADK's tool-calling interface lets us bind each of the 6 pipeline stages as discrete tools with structured inputs and outputs.
Observability: OpenInference auto-instrumentation wraps every ADK tool call and LLM invocation, emitting spans to Arize Phoenix Cloud via OTLP/HTTPS. Zero manual span creation needed.
The MCP loop: We run npx @arizeai/phoenix-mcp as a sidecar and expose its tools — list_recent_traces, search_spans, run_evaluator, get_drift_signals, save_to_dataset — directly to the Gemini agent. This is the core innovation: Phoenix becomes the agent's long-term memory and self-evaluation oracle.
Confidence Engine: A pure-Python, LLM-free deterministic scorer in agent/confidence.py that produces a per-stage risk tier and recommended action. The four tiers map to score ranges:
\[ \text{tier} = \begin{cases} \text{LOW} & \text{if } s \geq 0.85 \ \text{MEDIUM} & \text{if } 0.65 \leq s < 0.85 \ \text{HIGH} & \text{if } 0.40 \leq s < 0.65 \ \text{CRITICAL} & \text{if } s < 0.40 \end{cases} \]
UI: A single-file SRE Control Cockpit (ui/index.html) served by FastAPI, connected to the backend via WebSocket for real-time event streaming. Falls back gracefully to a local demo simulation when the backend is offline.
Safety: Google Cloud Model Armor (agent/model_armor.py) screens every LLM output for prompt injection and policy violations before any tool call is executed.
Challenges we ran into
Making the self-improvement loop actually close was the hardest problem. It's easy to save traces to Phoenix. It's much harder to make the agent use those traces to change its behavior in the next run without a full retraining cycle. We solved this by having self_improve_prompt() parse failing trace patterns directly from Phoenix evals and inject targeted addenda into the system prompt at runtime.
Confidence calibration took far more iteration than expected. The weighted formula went through 11 versions before the tier boundaries felt right. The weights \(w_1 = 0.45,\ w_2 = 0.30,\ w_3 = 0.15,\ w_4 = 0.10\) were chosen so that:
\[ \sum_{i=1}^{4} w_i = 1.0 \]
and so that a single catastrophic signal (e.g. \(\text{drift} = 1.0\)) can push the score below the CRITICAL threshold even when all other signals are green.
WebSocket + FastAPI + orchestrator threading created subtle race conditions when the orchestrator emitted events from a background thread while the WebSocket manager used asyncio. Fixed with call_soon_threadsafe wrapping ensure_future to bridge the sync/async boundary cleanly.
Keeping the demo fully offline while demonstrating live Phoenix integration required a clean mock/live switch in every adapter. ArizeAdapter checks for PHOENIX_API_KEY at init and routes every call to either the live client or a rich local mock — so the demo never fails regardless of environment.
Accomplishments that we're proud of
- The self-improvement loop is real code, not a slide.
self_improve_prompt()actually reads Phoenix evals, identifies failing patterns, and mutates the system prompt. You can watch it happen live in the cockpit. - The confidence engine has 34 unit tests covering score range, risk tier boundaries, recommended action routing, and contributing factor generation — all passing.
- The agent knows when to refuse. The CRITICAL-tier escalation path is a first-class feature. A dedicated "Agent Refused" demo scenario lets judges see the safety guardrail fire in real time.
- Zero render-blocking resources. The cockpit UI loads in under 200ms on a cold cache — non-blocking fonts, inline favicon, no external JS dependencies.
- WebSocket broadcasting to all tabs simultaneously — multiple engineers can watch the cockpit and see every stage event in real time.
What we learned
Arize Phoenix MCP is more powerful than we initially realized. We started thinking of it as a logging sink. By the end we understood it as a queryable knowledge base the agent can use to make better decisions in real time. The paradigm shift — from "trace for humans to read later" to "trace for the agent to read right now" — is the core insight of this entire project.
Deterministic confidence scoring beats LLM-based scoring for latency-sensitive safety gates. We tried an LLM-graded confidence approach first. It added ~800ms per stage and introduced its own hallucination risk. The weighted formula is faster, cheaper, more predictable, and easier to explain to a judge.
The UI is part of the agent. A cockpit that shows the agent's reasoning in real time isn't just a demo aid — it's the observability layer that makes the agent trustworthy to operators. Designing the UI and the agent architecture together, rather than bolting the UI on at the end, made both better.
What's next for SENTINEL
- Live Phoenix Cloud integration end-to-end — wire
PHOENIX_API_KEYinto production and run SENTINEL against real MongoDB Atlas incidents - Drift sparklines per service — pull
get_drift_signals()time-series data and render a 20-point sparkline inside each topology node - Multi-agent coordination — spawn parallel SENTINEL sub-agents for Stage 2 and Stage 5, reducing MTTR from \(\approx 8\text{s}\) to \(\approx 3\text{s}\) on complex incidents
- Slack / PagerDuty integration — emit confidence score and recommended action to the on-call channel before human approval decisions
- Phoenix dataset fine-tuning pipeline — use
sentinel_golden_tracesas a supervised fine-tuning dataset to periodically update Gemini model weights, closing the offline learning loop that the online self-improvement loop cannot reach
Log in or sign up for Devpost to join the conversation.