Sentinel

Inspiration

Every team shipping LLM products lives with a quiet fear: the model is fluent, the responses look reasonable, and the product is silently broken. A "harmless" prompt update can change a chatbot's compliance behavior. A model upgrade can quietly reduce factuality. A data shift can drift outputs in ways no dashboard catches.

Traditional observability tools detect when things crash. They don't detect when things are wrong. And when they do detect anomalies, they alert humans, they don't act. The on-call engineer still has to diagnose root cause, propose a fix, and execute it manually. For an LLM product, that loop can take hours, while the broken model is approving refunds, leaking PII, or giving dangerous advice in real time.

I built Sentinel to close that loop. Not just an alerting layer, but an autonomous SRE that diagnoses LLM regressions with experimental evidence, proposes remediations, and critically asks the human before pulling the trigger.

What it does

Sentinel watches a production LLM application (in my demo: "RefundBot," a customer support agent for a fictional retailer). It executes an 8-step autonomous loop:

Detect :- pulls live traces from Arize Phoenix; compares the last 15 minutes against a 24-hour baseline
Triage :- runs statistical regression detection on eval-pass rate, latency, and error rate
Diagnose :- identifies recently deployed prompt versions as likely root causes
Verify the hypothesis :- runs experimental evaluation: the suspect prompt and the previous known-good prompt, both against a golden dataset of 30+ labeled scenarios, in parallel
Propose :- opens an incident in Firestore with the diagnostic evidence and a remediation plan
Await human approval :- blocks execution until a human approves or rejects through the UI
Execute :- on approval, deploys the rollback through Arize Phoenix's prompt management (a new audited version, not an overwrite)
Verify the fix :- re-runs the golden dataset against the now-live prompt, confirms accuracy restored, writes a postmortem

The whole loop happens in 90 seconds. The human-in-the-loop happens in one click.

How I built it

Three services, all on Cloud Run:

RefundBot is the patient, a FastAPI service backed by Gemini 2.5 Flash, instrumented with OpenInference to send traces to Phoenix. Critically, it fetches its system prompt from Phoenix on every request (with a 5-second cache), so prompt rollbacks take effect immediately without redeployment.

Sentinel is the agent, built on Google ADK with Gemini 3.1 Pro as the reasoning model. Eight tools across two domains:

Phoenix integration: compute_metric_window, list_phoenix_prompts, deploy_prompt_version, run_dataset_evaluation, fetch raw traces
Incident management: open_incident, update_incident, request_human_approval (with blocking Firestore polling on a 2-second cadence and a 5-minute timeout)

The agent reasons step-by-step through its protocol, emitting plan steps that stream to the UI in real time via Server-Sent Events.

Next.js cockpit is the human interface, a three-panel React app (Next.js 15, Tailwind v4, Framer Motion). Live incident feed on the left, streaming reasoning trace in the center, approval queue on the right. Operators trigger Sentinel runs from the UI and approve remediations with one click.

Stack:

Google ADK + Gemini 3.1 Pro · Vertex AI Agent Builder
Arize Phoenix Cloud (traces · prompt management · golden datasets)
Firestore (incident state · approval queue)
Cloud Run (3 services · deployed via gcloud run deploy --source)
Next.js 15 · React · TypeScript · Tailwind v4
FastAPI · uvicorn · Python 3.11
Server-Sent Events for live agent reasoning stream

Challenges I ran into

Phoenix Cloud auth varies by deployment. Different Phoenix spaces use different auth schemes, some expect Bearer tokens, some api-key headers, some need space-scoped base URLs. Took half a day with a diagnostic script to figure out our specific setup uses Bearer.

Gemini 3.1 Pro Preview is global-only. Setting location="us-central1" fails silently with 404. Pinning location="global" was the fix, and required setting four separate environment variables (GOOGLE_GENAI_USE_VERTEXAI, GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION, plus the custom ones) for ADK to consistently pick it up.

Eval determinism. Initial runs gave +/- 10% accuracy variation on identical inputs. Fixed by setting temperature=0 on RefundBot's Gemini call, using a fixed random seed on dataset sampling, and writing a smarter response classifier that handles "I'm sorry, but I'll process..." (approval signals win over decline signals).

The human-approval gate must actually block. First version returned immediately. I now poll Firestore on a 2-second cadence with a 5-minute timeout, returning either {approved: true}, {approved: false, reason: "human_rejected"}, or {approved: false, reason: "timeout"} so the agent knows the difference.

Cloud Run + service account drama. A late-night billing reconfiguration deleted the default Compute service account by tombstone, breaking all Cloud Run deploys. Resolved by creating a custom service account (sentinel-runner@...) with explicit grants for Editor, Secret Manager Accessor, and Vertex AI User roles.

Accomplishments that I am proud of

The closed loop actually closes. Most "AI observability" projects stop at detect-anomaly. Sentinel detects -> diagnoses with experimental evidence -> proposes -> human-approves -> executes -> verifies the fix. The verification step (post-rollback accuracy = 73%, restored from 33%) is what separates a real agent from a chatbot with tools.

Sentinel is observable in Phoenix. Every reasoning step the agent takes is itself a trace in the same Phoenix platform it operates on. The agent watching the AI is itself watched by the AI platform. That recursive auditability is what makes Sentinel deployable rather than just demo-able.

Live reasoning visualization. The Server-Sent Events pipeline streams every plan step, tool call, and result from the Python backend to React in real time. Operators don't wait in silence they watch the agent think. This turned out to be the most satisfying part of the demo.

Production deployment under a hackathon timeline. Three independent services deployed to Cloud Run, with SSE streaming, CORS, Secret Manager, Firestore, and a polished React UI, all on a free trial account in 12 days.

What did I learn

Tool design matters more than prompt design. Sentinel's reliability comes from giving the model the right eight tools, not from cleverly worded instructions. When a tool returns clean structured data (e.g. accuracy: 0.73, correct: 11, by_category: {...}), the agent's reasoning becomes near-perfect. When a tool returns ambiguous results, no amount of prompt engineering recovers it.

Determinism is half the battle for evals. A "smart classifier" with temperature=0 and fixed sampling seed produces consistent benchmarks. Without those, you can't tell if the model got worse or if the eval is just noisy.

Human-in-the-loop is a UX problem, not just a safety check. The approval gate must feel low-friction (one button click), high-context (rationale + evidence visible), and revocable (clear reject path). We iterated three times on the approval card design.

Phoenix prompts as versioned artifacts is the right abstraction. Every rollback is a new prompt version, not an overwrite. Full audit history. Reversible. This is how prompts should be deployed in production.

What's next for Sentinel

LLM-as-judge for evaluation. Replace the heuristic accuracy classifier with Phoenix's evaluation framework, let an LLM grade outputs against a rubric instead of phrase-matching.
Multi-step remediation. Some regressions need more than a rollback (e.g. retraining, data filtering, scale-down). Plan-then-execute, not just single-step rollback.
Sentinel watches Sentinel. Agent-on-agent: run A/B experiments on Sentinel's own prompts using the same Phoenix infrastructure it operates through.
Multi-tenant. Today Sentinel watches one app. Production deployment would watch dozens of LLM services across an org, prioritizing incidents by business impact.