sentinel-agent

An MultiAgentic AI SRE for production AI in financial services detects incidents, writes regulator-grade postmortems with quantified customer impact, and gets better with every run.

Comment

Inspiration

Every bank running AI in production has the same nightmare: the fraud model starts misclassifying, KYC silently drifts, an upstream feature pipeline goes stale — and the on-call engineer has fifteen minutes to figure out what changed before customer accounts get frozen and revenue starts bleeding. The tooling for this doesn't exist. SREs have Datadog. ML engineers have W&B. But for production AI in regulated industries, there is no dedicated incident-response system. We built one.

What it does

Sentinel is an open-source AI Site Reliability Engineer for production AI in financial services. When a monitored AI system shows anomalies, Sentinel autonomously: detects the incident, correlates against recent deploys, runs distribution-drift and fairness audits, quantifies customer and dollar impact, drafts a remediation plan, generates a regulator-grade postmortem with citations to specific clauses of SR 11-7, EU AI Act, and OCC guidance, and routes notifications. It runs as 18 specialized sub-agents coordinated via Google ADK and A2A protocol, and it gets measurably better at the job over time by introspecting its own Arize Phoenix traces and evolving its own prompts from observed performance.

Before any regulator notification gets drafted, Sentinel pauses and waits for explicit human approval — a synchronous gate that maps directly to the hackathon's "under your oversight" mandate.

A typical fraud false-positive incident produces a postmortem with $84,293 at risk, 312 affected customers, 1,247 affected transactions, drift severity flags, a fairness audit per protected class, and a 7-day regulator-reporting obligation — every figure cited to a source.

How we built it

Sentinel is built on Google Cloud, Gemini 3, and Arize Phoenix. The agent runtime is Google ADK (Python), deployed to Cloud Run as two services a FastAPI backend and a Next.js 14 frontend (light, Linear/Stripe-aesthetic UI with full SSE streaming of agent reasoning). All 18 sub-agents are instrumented with OpenInference, sending traces to Phoenix. The system uses Phoenix MCP as load-bearing infrastructure: the Coordinator introspects its own traces at runtime to extract plan-shaping directives the self-improvement loop the Arize track explicitly rewards. Persistent memory and a regulatory corpus are indexed in Vertex AI Vector Search. Multi-MCP integration includes GitHub (for deploy correlation), Slack (for incident comms), and Phoenix (for self-introspection). CriticAgent grades every postmortem against a four-dimension rubric and triggers bounded iterative refinement. A Prompt Evolver agent demonstrably improves underperforming agent prompts based on rolling critic scores. PatternMiner detects recurring incident clusters across memory and proposes new directives. Eighteen ADRs document every non-trivial decision. 336 backend tests, 25 frontend tests, and a real-Vertex E2E suite cover the pipeline.

The full system is publicly deployed on Cloud Run — the web app, the API, and a live Phoenix trace explorer are all reachable from a single demo URL.

Challenges we ran into

Three real ones. First: the self-improvement loop is easy to fake (memory recap dressed as learning) and hard to make real. We had to redesign it twice first attempt produced an identical execution plan with noisy wall-clock variance; second attempt produced trace-derived directives that deterministically change plan structure. Second: Phoenix Cloud is geo-blocked in our region. We solved this with graceful-degradation on httpx.ConnectError and an in-process span cache so the agents reason about real evidence even when the collector is unreachable. Third: stopping the project from scope-creeping under the "don't worry about time" instinct. We enforced a strict operating protocol with anti-patterns, atomic commits, and ADR-gated decisions

Accomplishments that we're proud of

  • A multi-layer self-improvement loop that goes beyond RAG over past incidents: agents' own prompts measurably evolve based on observed critic-score outcomes, and recurring incident patterns auto-distill into new directives.
  • Regulator-grade postmortems with audit citations to real published regulations (SR 11-7, EU AI Act, OCC, NIST AI RMF) never hallucinated, always traced to a corpus document.
  • 18 specialized sub-agents, every one earning its place against a real regulatory requirement, operational need, or scoring criterion. No filler agents.
  • Production-grade engineering rigor 336 tests, TDD throughout, 18 ADRs, locked OpenAPI contract, atomic conventional commits.

What we learned

A multi-layer self-improvement loop that goes beyond RAG over past incidents: agents' own prompts measurably evolve based on observed critic-score outcomes, and recurring incident patterns auto-distill into new directives. Regulator-grade postmortems with audit citations to real published regulations (SR 11-7, EU AI Act, OCC, NIST AI RMF) never hallucinated, always traced to a corpus document. 18 specialized sub-agents, every one earning its place against a real regulatory requirement, operational need, or scoring criterion. No filler agents. Production-grade engineering rigor 336 tests, TDD throughout, 18 ADRs, locked OpenAPI contract, atomic conventional commits.

What's next for Sentinel

Sentinel is open-source and positioned as a framework, not a single project. Next: production pilots with regulated fintechs (active outreach), expanded regulatory corpus coverage (FCA, MAS, OSFI), Layer-5 eval-rubric self-modification, and a skills marketplace for organizations to publish custom incident-response agents tuned to their AI stack.

Built With

  • agent-builder
  • arize-phoenix
  • artifact-registry
  • cloud-build
  • cloud-run
  • fastapi
  • gemini-3
  • gemini-3-flash
  • gemini-3.1-pro
  • github-mcp
  • google-adk
  • google-cloud
  • lucide
  • mcp
  • model-context-protocol
  • next.js
  • openinference
  • opentelemetry
  • phoenix-mcp
  • playwright
  • pytest
  • python
  • recharts
  • server-sent-events
  • shadcn-ui
  • slack-mcp
  • tailwind
  • tanstack-query
  • typescript
  • vertex-ai
  • vertex-ai-vector-search
  • vitest
Share this project:

Updates