Probe

Autonomous closed loop that detects, diagnoses and reverts production AI-agent regressions — opening a validated merge request on GitLab in under five minutes.

Comment

Inspiration

Production AI agents fail silently. A fraud-detection co-pilot doesn't crash — it starts hallucinating ticker symbols at 3 a.m. with full confidence. A RAG assistant doesn't throw — its retriever silently degrades and every answer becomes "I don't know". Catching either today means paging a human on-call who greps traces for hours, at ~$400/hour. In Financial Services — fraud detection, KYC/AML, loan-officer assistants — those silent hours are mis-classified transactions in real time.

We built Probe to be the SRE for that new class of incident: regressions in AI agents themselves.

What it does

Probe runs an autonomous closed loop:

  1. Detect — reads execution traces of the target AI agent from Arize Phoenix (via the official Phoenix MCP server) and classifies the failure pattern: hallucination vs. retrieval degradation.
  2. Diagnose — lists recent branches and diffs them against main (via Probe's own MCP server) to find the suspect commit.
  3. Fix — applies a revert, runs the target's eval suite as a self-eval gate (Gemini judge, 10/12 pass = 83% on the case in the video), and only proceeds if the gate accepts.
  4. File — opens a merge request on gitlab.com (via the community GitLab MCP server) with an engineer view and an executive view, plus the trace evidence.

Every investigation is persisted in Firestore and can be replayed in the public UI, event by event, with each tool call tagged by the MCP server that carried it. To date: 11 validated MRs, $2.93 total compute vs $364.70 of human-baseline cost — 125× cheaper, 3m37s average MTTR across 18 investigations.

How we built it

  • Agent: Google Agent Development Kit with Gemini 2.5 Pro (reasoning) and Gemini 2.5 Flash (triage + eval judge) on Vertex AI.
  • Three MCP toolsets wired into one agent:
    • @arizeai/phoenix-mcp (official partner server) — list-traces, get-spans;
    • @zereight/mcp-gitlab (community, 181 tools) — create_merge_request, search_project_code;
    • probe.mcp_server (our own FastMCP) — branch listing, diffs, file content. It's both consumed by the agent and exposed as a provider, so Claude Code / Cursor can call Probe's tools too.
  • Backend: FastAPI on Cloud Run, streaming investigations over SSE, persisting to Firestore, secrets in Secret Manager.
  • Frontend: Next.js 15 + Tailwind on Cloud Run, with a replay timeline that color-codes every tool call by MCP transport (emerald = Phoenix, cyan = Probe, violet = GitLab).
  • Demo target: a public RAG bot with two reproducible failure branches (regression/prompt, regression/retrieval) and a 15-question eval harness — so the loop is verifiable end-to-end by anyone.

Challenges we ran into

  • Stdio MCP servers inside Cloud Run: concurrent npx installs raced (ENOTEMPTY); we pre-install both npm MCP servers in the image and run npx --no-install.
  • The official GitLab Duo MCP endpoint is gated behind Duo Enterprise seats (403 even on an Ultimate trial), so we integrated the community @zereight/mcp-gitlab server instead — same GitLab API surface, fully functional for MRs.
  • Next.js NEXT_PUBLIC_* vars + Cloud Run source deploys: --set-build-env-vars is silently ignored when a Dockerfile is present; the bundle shipped with an empty API URL. We moved the frontend build to an explicit Cloud Build config with real --build-args and added a deploy smoke that greps the served bundle.
  • Cost variance is real: rich Phoenix annotations ballooned one investigation to $1.36 while others ran at $0.14–$0.20 — so the UI reports per-case cost and the README quotes the honest range.

Accomplishments we're proud of

  • A real closed loop on public infrastructure — judges can open the live UI, replay a case, and land on a real MR on gitlab.com without logging in.
  • Self-eval gating makes autonomy safe: Probe doesn't merge anything; it proposes a revert that already passed the target's own eval suite.
  • Three MCP servers, two partners, one agent — and Probe is an MCP provider, not just a consumer.

What we learned

MCP is a genuine integration fabric: swapping the gated official GitLab server for the community one was a one-factory-function change. And an agent you can afford to let run needs two things: an eval gate it must pass, and an audit trail you can replay.

Honest scope & what's next

Probe doesn't replace the on-call engineer for novel failures — it automates the repeatable ones (regression-shaped incidents with a revertable cause). Next: the official GitLab Duo MCP server when it opens up, a broader failure taxonomy (tool-timeout storms, cost runaways), CI-triggered investigations, and a triage handoff that pages a human with the evidence pre-assembled when the pattern is novel.

Built With

  • arize-phoenix
  • cloud-build
  • fastapi
  • firestore
  • gemini
  • gitlab
  • google-agent-development-kit
  • google-cloud-run
  • mcp
  • next.js
  • node.js
  • python
  • secret-manager
  • tailwindcss
  • typescript
  • vertex-ai
Share this project:

Updates