Vigil

Vigil: An AI agent that knows when your data is too stale for the decision it's about to power and fixes it before anyone has to ask

Comment

Inspiration

Every morning, executives approve budgets, ship products, and sign off on forecasts — all powered by data pipelines running silently in the background. But when a pipeline fails at 2 AM on a Sunday, no one knows until Monday's board meeting runs on 3-day-old revenue numbers.

Existing tools like Fivetran show you a red status dot. They don't know that the CFO's Monday presentation depends on that connector, or that a stale pipeline isn't just a technical issue — it's a business decision about to go wrong.

We wanted to build an agent that understands this gap: one that maps data reliability directly to business decisions and acts autonomously before damage is done.

What it does

Vigil is a proactive Data SLA Guardian Agent. It:

  • Monitors business decisions (not just pipelines) — each decision has a deadline, a stakeholder, and data dependencies
  • Assesses risk using Gemini — weighing business context, failure patterns, weekend trends, and deadline proximity
  • Acts autonomously — triggers early syncs, reloads schemas, and escalates to the right person when automated fixes fail
  • Shows full transparency — every reasoning step is traced end-to-end in a live dashboard

The two-layer architecture is cost-efficient: Layer 1 handles ~95% of checks deterministically (no LLM cost), while Layer 2 invokes Gemini only when risk is elevated.

How we built it

Agent Layer: Google ADK 2.0 with Gemini 2.5 Flash (via Vertex AI). The agent uses Fivetran's MCP server (77 tools, read + write enabled) to pull live connector state and execute remediation — sync_connection, reload_schema, run_setup_tests, modify_connection. MongoDB MCP enables pattern recall during reasoning.

Orchestration Layer: FastAPI service with a periodic scheduler, two-layer risk engine, and SSE streaming. Motor (async pymongo) handles persistence. The orchestrator provides business context; the agent pulls live external state and acts.

Dashboard: React + TanStack Router + Tailwind + shadcn/ui + Framer Motion. Real-time SSE feed shows agent actions as they happen. Decision detail pages expose full reasoning traces, confidence drivers, dependency graphs, and action timelines.

Infrastructure: MongoDB Atlas for persistence, Render for deployment, real Fivetran connectors (Google Sheets, GitHub, Fivetran Metadata).

Execution Flow: Scheduler → Layer 1 (fast check) → Agent (Gemini) → Fivetran MCP (act) → Persist + SSE (notify)

Challenges we ran into

  1. Agent response parsing — ADK's is_final_response() doesn't always capture text when the agent calls MCP tools mid-reasoning. We built fallback accumulators and inference logic to handle partial responses.

  2. Vertex AI rate limits — Switching from Gemini API keys (exhausted credits) to Vertex AI via Application Default Credentials required careful Pydantic settings management and environment variable handling.

  3. MCP stdio subprocess reliability — The Fivetran MCP server runs as a stdio subprocess within ADK. Startup timing, credential passing, and error propagation needed careful handling.

  4. SSE connection lifecycle — EventSource onopen doesn't fire until the server sends an initial event. We added a connected event + periodic heartbeats to maintain dashboard liveness.

Accomplishments that we're proud of

  • End-to-end autonomous remediation — The agent diagnoses, fixes, verifies, and escalates in one reasoning turn. No human-in-the-loop unless it explicitly decides to escalate.
  • Two-layer cost efficiency — 95% of checks resolve in Layer 1 with zero LLM cost. Gemini only fires when there's genuine risk.
  • Real Fivetran MCP write operations — The agent triggers real syncs, reloads schemas, and modifies connections on live connectors. Not mocked.
  • Full reasoning transparency — Every decision is traceable: facts in → Gemini reasoning → decision → actions taken → outcome. Auditable by default.
  • Live SSE dashboard — Real-time updates as the agent works. You can watch it assess, act, and escalate in seconds.

What we learned

  • Business context transforms monitoring into decision support. A red dot means nothing without knowing who depends on that data and when they need it.
  • Two-layer architectures save cost. Most checks are healthy — don't burn LLM calls on them.
  • MCP enables true agent autonomy. Read + write in one reasoning turn means the agent can close the loop itself.
  • Transparency builds trust. Full reasoning traces make AI-driven automation auditable and explainable to stakeholders.

What's next for Vigil

  • Multi-connector dependency graphs — Model complex decision chains where one pipeline feeds another
  • Slack/Teams integration — Escalation messages delivered directly to the responsible person with full context
  • Learning from outcomes — Agent improves remediation strategies based on what worked historically
  • Custom SLA definitions via natural language — "Make sure the board has fresh revenue data every Monday by 9 AM" → auto-configured monitoring
  • Multi-tenant support — Organizations define their own decisions, stakeholders, and risk thresholds

Built With

  • fastapi
  • fivetran-mcp-server
  • framer-motion
  • gemini-2.5-flash
  • google-adk-2.0
  • mongodb-atlas
  • motor-(async-pymongo)
  • pydantic
  • python
  • react
  • render
  • server-sent-events
  • shadcn/ui
  • tailwind-css
  • tanstack-router
  • typescript
  • vertex-ai
  • vite
Share this project:

Updates