Tiresias

The pre-cognitive data quality agent that catches silent semantic failures in Fivetran pipelines, the kind every monitor misses, where the data is green but quietly wrong.

Comment

Inspiration

Tiresias started with a question I couldn't shake: what if the worst data failures are the ones that never trigger an alert?

In Fivetran's own 2026 Enterprise Data Infrastructure Benchmark Report, the company surveyed 500 senior data leaders and found that 97% said pipeline failures had slowed their analytics or AI programs, with enterprises spending an average of $2.2 million per year just keeping pipelines running. Fivetran's CEO George Fraser put it plainly: "too much spend is still going toward maintaining fragile pipelines built for a pre-AI era."

But there's a category of failure even more dangerous than the loud ones — the silent semantic failures. A salesperson renames a deal stage in HubSpot from "Contract Sent" to "Contract Under Review." Fivetran syncs the change perfectly. Every connection is green. No error fires. No monitor catches it.

And yet, somewhere downstream, a dbt model filters on the literal string 'Contract Sent'. That filter now matches nothing. The VP of Sales's revenue dashboard silently drops from $2.4 million to $0. Nobody knows for weeks , until someone notices Q3 looks wrong and a 2 AM Slack thread starts.

That's the gap Tiresias was built for. Every other "data observability" tool reacts to broken pipelines. Tiresias is pre-cognitive — it catches failures that hide inside healthy pipelines.

What it does

Tiresias is a multi-agent system that watches Fivetran-synced data, detects silent semantic failures, and acts on them through Fivetran's own MCP.

The system runs as a coordinated team of four agents:

  • Memory — builds statistical fingerprints of every Fivetran-synced table in BigQuery. Tracks row counts, distributions, cardinality, null rates, schema deltas, and value frequencies. Knows what "normal" looks like.
  • Lineage — parses the dbt manifest to trace the downstream blast radius of any change. When a column drifts, Lineage knows which models, dashboards, and executives are at risk.
  • Oracle — powered by Gemini 3.1 Pro on Vertex AI. Reasons over the drift, the blast radius, and the historical fingerprints to classify the change as one of four categories: SILENT_SEMANTIC_FAILURE, UPSTREAM_DATA_QUALITY, BENIGN_GROWTH, or NORMAL_SEASONAL. Returns a confidence score and a natural-language reasoning trace.
  • Tiresias (the orchestrator) — wires them together. When a Fivetran sync completes, Tiresias triggers Memory → Lineage → Oracle → MCP, gates every destructive action behind a human approval, and logs every decision for audit.

When a SILENT_SEMANTIC_FAILURE is detected with high confidence, Tiresias proposes a fix — and on human approval, it executes a real modify_connection_table_config call through the Fivetran MCP server to quarantine the affected table at the Fivetran source. The corrupted data stops flowing downstream until an engineer fixes the root cause. Scoped. Reversible. Real.

The demo scenario

The end-to-end demo is built around one realistic, plausible, two-second human action: a sales ops admin renaming a HubSpot stage from "Contract Sent" to "Contract Under Review."

Nothing about that action is wrong — it's a legitimate change someone might make every week. But because a downstream dbt model filters on the literal string, the VP of Sales's revenue dashboard silently drops from $2.5M to $0.

In the demo, you watch the dashboard collapse in real time. Then you watch Tiresias catch it within seconds: Memory's PSI score on the label column spikes to ~2.1 (vs. a threshold of 0.25), Oracle reasons over the disappear/appear pattern and classifies it as a silent semantic failure with 95% confidence, Lineage traces the impact to the VP's dashboard, and on approval, the real modify_connection_table_config MCP call fires against Fivetran — the table is quarantined, the bleeding stops.

Everything in the demo is real: real Fivetran sync, real BigQuery data, real Gemini reasoning, real MCP call against the live Fivetran account.

How we built it

Tiresias is a multi-agent system orchestrated by FastAPI, with three specialized agents — Memory, Lineage, and Oracle — coordinated under a single orchestrator that drives writes to Fivetran through MCP.

Detection (Memory). Memory profiles BigQuery tables on every Fivetran sync and stores per-snapshot fingerprints in a tiresias_fingerprints table in BigQuery itself. For each watched table it computes row counts, null rates, distinct counts, per-column type, top-N value frequencies, entropy for categoricals, and min/max/mean/stddev/p1/p25/p50/p75/p99 for numerics — all in a single scan where possible to keep BigQuery costs bounded. Downstream tracing (Lineage). Lineage parses a dbt manifest.json into a networkx directed graph of sources → staging models → mart models → exposures. Reasoning (Oracle). Oracle is the inference layer, powered by Gemini 3.1 Pro on Vertex AI via Google's Gen AI SDK (the legacy Vertex AI SDK doesn't support Gemini 3). It receives the DriftReport and BlastRadius and classifies the drift into one of four enums: SILENT_SEMANTIC_FAILURE, UPSTREAM_DATA_QUALITY, BENIGN_GROWTH, or NORMAL_SEASONAL. Orchestration and the MCP write path. The orchestrator is a FastAPI app exposing POST /webhook/fivetran (HMAC-signature-verified, production path) and a development trigger gated by TIRESIAS_DEV_TRIGGERS=true (only used to fire the demo without a real Fivetran webhook signature). Frontend. Next.js 14 (App Router), TypeScript, Tailwind, and shadcn/ui, deployed on Vercel. The main page is a two-panel layout: a React Flow graph driven by the live /lineage/blast-radius response (not a hardcoded picture — nodes and edges come from Lineage's actual output, with Framer Motion edge-pulse animations sweeping downstream when a verdict arrives), and Oracle's verdict card with classification, confidence, reasoning rendered in a serif italic to give the AI a distinct voice, severity-color-coded blast radius, proposed action, and Approve / Dismiss buttons.

Architecture

Fivetran sync ──webhook──> FastAPI orchestrator
                                  │
                                  ├──> Memory (BigQuery profiling, KS / PSI / Z-scores)
                                  ├──> Lineage (networkx graph from dbt manifest)
                                  └──> Oracle (Gemini 3.1 Pro structured output)
                                          │
                                          └──> Pending verdict → human approval (Next.js UI)
                                                                       │
                                                                       └──> Fivetran MCP
                                                                            modify_connection_table_config
                                                                            (enabled=false, scoped, reversible)

Tech stack

  • AI: Gemini 3.1 Pro on Vertex AI via Google's Gen AI SDK, with structured output for the four-class classification.
  • Partner integration: Fivetran's official MCP server (forked as a submodule), driven over stdio with FIVETRAN_ALLOW_WRITES gating all destructive calls.
  • Data: BigQuery as the destination (Fivetran's recommended quickstart pattern), HubSpot as the source connector.
  • Backend: Python 3.11, FastAPI, Pydantic for structured contracts.
  • Statistical detection: scipy / numpy — KS test for numeric distributions, Population Stability Index for categoricals, Z-scores for row counts, schema-delta diffs. No heavy ML — clear, interpretable math.
  • Lineage: networkx graph built from a structurally-valid dbt manifest fixture (Lineage code parses any real dbt manifest unchanged).
  • Frontend: Next.js 14 (App Router), TypeScript, Tailwind, shadcn/ui, React Flow for the dependency graph, Framer Motion for the propagation animations. Hosted on Vercel.
  • VP dashboard: A second route that queries live BigQuery on a 5-second poll, so the "$2.4M → $0" collapse during the demo is an honest, real consequence of the broken downstream filter — not a display trick.

Challenges we ran into

thought-signature behavior in function-calling. The first real Oracle call also silently produced empty responses because max_output_tokens=512 was being consumed entirely by thinking tokens — the fix was a thinking_budget=1024 cap plus max_output_tokens=2048 headroom. Real schema names, not assumed ones. Fivetran-synced HubSpot data doesn't expose a dealstage column — every column gets a property_ prefix, and what I assumed would be a string "Contract Sent" lives in a separate deal_pipeline_stage lookup table joined via stage_id. This forced a better design: instead of detecting the rename on the deal fact table, Memory watches the dimension table where the label actually lives — which is exactly the kind of silent-failure signal nobody monitors, because "it's just a lookup table." Distinguishing real reasoning from prompted reasoning. My first Oracle prompt included a confidence heuristic ("PSI > 2.0 with a clean rename signature is near-certain"). That was leading the model toward the answer. I rewrote the prompt to make the confidence guidance general, so Oracle reaches the verdict by genuine pattern recognition — which is exactly what it does now ("'Contract Sent' completely disappeared at the exact same time 'Contract Under Review' appeared with the same frequency... this strongly suggests a simple rename event"). Gemini 3.1 Pro setup quirks. The model only runs on the global endpoint, not a regional one — a detail buried in the docs that would have caused a silent "model not available" error if I'd missed it. I also had to use Google's new Gen AI SDK, not the legacy Vertex AI SDK, and handle Gemini 3's new

Accomplishments we are proud of

The whole pipeline works on real infrastructure. Real Fivetran sync, real BigQuery, real Gemini 3.1 Pro reasoning, real MCP call quarantining a real table. End to end. Not a single mocked component in the demo path.

Oracle reasons like a senior data engineer. Read its actual output: "The 'label' column exhibits a classic rename pattern where 'Contract Sent' dropped from 14.30% to 0% and 'Contract Under Review' appeared at 14.29%, while the underlying 'stage_id' ('contractsent') and row counts remained identical." That last clause — the fact table is unchanged yet revenue silently breaks — is the kind of insight a sharp engineer would write in a postmortem.

The dimension-table failure mode. The demo catches a class of failure (a label change in a dimension table while the fact table is identical) that almost no existing monitor would detect. It's exactly the gap Fivetran's own 2026 benchmark calls out.

The VP dashboard's honest $0. When the rename hits, the dashboard's live BigQuery query genuinely returns zero — not a display trick. The audience sees the consequence and the diagnosis in the same demo.

A multi-agent architecture that's actually multi-agent. Memory, Lineage, and Oracle each have a clear, isolated job and a structured Pydantic contract between them. The orchestrator coordinates without entangling.

What we learned

Agents become genuinely useful the moment they can act, not just observe. The difference between Tiresias and a static dashboard isn't the analysis — it's the MCP call that quarantines the table on approval. That's the agent moment. The hardest hackathon problems aren't building features — they're choosing the right failure scenario. The silent-rename-on-a-dimension-table scenario took longer to design than to implement. But it's what makes the whole demo land. Fivetran's MCP is a serious agent surface. The OpenAPI schema-file requirement, the writes-enabled gating, the scoped table-level operations — it's designed for AI agents to act safely on production data. That changes what you can build on top of Fivetran fundamentally.

What's next for Tiresias

An incident memory. Every past detection stored, queryable, learnable. Over time, Tiresias would know "this exact pattern happened six months ago, and the fix was X." A lighter screening tier. Use Gemini 3.1 Flash-Lite for a fast first-pass check on every sync, escalating to Pro only when something looks suspicious — cheaper and faster at scale. Open-source the watching layer. A standalone tiresias-watch library that any company on Fivetran could point at their warehouse, with their own MCP credentials, to get this protection.

The bigger thesis Tiresias points at: as enterprises move toward agentic AI on top of data warehouses, the gating problem isn't model intelligence — it's whether the data those agents reason on is trustworthy. A brilliant AI agent making decisions on silently-wrong data is worse than no agent at all. Tiresias is one answer to that problem.

Built With

Share this project:

Updates