Fadlie

The same data lives in four systems; only one copy is governed. Fadlie finds the twins in DataHub and copies the missing governance across — and never invents a value.

Comment

Inspiration

I started from a question I could not answer with a catalog: where is the personal data?

The contest's own showcase-ecommerce datapack answers it — confidently, and wrongly. customers.customer_id carries the PII_Data tag on Postgres. The same column, in the same table, on dbt, S3 and Snowflake, carries nothing. The catalog does not say "I am not sure". It returns one dataset and stops.

That is not a tagging bug. The data was copied across four platforms; the governance was not. Somebody curated one copy, and the knowledge stayed there.

So the obvious fix was lineage — the graph already knows all four. I measured it before building on it, and it does not work.

What it does

Fadlie is an MCP agent that finds datasets holding the same data across different platforms, then finds the governance that stopped at one of them — and levels it.

Four tools, over MCP:

Tool What it returns
catalog_summary pairs examined, groups found, gaps, disagreements
find_duplicate_datasets the groups, with the judge's verdict and reason for each pair
governance_gaps every gap, each naming the dataset the value would be copied from
apply_governance writes them. Dry run by default

Two rules shape all of it.

It copies; it never writes. No value put into the catalog was produced by a model. Every one comes from a twin that already carried it, and names its source — the Ecart type refuses to exist without one. A description generated by a machine is indistinguishable from a description written by the team that knows the data. Six months later nobody can tell which is which, and the catalog has quietly become fiction.

A disagreement is not a gap. snowflake/ORDER_DETAILS sits in Ecommerce Operations while its twins sit in Data Platform Team. Fadlie reports that and does not choose. Someone decided, or someone erred, and neither is an agent's call.

What the measurements found

Every number here comes from a script in the repository, run against a live DataHub.

Lineage tells you nothing about whether two datasets hold the same data. The graph is a single connected component: 103 nodes, 161 edges, not one isolated dataset. All 88 same-name pairs are connected — and so is every one of the 316 pairs picked at random, so connectivity carries no information at all. Distance carries almost none: cut at 4 hops and you keep 86 of the 88 twins, along with 199 of the 316 strangers. There is no threshold that keeps the twins and drops the rest.

Names are almost as bad. Four Tableau datasets are all called Custom SQL Query and share 0 % of their columns; promotions shares 9 %. Three of fifteen same-name groups are not the same thing.

So structure only shortlists — column overlap cuts 2 211 pairs down to 97 — and a model decides, one pair at a time.

How we built it

  • DataHub Core v1.7.0 on EC2, loaded with the contest datapack — 67 datasets across Snowflake, dbt, Postgres, S3, Tableau, PowerBI and Looker. Read through the GraphQL and OpenAPI v3 endpoints; written back through v3 aspects.
  • Amazon Bedrock, Amazon Nova Micro as the judge, temperature 0, in eu-central-1.
  • AWS App Runner for the MCP server over HTTPS, image in ECR, DataHub token in Secrets Manager.
  • Python and the MCP SDK, with 77 tests that need no network and cost nothing to run.

Fadlie does not sit on top of DataHub's own MCP server. It exposes its own, because the agent had to write to the catalog, not only read from it — and writing needs guarantees a read-only surface has no reason to carry: a dry run by default, a named source on every value, and a refusal to choose when two twins disagree. The catalog itself is reached through DataHub's Python SDK and GraphQL, on a DataHub Core instance loaded with the contest's own datapack.

Challenges we ran into

The judge is the one component that must never fail silently. If it could quietly return "different", Fadlie would report no duplicates found — a catalog in perfect order. Nobody audits good news. So it raises instead, the server probes the model before its first verdict, and both real failure modes are tested: a bare model id (Frankfurt requires the regional inference profile) and credentials without Bedrock access.

App Runner cuts every request at 120 seconds. Measured: 504 upstream request timeout at 120.58 s. A full analysis takes about 240. Worse, the writes had already succeeded when the 504 came back — a failure that was actually a success, which is the same lie in the other direction. The analysis now runs in a background thread; a stale report is served immediately while a fresh one is computed, and applying gaps subtracts what it wrote instead of forcing a recompute.

The EU inference profile routes outside the region you call from. An AccessDeniedException named eu-west-3 when everything was configured for eu-central-1. The IAM policy needed a region wildcard on the foundation model.

A deployed server answered /health with 200 and every MCP session with 421 Invalid Host header. DNS-rebinding protection turns on automatically for the default host, and the App Runner domain has to be declared. The probe now performs a real MCP initialize — anything less would have gone on looking green.

Accomplishments that we're proud of

scripts/mesurer-le-juge.py puts the judge against 16 hard pairs drawn from the real catalog: replicas differing in case and column count, reference tables of identical shape, aggregates computed from a table rather than copied from it, and the four homonymous Tableau queries that share only measure names. 16 out of 16. Ten runs of the same pair, ten identical verdicts.

And it is not a demo that only runs on a laptop. On the deployed instance, Fadlie has levelled three groups — customers, orders, products — writing 102, 60 and 53 values it read from their twins, the last 53 in 11 seconds. 24 checks through a real MCP client pass against the public URL.

What we learned

Measure the obvious answer before building on it. Lineage was the design, right up until the median distance between random datasets came back equal to the distance between twins.

A number read yesterday is not a number. The judge confirms 83, 85 or 84 pairs out of the same 97 across runs. The structural findings never move; counting totals do. Every figure quoted here is read off the run that produced it.

A failure must not look like good news. Most of the hard work was not making the agent right — it was making sure that when it is wrong, it says so.

What's next for Fadlie

Scheduled sweeps that report drift instead of fixing it; write-back to more aspects; and a review queue for disagreements, which are the interesting half and the half no agent should settle.

Judges — try the deployed agent.

MCP endpoint: https://yvh3rv2qmp.eu-central-1.awsapprunner.com/mcp

Header: Authorization: Bearer 2fbf762df1fc8c1ae977fa90138c967e1118df92839fc7e0

Start with catalog_summary, then governance_gaps on any dataset name. apply_governance is a dry run unless you pass dry_run: false.

Built With

  • amazon-bedrock
  • amazon-ec2
  • amazon-ecr
  • amazon-nova
  • aws-app-runner
  • aws-secrets-manager
  • boto3
  • datahub
  • docker
  • graphql
  • mcp
  • pytest
  • python
Share this project:

Updates