TrustBoard

Multi-agent trust scoring for DataHub: it writes the score back into the graph, and the next agent reads it over MCP before acting.

Comment

About the project

Inspiration

Every data team says their data is trustworthy. Almost none of them can prove it to an AI agent that is about to build on it. The quality signals exist — assertions, ownership, documentation, freshness — but they sit scattered across the catalog, and the dashboard that adds them up is the one nobody opens. So agents keep building on data that nobody vouched for.

DataHub wants to be the universal registry where an organization records what it knows about its data. That felt like the right place to put a verdict an agent could act on, instead of one more read-only dashboard.

What it does

TrustBoard scores how trustworthy each data team's data is, writes that score back into DataHub as first-class metadata, and lets the next agent read it over MCP before it acts.

One thing to be straight about, because the word "agent" is doing a lot of work in this competition. Four of the five components make no model call at all. They are deterministic functions with role names, and the weekly pipeline is four sequential steps rather than a planner. Calling that multi-agent would be padding.

  • The Auditor walks every dataset by domain and computes a composite Trust Score from four signals: quality, documentation, ownership and freshness. Each signal has a preference order and reports which source answered. A signal that could not be read drops the dataset from the audit rather than being scored as absent, and a run that loses more than 20% of the graph refuses to publish at all rather than ranking whoever answered.
  • The Scribe writes the verdict back to the graph: four structured properties on each domain and on every dataset, a Gold/Silver/Bronze/At-Risk/Unrated tag, a scorecard inside the domain description, and a real DataHub incident opened on datasets that are dragging a team down, resolved automatically when the data recovers. Every write is idempotent; a second run moves nothing.
  • The Herald builds the weekly ranking against last week and posts it to Slack as a sports scoreboard.
  • The Gatekeeper consults the score before using a dataset. It spawns the TrustBoard MCP server as its own process and calls is_trustworthy over the protocol. The only TrustBoard module it imports is the MCP transport itself: no scoring code, no lookup module, no shared database. Its verdict has three outcomes, not two: below the bar, unrated, or not in the graph at all.
  • The Navigator is the one component that reasons. It is handed a task in plain English and nothing else: no URN, no shortlist, no expected answer. It searches the catalog, asks TrustBoard about candidates over MCP, and decides. On the run recorded in examples/navigator_run.txt it searched five times, adapting its vocabulary when the catalog did not use the words the task did, found that the obvious table rated at-risk (39.75), went looking for the same entity on other platforms, found four of them, compared their scores and substituted a gold one. Then it wrote the refusal back onto the rejected dataset as an incident, so the team that owns it learns that real work did not get built on their data, and why.

That last step is the loop closing. TrustBoard wrote a score; a separate process read it over MCP and made a decision; the decision went back into the graph. Graph to agent to graph.

How the score works, and why it is hard to game

A weighted average of four components, each 0-100: quality 35%, documentation 25%, ownership 20%, freshness 20%. Tiers: gold 80+, silver 60+, bronze 40+, at-risk below 40, plus unrated.

Unrated is not a bad grade. It means TrustBoard could not judge the asset, which is a gap in the catalog rather than a finding about the data, and every consumer of the score can tell the two apart.

The interesting part is what happens when someone tries to cheat it:

  • Quality counts checks that pass, not a pass rate. Any rate pays a team to delete its failing checks. Under a rate, ten checks with one passing scores 10, and deleting the nine failures scores 100. Counting passing checks against a breadth target means deleting a failure moves the score by exactly zero. The failure is priced as an incident instead. This is pinned by an exhaustive test over a 13x13 grid: no action that removes a check may raise the score, and none that adds one may lower it.
  • Unrated datasets stay in the team denominator. Excluding them meant a team with six healthy and six broken tables could delete the assertions on the broken six and jump from 50 to 100. Now hiding a table costs the same as leaving it broken.
  • Quality is required. Perfect documentation, ownership and freshness with nothing ever checking a row does not reach gold; it reaches unrated. Catalog tests are accepted as a fallback but capped at 60%, so metadata alone cannot buy a top tier.
  • The gate recomputes the tier from the score rather than reading the tier string, because structured properties and tags are editable by anyone with catalog permissions. Recognising that your own written output is an untrusted input.
  • Leaving a domain does not pay. A dataset unassigned from a domain is still counted against the team that let it go, using a persisted weekly roster.

Every threshold above is served by the API at /api/model and printed into every scorecard, so the rules that govern a badge are visible to the people being ranked by them instead of buried in the source.

How we built it

Python for the agents and scoring, using the acryl-datahub SDK to read eleven aspects and GraphQL to write structured properties, tags, descriptions and incidents. FastAPI plus SQLite for the weekly history, and a Next.js dashboard for the league. The MCP server is FastMCP over stdio, exposing five tools: get_trust_score, is_trustworthy, get_team_leaderboard, find_datasets and record_refusal. The first four are read-only; the last is the one that writes, annotated as such so a client can gate it behind approval. Each publishes an output schema, and a failure arrives as a proper MCP error with an actionable message rather than as a successful result the caller has to parse for bad news.

Any agent can register it:

pip install -e .
claude mcp add trustboard -- trustboard-mcp

DataHub runs from the official quickstart with the showcase-ecommerce datapack. 113 tests and CI, which also builds the backend image and curls it, because tests and build both passed once while an endpoint 500'd in production.

An honest note on the demo. The showcase datapack ships with almost no quality signals, so a seed script writes them first: ownership, descriptions, glossary terms, timestamps and test results, following a health profile per team. The inputs are staged so there is something to score. The scoring and the write-back are not, and pointing the same pipeline at an instance with real assertions scores it unchanged. The hosted board serves a saved snapshot of one real run; this week is measured, and the three earlier weeks in every trend line are authored demo history, drawn dashed and labelled as such.

Challenges we ran into

  • Structured properties on domain entities are barely documented. A day-one probe confirmed the write path before anything was built on it.
  • The DataHub CLI fails on Windows because it parses a drive letter like C: as a URI scheme, so loading the demo datapack broke. We fixed it and sent the fix upstream with a regression test.
  • Idempotency was the hard part. The first seeder skipped emitting absent signals, so metadata only ever accumulated, and it indexed datasets by search order, which shifts once tags are written. Both made scores drift on every re-run.
  • Every anti-gaming fix opened a new hole. Requiring quality made a coverage floor necessary; excluding unrated assets from the aggregate created the go-dark exploit; keeping them created a dilution problem for new teams. The comments in scoring/trust_score.py read as a changelog of self-inflicted bugs, each with the test that killed it.
  • Making the consuming agents genuinely independent. Reading the score in-process would have been simpler and proved nothing about interoperability, so both go over MCP as separate processes.

What we learned

Writing back to the graph is what separates a governance tool from a dashboard, and it is also where all the care goes: idempotency, remove-before-add on tags, delimited blocks inside descriptions, querying active incidents before raising a new one.

The bigger lesson was about honesty in a metric. Distinguishing "we measured this and it is in poor shape" from "nobody has attached enough signal to judge it" is the difference between a governance tool a team keeps and one they switch off in week two, because collapsing the two blocks the first asset anyone points it at for the crime of being new. And a score that can be gamed will be gamed, so publishing the gaming rules alongside the score is not a weakness, it is the only thing that makes the number worth arguing about.

What's next

Point it at a catalog with real assertions rather than seeded ones. Move the weekly run from a manual command to a scheduler or a DataHub Action driven by the metadata change log. Make the score unforgeable by requiring that it was written by the Scribe's own service principal, so a hand-typed structured property cannot open the gate.

Open source contributions

  • datahub#18479: a fix for the Windows drive-letter bug in the DataHub CLI, with a regression test.
  • datahub-skills#39: the scoring and write-back pattern packaged as a reusable datahub-trust-score skill.

Built With

Share this project:

Updates