Inspiration
AI agents now write the SQL, build the pipeline, and train the model — but nobody audits whether the model they ship is trustworthy, or just an overfit lie that looks great on paper. DataHub tells you if the data is fresh and complete; nothing tells you if a number built on it — "95% accurate", "+40% ROI" — is honest. That gap is where careers and money die: a random train/test split masquerading as walk-forward, a leaked target, an overfit curve that evaporates in production.
What it does
Legibright is an autonomous DataHub agent that audits statistical honesty — temporal/target/group leakage, overfit, calibration — and writes its verdict back into DataHub: an Assertion, an ACTIVE Incident, Tags, a typed 0–100 Trust Score property, and a deprecation proposal (agent proposes, a human approves). Other agents call it over the MCP Server; it installs as a 6th DataHub Skill beside the official five.
Closed loop, no human in the middle until a decision is queued: it reads the dataset, lineage, and query history over MCP → runs the honesty checks → acts on the graph → hands off to the next engineer and to other agents.
Trust ≠ accuracy. Legibright scores honesty, not performance. A bike-demand model that's only ~57% accurate but honestly 57% earns Trust Score 100; a "perfect" revenue forecaster scoring R² 1.00 on a leaked split earns 28. It rewards the modest-but-honest number and punishes the impressive lie.
The demo (live, on real data)
On 604 days of real e-commerce revenue (UCI Online Retail II, CC BY 4.0), Legibright audits a forecaster built the standard naive way (a default DecisionTreeRegressor, sklearn's default random train_test_split) and, in one run:
- Catches the overfit: a perfect R² 1.00 in training collapses to R² −0.05 on genuinely unseen days — worse than guessing the average.
- Names the cause: the split was random, not chronological — the model trained on days inside the test period (the fingerprint of a random split masquerading as walk-forward).
- Verdict: NOT TRUSTWORTHY, Trust Score 28/100 — written into DataHub as 2 assertions + 1 incident + tags + a Trust Score property + a deprecation proposal.
Then it proves it doesn't cry wolf: an honest Bike Sharing model passes 100/100, a leaky Titanic model fails.
How we built it
Python 3, built on DataHub's MCP Server (FastMCP) + DataHub Skills + acryl-datahub (SDK 1.6). Honesty checks use sqlglot (temporal leakage) and numpy/scipy (overfit collapse, sample-size-aware Hosmer–Lemeshow calibration). Verdicts are written back via native Assertion / Incident / Tag / structured-property entities. It extends DataHub's Data Quality Agent pattern — not a rewrite.
Challenges we ran into
The hardest part was our own credibility: a tool that audits other people's honesty has to survive the same scrutiny. So we ran a 3-round adversarial self-audit (a separate clean-context agent trying to break it each round) plus a fresh delta pass. We found 15 real bugs in our own code and fixed every one — including a false-negative in the calibration check and an over-claim in how we described the leak. Nothing was loosened to go green.
Accomplishments we're proud of
- 15 bugs found in ourselves, and fixed — the one honesty tool that survives its own honesty test.
- 53/53 unit tests + a 16-check adversarial suite, deterministic idempotent write-back, judge-verifiable from a clean clone.
- Two upstream open-source PRs (a new DataHub skill + a docs fix) and an RFC.
What we learned
"Trustworthy" and "accurate" are different questions — and the second one hides the first. Auditing yourself, assuming your own tool is wrong until proven otherwise, is exactly what a trust tool should model.
What's next
Auto-feed the target/group/calibration checks from lineage; a Trust Score trend over time; deeper composition so a low Trust Score can gate a downstream agent's action.
Try it in 60 seconds
git clone https://github.com/bogacsmz/legibright-trust-audit && cd legibright-trust-audit pip install -e '.[dev]' python scripts/fetch_data.py python scripts/verify_all.py # 15 adversarial checks, no DataHub needed bash demo/run_demo.sh # full demo with DataHub up
Repo (Apache-2.0): https://github.com/bogacsmz/legibright-trust-audit Open-source PRs: datahub-skills #30 · datahub #18272
Honest limitations
The auto-fed verdict covers split leakage; target/group/calibration checks are opt-in. Calibration has a ~5% nominal false-positive rate by construction. Full ledger in docs/VERIFICATION.md — 15 flaws found + fixed, 6 limits documented, nothing loosened to go green.
Log in or sign up for Devpost to join the conversation.