-
-
Synthetic, PHI-free trauma-registry pipeline in DataHub: raw ED docs → staging → trauma registry → de-identified research export.
-
The agent flags patient_id in the de-identified research export as a DirectIdentifier — a PHI leak found from metadata alone, no data read.
-
Closed loop: the agent writes back a governance incident, tag, term, and plain-language finding with the fix—drop patient_id from the export
I'm an orthopaedic surgeon who builds clinical software. The failure I've watched break registries isn't a dramatic one — it's a schema migration that half-lands.
Someone renames a column. The rename ships. And three quiet things go with it: a direct identifier survives into the "de-identified" research export, a downstream transform collapses distinct documented reasons for a missing value into a single indistinguishable null, and an ML feature quietly starts drawing from raw un-validated data instead of the governed registry.
None of that is a bug in any one query. Every model builds. Every test passes. The damage is in the contracts between the models — and contracts are metadata, which is exactly what a metadata graph knows and a SQL linter doesn't.
DataHub already answers "what breaks if I change this column?" What it can't know is that in a trauma registry, a governed absence is not the same as a null, and that a research export carries a de-identification promise. Those rules are clinical, not generic. So I built the layer that holds them.
What it does
OrthoLineage Guardian reads a DataHub context graph over the DataHub MCP Server and detects three healthcare-specific data-contract violations that DataHub does not ship checks for:
PHI_EXPORT_PATH — a column classed as a direct identifier has a column-level lineage path into a dataset marked as a research export with a de-identification requirement. MISSINGNESS_COLLAPSE — a column under an explicit-missingness contract loses its paired state column downstream, so a documented "not assessed" becomes indistinguishable from a bare null. UNVALIDATED_ML_SOURCE — an ML feature bypasses the validated registry and draws straight from an unvalidated source. (Naively, every column descends from the raw source, so the discriminating signal is the bypass: unvalidated source present in the column's upstream set while the validated registry is absent from that path.)
It then reports the full downstream impact of the migration, flags a stale post-rename reference, generates a real, git apply-able dbt patch plus a migration-impact report, and writes its findings back into DataHub as a governance tag, a dataset description, and an incident per finding — so the next person or agent inherits the knowledge instead of rediscovering it.
Everything runs on synthetic, PHI-free trauma-registry data.
The safety property is the point. Deterministic code decides whether a violation exists. The language model never decides — it takes a lineage-grounded finding and turns it into an explanation and a draft remediation, which is Pydantic-validated, entity-whitelisted, and contract-checked before anything is used. Write-back is performed by validated application code, never by the model. The agent governs data systems; it never diagnoses, treats, or triages.
How I built it
A synthetic six-model dbt-duckdb pipeline mirrors a real registry path: ED documentation → trauma registry → data-quality metrics and de-identified research export → ML feature table → dashboard. One model tree, two worlds selected by a deterministic dbt variable — a clean baseline and a faulty migration carrying four deliberately planted, in-code-annotated defects.
Both worlds are ingested into DataHub OSS under distinct URN namespaces (env=PROD for faulty, env=DEV for baseline). The dbt ingestion source supplies schemas, column-level lineage, and ownership; a custom clinical emitter adds the governance layer — glossary terms at column granularity, structured properties at dataset granularity.
The agent reads exclusively through the DataHub MCP Server. A deterministic policy engine runs the three checks and the impact traversal. The LLM layer explains and drafts. An artifact renderer produces the patch, the report, and machine-readable findings. Controlled write-back — dry-run by default, idempotent, with a reset command — closes the loop through the DataHub Python SDK.
A guardian CLI wraps the whole thing; scripts/demo.sh runs the hero path end to end.
Challenges I ran into
Column-level signals are nearly invisible to MCP. I spiked all three representations against both granularities before building anything on them. Structured properties and tags applied to schemaField entities are simply not returned by MCP's field view — column signals are readable only as glossary terms attached through editableSchemaMetadata. I found this because I insisted on verifying every emitted signal by reading it back through MCP rather than through the SDK that wrote it. Had I verified the easy way, I'd have built the entire policy engine against signals it could never see.
Proving the absence of false positives. Any checker can fire. The hard claim is that it stays silent when it should. Testing the clean scenario through DuckDB would have violated my own rule that the agent reads metadata only — so I put both worlds into the graph under separate namespaces. The same engine, the same contract, the same code path, pointed at the clean world, returns zero findings.
Guarding the model without crippling it. My first live run against a real model rejected two of four calls. Neither was hallucination: the entity whitelist had atomized "e.g." into the tokens e and g and rejected them as unknown entities, and a long remediation had hit the token ceiling, surfacing as a misleading JSON error. I fixed the tokenizer and the token limit — never the whitelist, and never the guard. A fabricated table is still rejected. Accept rate went from 2/4 to 4/4.
The demo that only works on your own machine. Before submitting I cloned the public repo into a fresh directory with no environment file and followed my own README literally. Two silent defects surfaced: uv run doesn't auto-load .env, so a judge's API key would have been ignored and the run would have quietly stayed in template mode; and no CLI subcommand actually invoked the model, so the LLM path was unreachable through the interface I'd documented. Both were invisible from inside my working directory.
Accomplishments I'm proud of
The patch is real. git apply --check exits 0 — it's a diff computed from the actual model files that removes all four planted defects, not an illustrative snippet.
The safety claim is machinery you can run, not a paragraph in a README. There's a test that installs a CPython audit hook, records every file opened during a real engine run, and asserts that no .sql file, dbt manifest, catalog, or database was touched — plus a non-vacuity check that the run still produced findings. There are tests proving that a fabricated table name, a relabelled check, an invented finding, and a claimed data observation are each rejected.
And it works from a clean clone with no API key at all. The model improves the prose; it is never required for correctness.
What I learned
Prove the primitives before you build on them. Two short capability spikes — one for write-back, one for metadata representation — each cost an hour and each prevented a multi-day rebuild.
Verify on the path that will actually consume the data. A signal that is writable but not readable where it's needed is not a signal.
Build the negative case as machinery. "No false positives" as a sentence is worth nothing; as a second namespace in the graph, it's a demo.
And run your own instructions on a machine that knows nothing about your project. Every accumulated convenience in a working directory is a defect waiting for a stranger.
What's next
Deriving the governance contract from the registry data dictionary rather than hand-authoring it; expanding to the full two-axis missingness vocabulary real registries use; opening pull requests directly rather than emitting a patch file; and a maturity ladder in place of the binary validation status.
Built with
python · dbt · duckdb · datahub · datahub-mcp-server · model-context-protocol · pydantic · anthropic-claude · uv · pytest · docker · github-actions
Links Repository: https://github.com/orthopendar/ortholineage-guardian (public, Apache-2.0) Demo video: https://www.youtube.com/watch?v=hF-s5Sk5uxQ
Note for judges
All data is synthetic and PHI-free. The project governs metadata, provenance, and data quality only — it never produces clinical advice. README.md includes a section on how to verify each claim yourself, and examples/ contains generated artifacts you can inspect without running anything.
Log in or sign up for Devpost to join the conversation.