Inspiration
Production ML systems can fail without producing a clear error. A schema change may be valid, pass normal tests, and still silently break a feature pipeline or degrade a model several hops downstream. We built Faultline around one question: what if an agent could catch that risk at the pull request, investigate failures when they happen, and preserve what it learns for the next engineer or agent?
DataHub already contains the lineage, schemas, ownership, and asset relationships needed to answer these questions. Faultline turns that metadata graph into an active control system for production ML. It addresses both hackathon themes by protecting live models and by performing real investigative work that produces durable results.
What it does
Faultline provides three connected workflows.
Blast Radius analyzes proposed schema changes before merge. In our demo, a pull request renames billing_amount to amount_billed in the upstream raw_patients table. The SQL is valid, but DataHub lineage shows that the change affects seven downstream entities: two feature tables, three ML models, and two deployments. Faultline applies a deterministic policy and returns a blocking decision with supporting evidence.
Root Cause investigates silent model degradation by checking prior findings, following lineage, reviewing freshness and schema events, and inspecting source-data evidence. In the demo, it identifies 277 patient records where admission and discharge dates are reversed, creating negative length-of-stay values that corrupt a downstream feature without raising a schema error or application exception. The agent produces a ranked RCA for human review, and the approved finding is written back into DataHub.
Readiness Gate evaluates whether a model is ready to ship. It checks feature freshness, schema compatibility, ownership, lineage completeness, and quality evidence. In our demo, feature freshness is 42 minutes against a 30-minute threshold, producing a clear NO-GO decision with remediation evidence.
How we built it
Faultline uses a Python and FastAPI agent service to coordinate DataHub, GitHub, policy evaluation, and human review. The interface was built with React and TypeScript, and the system uses the DataHub MCP Server, DataHub Agent Context Kit, GitHub Actions, Claude, REST APIs, JSON, and Render.
The agent uses DataHub tools to search for assets, retrieve schemas, follow lineage, and confirm exact downstream paths. We deliberately separated agent-generated interpretation from operational decisions. The agent gathers and organizes evidence, the policy layer produces deterministic verdicts, the engineer reviews generated RCA content, and DataHub stores the approved result.
Each run preserves an auditable record containing the trigger, run ID, evidence source, policy version, verdict, lineage paths, tool calls, logs, lifecycle events, and write-back links.
Challenges we ran into
One of the hardest challenges was mapping GitHub code changes to the correct DataHub entities. Pull requests use table and column names, while DataHub represents assets through metadata identifiers and relationships. We needed normalization and fallback logic to connect those systems reliably.
Another challenge was preserving exact lineage paths. A high-level dependency was not enough to justify a blocking decision. Faultline needed to retain the full path from the changed upstream column to affected features, models, and deployments.
We also had to balance agent flexibility with production controls. Language models are useful for investigation and explanation, but release decisions must remain predictable. This led us to separate evidence collection, deterministic policy, human review, and write-back.
Finally, the demo environment did not always contain complete live production metadata. We added clear evidence-source labels so controlled fixtures were never presented as live DataHub evidence.
Accomplishments that we're proud of
We are proud that Faultline connects prevention, investigation, readiness, and organizational memory in one workflow.
It can detect a risky schema change at the pull request, trace its downstream impact through DataHub, apply a deterministic policy, investigate a separate silent data-quality failure, support human review of the RCA, and write governed findings back into the metadata graph.
The most important accomplishment is that Faultline does not stop at generating an alert. It creates durable context that the next engineer or agent can reuse without repeating the entire investigation.
What we learned
We learned that metadata becomes much more valuable when agents can act on it instead of only displaying it.
Data lineage can support change control, release policy, incident investigation, model readiness, and organizational memory. We also learned that trustworthy production agents need more than an LLM. They require reliable context, explicit tools, provenance, deterministic controls, human review, and durable outputs.
The strongest architecture was not an agent that tried to decide everything independently. It was an agent that combined graph context, tool-based evidence, explicit policy, and human accountability.
What's next for Faultline
Next steps include native GitHub status checks and review comments, support for additional schema-change types, expanded column-level lineage, configurable policy packs, automated owner notifications, reuse of historical incidents during new investigations, and deeper integration with DataHub assertions, ownership, and incidents.
Our longer-term goal is to make Faultline a reusable control layer for production ML: catching risks before deployment, accelerating investigations when failures occur, and continuously enriching DataHub with reviewed knowledge.
Log in or sign up for Devpost to join the conversation.