RuleCraft

Inspiration

An experiment can improve overall while regressing on a critical task. A run log alone cannot tell whether that regression reaches a benchmark, dashboard, model release, or another team's decision. RuleCraft uses DataHub's metadata and lineage graph to make that organizational blast radius actionable.

What it does

RuleCraft's signature feature is the Counterfactual Experiment Blast-Radius Gate. It reads live experiment context from DataHub, traverses transitive downstream lineage, scores the risk inherited by every affected asset, produces owner-specific remediation, applies a deterministic release gate, verifies the decision, and writes durable decision memory back to DataHub.

The successful Live DataHub run at 2026-08-08 16:02 KST produced:

• 5 entities inspected
• 3 lineage edges traversed
• 3 impacted assets
• 3 detected risks
• 3 owner-routed remediations
• Maximum risk: 95/100
• Release gate: blocked
• Deterministic verification: PASS
• DataHub write-back: PASS

The impacted assets were ARC Hard Benchmark (95, eval-team), Research Leaderboard (80, research-ops), and Solver Production Release (90, ml-platform).

How we built it

RuleCraft is implemented in Python with a small context-graph interface and a production DataHub adapter. The adapter reads metadata properties and downstream lineage from DataHub, while Metadata Change Proposals persist findings, decision tags, maximum risk, impacted-asset count, verification state, and remediation notes.

The gate is deterministic rather than an unverifiable LLM claim. Risk combines measured regression severity, lineage depth, and asset criticality. High-risk regressions block promotion; lower-risk regressions require approval; candidates without material regressions can proceed.

Live DataHub proof

We ran the complete flow against a healthy local DataHub Quickstart:

experiment change → DataHub metadata read → downstream lineage traversal → blast-radius scoring → blocked gate → owner remediation → deterministic verification → DataHub write-back.

The write-back was independently queried again through the DataHub API and confirmed in the UI as blocked / 95 / 3 / pass, together with the rulecraft-blocked tag and the three remediation actions.

The first attempt encountered normal lineage-index propagation delay and is excluded from the successful metrics. Fixture and in-memory output are never presented as live evidence.

Challenges

The main challenge was treating metadata indexing and lineage traversal as real distributed-system boundaries. We preserved the first index-delay observation, waited for DataHub's lineage index, reran the same workflow, and independently verified the persisted result instead of claiming fixture output as live evidence.

Accomplishments

• A memorable lineage-aware safety gate rather than a generic metadata chatbot
• Actual transitive DataHub lineage read and durable write-back
• Owner-specific remediation for every impacted asset
• Deterministic PASS verification and 10/10 automated tests
• Apache 2.0 public repository with machine-readable Live DataHub evidence

Links

• GitHub: https://github.com/rulecraft2026-dot/RuleCraft
• Final Live DataHub demo: https://youtu.be/05c3MZno0l0

What's next

The implementation is frozen for submission. Future work would focus on broader production integrations and operational policy configuration without changing the verified core workflow.

Live DataHub screenshots

Downstream lineage

Blocked decision and owner remediation

Persisted DataHub properties

Built With

• ai-agents
• datahub
• lineage
• machine-learning
• metadata
• python