DataHub Reflex
Inspiration
Data incidents are usually fixed once and forgotten.
An engineer finds the cause, patches the pipeline, closes the incident, and writes a post-mortem. But the lesson usually remains trapped in a ticket, a Slack thread, or someone’s memory.
Weeks later, another team makes the same mistake on another dataset.
We asked ourselves:
What if a resolved incident could become a reusable organizational capability?
DataHub already contains the relationships needed to understand where a failure can spread: lineage, ownership, domains, tags, assertions, incidents, and related assets.
Instead of using DataHub only to investigate what happened, we built DataHub Reflex to make the organization learn from what happened.
Reflex treats every confirmed incident lesson as a candidate “organizational reflex”: a preventive control that can be tested, approved, reused, and propagated across similar assets.
The goal is simple:
Make every resolved incident make the data platform harder to break twice.
What it does
DataHub Reflex converts a resolved data incident into a tested preventive control.
The workflow begins with a real incident stored in DataHub. Reflex never treats an AI-generated explanation as authoritative. A human must confirm or edit the root cause first.
Once approved, Reflex:
- Extracts a structured failure pattern.
- Identifies similar assets through DataHub metadata and lineage.
- Explains why each asset was selected.
- Generates a typed preventive control.
- Backtests the control against historical snapshots.
- Reports precision, recall, false positives, and coverage.
- Requests explicit human approval.
- Publishes the approved lesson and coverage metadata back to DataHub.
- Executes the control on a later analogous failure.
- Raises a new incident when the same pattern appears elsewhere.
The MVP demonstrates two forms of organizational learning:
- Duplicate financial rows caused by non-idempotent pipeline retries.
- Orphaned ownership after an employee leaves the organization.
The key demonstration is not that Reflex can describe an incident.
It is that incident A produces a control that detects incident B on another asset.
Reflex is not self-healing and does not silently modify production systems. Every generated control and ownership change requires human approval.
How we built it
DataHub Reflex is built around a clear separation of responsibilities.
DataHub OSS acts as the organizational graph. It stores and exposes:
- Incidents
- Dataset and column-level lineage
- Ownership
- Domains
- Tags
- Structured properties
- Assertion definitions
- Assertion run events
- Asset relationships and provenance
Reflex provides the capabilities that are not available as a complete end-to-end loop in DataHub OSS:
- Human root-cause approval
- Structured lesson extraction
- Similar-asset resolution
- Deterministic control generation
- Historical backtesting
- Control execution
- Evaluation metrics
- Approval workflow
- Recurrence detection
The system includes two typed controls:
UniquenessControlActiveOwnershipControl
The LLM proposes a structured lesson and candidate control. It does not directly execute arbitrary code or write uncontrolled metadata. Final controls are validated and compiled through deterministic templates.
For example, a duplicate-row lesson becomes:
type: uniqueness
field: transaction_id
max_duplicate_count: 0
The control is executed by the Reflex backtester against synthetic historical runs containing both normal and known-failure snapshots.
A similarity resolver uses inspectable signals such as:
- Shared domain
- Compatible schema
- Common key fields
- Similar lineage
- Common ingestion framework
- Existing control coverage
DataHub OSS stores the resulting lesson, control identifiers, coverage status, incidents, ownership changes, and execution results.
Because native assertion execution is not available in DataHub OSS, Reflex explicitly owns control execution and backtesting rather than presenting a Cloud-only capability as an OSS feature.
Challenges we ran into
The first challenge was technical honesty.
DataHub has powerful assertion and observability concepts, but assertion execution is not available in the same way in DataHub OSS. We had to inspect the repository and design the architecture around the real OSS boundary.
That led to an important decision:
DataHub stores the organizational knowledge and provenance. Reflex executes and evaluates the generated controls.
The second challenge was preventing the project from becoming an “LLM plus metadata” wrapper.
A natural-language recommendation is not enough. Reflex must produce an executable artifact and demonstrate that it works on a second asset.
The third challenge was generalization.
A lesson from one dataset should not automatically become a global rule. Reflex explains which signals support propagation and allows the human to approve the intended scope.
The fourth challenge was governance.
A system that automatically changes ownership or publishes controls would be dangerous. Reflex keeps human approval at two critical points:
- Confirming the root cause.
- Approving the generated protection.
Finally, we had to make the demo reproducible. The entire environment, historical snapshots, incidents, approvals, and future failures are generated from scripts so the judge can inspect the complete chain.
Accomplishments that we’re proud of
We are proud that DataHub Reflex demonstrates a complete closed loop:
Resolved incident
→ human-confirmed lesson
→ generated control
→ historical backtest
→ human approval
→ DataHub write-back
→ analogous incident detection
This is different from a typical metadata assistant, which stops after answering a question.
Reflex produces measurable and inspectable results:
- A structured lesson
- A typed preventive control
- Backtest evidence
- Similar-asset explanations
- Approval provenance
- DataHub coverage metadata
- A later detected incident
We are especially proud of the distinction between what belongs to DataHub and what belongs to Reflex.
DataHub provides the graph that makes propagation possible. Reflex provides the learning and execution loop that turns an incident into reusable protection.
The two scenarios demonstrate different forms of learning:
- Technical learning: detecting duplicate rows caused by retry behavior.
- Organizational learning: detecting assets whose operational owners are no longer active.
The project also preserves important safety properties:
- No autonomous production remediation
- No hidden approval bypass
- No unsupported Cloud-only claims
- No opaque similarity-only decisions
- No root-cause authority without human confirmation
What we learned
We learned that “AI learns from incidents” is only meaningful when the learning becomes executable.
A summary, post-mortem, or embedding is not enough. The lesson must become a control that can be:
- Executed
- Backtested
- Measured
- Reviewed
- Versioned
- Propagated
- Audited
We also learned that DataHub’s value is not limited to catalog search. Its real strength is the combination of relationships: assets, owners, domains, lineage, quality signals, incidents, and governance metadata in one graph.
Those relationships allow Reflex to move from:
What happened to this dataset?
to:
Where else could the same failure happen, and can we prove that our protection works there?
Finally, we learned that technical credibility matters more than ambitious language. Reflex does not claim to be fully autonomous or self-healing. It focuses on a narrower but verifiable capability: converting human-approved operational knowledge into reusable controls.
What’s next for DataHub Reflex
The next step is to expand the library of incident-to-control transformations without weakening the approval and evaluation model.
Planned capabilities include:
- Breaking schema-change protection
- Freshness and volume anomaly controls
- Cross-domain KPI consistency checks
- Data contract evolution rules
- Ingestion idempotency verification
- Ownership lifecycle monitoring
- Reusable DataHub Agent Skills generated from approved lessons
- Integration with warehouse query history and pipeline execution logs
- Pull-request generation for approved controls
- Versioning, rollback, and control lifecycle management
- Evaluation against real historical incident data
The long-term vision is not another chatbot inside a data catalog.
It is a learning layer for the data organization:
DataHub records how the organization is connected. Reflex records what the organization has learned, tests that lesson, and makes it reusable.
Log in or sign up for Devpost to join the conversation.