Lineage - DataOps Memory Agent

How we built it

The agent is orchestrated by OpenClaw, using a native Codex app-server runtime.

Codex can access two MCP servers:

• The official DataHub MCP server for catalog search, schema inspection, lineage traversal, ownership, documents, and approved writeback.

• A custom Agentix Memory MCP server for storing incidents, recalling related episodes, examining memory state, and recording observed outcomes.

The memory plane has three complementary layers:

• A raw episodic store preserves the original evidence and outcomes.
• A provenance knowledge graph connects incidents, assets, schema changes, actions, and impacted entities.
• Hierarchical recurrent states summarize history across organizational scopes.

The hierarchy follows a structure such as:

Organization └── Domain └── Data product └── Asset └── Incident

Each incident updates its asset node and the relevant ancestors. A dependency-light, LSTM-style recurrent cell maintains compact temporal state at every level, while the original episodes remain available for inspection and auditability.

Recall uses a deterministic ranking baseline:

[ S = 0.55S_{\text{semantic}} + 0.15S_{\text{hierarchy}} + 0.15S_{\text{confidence}} + 0.10S_{\text{outcome}} + 0.05S_{\text{recency}} ]

When an episode is recorded, its normalized DataHub context is fingerprinted. During recall, Lineage fingerprints the current catalog evidence again:

• Matching fingerprints mean the memory is current.
• Changed fingerprints mean it is stale.
• Missing evidence makes it unverified.

For stale memories, Lineage produces explicit differences across schemas, owners, upstream and downstream lineage, and quality signals.

## DataHub integration

Our live demonstration uses DataHub’s official showcase-ecommerce and bootstrap datapacks.

The workflow inspected the live dbt orders asset and found:

• 15 schema fields
• 4 current owners
• 2 upstream relationships
• 1 downstream relationship

The recalled incident referenced both a removed legacy_discount_code field and a former legacy-orders-team owner. Lineage therefore marked the historical advice as stale and assigned it a high catalog-difference risk instead of replaying it blindly.

After approval, the generated incident decision was published to DataHub and retrieved using its exact document URN, proving that the knowledge was available for inheritance.

## Challenges we faced

### Separating current truth from historical memory

The hardest conceptual problem was deciding which system had authority. Combining everything into one vector store would have blurred the difference between current catalog facts and historical experience.

We kept them as separate truth planes: DataHub is authoritative for the present, while memory remains advisory until revalidated.

### Normalizing rich lineage responses

DataHub MCP lineage responses contain nested metadata, not only lineage nodes. Without careful normalization, owner URNs or other related entities could be misclassified as impacted datasets. We introduced typed extraction and canonical evidence snapshots to keep those relationships distinct.

### Making stale memory explainable

A simple “similarity score” was not sufficient. The agent needed to explain exactly why an old solution might no longer be safe. We implemented field-level and relationship-level comparisons so the report can identify removed fields, type changes, ownership changes, and lineage drift.

### Verifying asynchronous writeback

DataHub search indexing is asynchronous. Instead of assuming a successful write meant the document was immediately discoverable, the workflow polls for the exact saved document URN and records whether inheritance was verified.

### Preserving safety

The default workflow is read-only. Catalog mutation requires explicit approval, and reports distinguish current DataHub evidence, recalled memory, inferred recommendations, and observed outcomes.

## What we learned

Agent memory needs more than a larger context window or vector database. Useful institutional memory requires:

• Clear authority rules
• Evidence provenance
• Freshness validation
• Inspectable differences
• Recorded outcomes
• Explicit mutation boundaries

We also learned that recurrent state and knowledge graphs solve different problems. Recurrent state compresses temporal patterns across a hierarchy, while the graph preserves explicit relationships and provenance. Neither should replace the raw incident evidence.

## Accomplishments

• Completed an end-to-end DataHub read, recall, reconciliation, report, approval, writeback, and inheritance- verification flow.

• Connected a real OpenClaw/Codex run to both MCP servers.

• Produced field-level stale-memory explanations.

• Added idempotent DataHub document updates.

• Implemented a read-only default with an explicit approval gate.

• Passed all 8 automated tests.

• Achieved 3/3 top-1 retrieval and 3/3 stale/current classification on our deterministic synthetic regression benchmark.

The benchmark validates the software contracts, not production incident-resolution quality; broader real-world evaluation remains future work.

## What’s next

We plan to:

• Train learned recurrent gates and compare them against the deterministic baseline.
• Add semantic embeddings and a larger labeled incident benchmark.
• Connect approved actions to DataHub assertions and incident-management APIs.
• Build a compact review interface for approvals and field-level differences.
• Package the workflow as a reusable DataHub skill.
• Evaluate multi-agent inheritance across longer, real-world incident histories.

Lineage’s goal is straightforward: remember what worked, verify what is true now, and leave better context for whoever investigates next.

Built With

• ai-agents
• data-lineage
• datahub
• dataops
• docker
• generative-ai
• hierarchical-memory
• incident-response
• javascript
• knowledge-graphs
• lstm
• mcp
• metadata
• model-context-protocol
• node.js
• openai-codex
• openclaw
• sqlite