-
-
architetural design for Cortex
-
promoted lesson
Inspiration
Data incidents are rarely difficult because engineers don't know how to fix them. They're difficult because the investigation has to be repeated.
When a pipeline breaks, an engineer inspects the failing dataset, traces its lineage, compares schemas, checks upstream dependencies, searches previous incidents, determines the root cause, and finally figures out whether someone has already solved the same problem before.
Organizations accumulate this knowledge over time, but it is scattered across metadata catalogs, incident reports, documentation, logs, and the engineers who investigated them.
I wanted to change that.
I built Cortex, a solo exploration into whether an AI agent could investigate data incidents, learn from previous investigations, and reuse that knowledge when similar failures occur.
But I identified a fundamental problem with simply giving an agent a vector database of previous incidents:
A similar incident does not necessarily mean the same solution is still correct.
The data environment may have changed.
That led me to build Cortex around one core idea:
Self-Verifying Memory.
Cortex doesn't blindly trust what it remembers. It retrieves a previous experience, one which comprises of both successful and unsuccessful fixes, checks it against the current state of the data environment, and only reuses it when the evidence supports doing so.
What it does
Cortex is an AI-powered data incident investigation and reliability platform.
When an incident occurs, Cortex:
Identifies the affected dataset.
Retrieves similar historical incidents from its episodic memory (ChromaDB).
Inspects the dataset's current schema, lineage, metadata, and persistent freshness through DataHub.
Compares the current environment with the historical experience.
Determines whether the previous solution is still applicable.
Reuses the experience when it is valid, or investigates the incident from scratch when it isn't.
Collects evidence and determines the likely root cause.
Generates a proposed remediation.
Keeps a human in the loop before consequential actions.
Verifies the outcome and stores the investigation as a new experience.
Writes successful, generalized lessons back to DataHub—turning it from a read-only catalog into an active knowledge base.
The result is a system that can progressively turn previous investigations into reusable operational knowledge.
How I built it
Cortex is built around two complementary forms of memory:
DataHub provides semantic memory of the data environment, exposing schemas, lineage, metadata, and persistent freshness information. It also serves as Cortex's long-term knowledge layer—successful investigations write generalized lessons back to DataHub, meaning the platform isn't just read-only; it accumulates organizational knowledge over time.
ChromaDB provides episodic memory, allowing Cortex to retrieve previous incidents and their outcomes as structured experiences rather than plain text.
LangGraph orchestrates the entire investigation workflow, managing state across retrieval, verification, evidence gathering, diagnosis, remediation, and learning.
Groq provides the LLM inference layer used for reasoning, diagnosis, and remediation generation. Its fast inference allows Cortex to reason over lineage and schema evidence without introducing frustrating latency during the investigation flow.
A key design decision was treating freshness as persistent environmental state rather than a transient alert. Because the ecommerce environment used in my demonstration is intentionally not continuously refreshed—it's a local instance that has become stale over time—Cortex observes the accumulated age of assets as part of the asset's state. This allows freshness to participate in incident classification and state comparison alongside schema and lineage, rather than being treated as a momentary failure signal. The system is therefore capable of working with real-world data that is not being actively maintained.
Cortex therefore creates a complete feedback loop:
Observe → Retrieve → Verify → Investigate → Fix → Learn → Write Back to DataHub.
The system doesn't just consume organizational knowledge; successful investigations actively contribute to the organization's metadata and future reasoning context.
Challenges I ran into
The hardest problem wasn't getting an LLM to produce an answer. It was making the answer trustworthy.
I had to determine how to represent the state of a dataset at the time an incident occurred, how to compare that state with the current environment, and how to decide whether an old experience should still be trusted.
Working with Stale Data
My demonstration environment is deliberately built on an ecommerce dataset that is already stale (showcase-ecommerce datapack). This wasn't a limitation—it was a design constraint that forced me to think differently.
In a production environment, a pipeline failure would trigger an alert, and the dataset would be actively monitored. In my case, Cortex observes persistent staleness as an environmental property rather than a transient incident signal.
This meant Cortex had to learn to work with what was available—observing the accumulated age of assets, comparing historical snapshots against currently available metadata, and reasoning about freshness as part of the asset's state rather than relying on real-time monitoring signals.
DataHub SDK Limitations
During development, I discovered that the DataHub Python SDK does not expose lineage operations directly through its Dataset representation. Lineage is treated as a separate concern in the GraphQL API.
I initially attempted to access lineage as a property of a Dataset object. When that failed, I had to pivot to querying DataHub's GraphQL endpoint directly.
This had two effects:
It forced me to design a clean abstraction layer for DataHub interactions—separating metadata retrieval from lineage operations.
It made the snapshot architecture stronger, because I could no longer assume relationships were direct dataset attributes.
I also encountered method limits when attempting to retrieve large lineage graphs. The SDK's query patterns don't always align with the investigation's need to explore upstream and downstream dependencies in a single pass.
I worked around this by implementing a lightweight caching layer and designing the lineage traversal to request only what was needed for the investigation at hand—rather than attempting to materialize the entire graph upfront.
Distinguishing Symptoms from Root Causes
Another challenge was distinguishing symptoms from root causes.
A schema change can cause a pipeline failure, which then causes a dataset to become stale. Simply observing the stale dataset could lead an agent to classify the incident as a freshness problem.
Cortex therefore prioritizes structural evidence—schema and lineage changes—when determining what actually caused the failure.
Integrating Multiple Systems as a Solo Builder
Building a system that integrates DataHub, ChromaDB, LangGraph, and Groq—with a self-verifying memory architecture—as a solo project meant every integration issue, debugging session, and architectural decision was my own.
This required disciplined scoping and a clear separation of responsibilities across components. I made every node in the LangGraph log what it did and why. The system is designed to be read from its logs rather than requiring a debugger—because as a solo developer, I couldn't afford to spend time guessing.
Accomplishments that I am proud of
I'm proud of building an end-to-end prototype where memory is not treated as truth.
Cortex can:
Retrieve previous incident experiences from ChromaDB.
Inspect the current DataHub environment, including persistent freshness.
Compare historical and current asset state.
Reason over schema, lineage, and freshness evidence.
Decide whether a previous solution should be reused.
Investigate new incidents when historical knowledge is no longer applicable.
Generate proposed remediation.
Keep humans in control of consequential actions.
Store successful investigations as future experiences.
Write generalized lessons back to DataHub, making the platform a learning system rather than a read-only catalog.
The biggest accomplishment is the self-verifying memory architecture.
Instead of:
"I've seen this before, so I'll use the same fix."
Cortex asks:
"I've seen this before. Does the evidence show that the same solution still applies?"
This is the difference between a lookup table and an evidence-backed reasoning system.
Building this as a solo project also means I owned every component—from the LangGraph orchestration to the ChromaDB integration to the DataHub write-back layer. The system is cohesive because there was no handoff between team members; every decision was made with the full architecture in mind.
What I learned
Building Cortex taught me that persistent AI memory is much more than storing conversations or embeddings.
A useful memory system needs to understand:
What happened.
Why it happened.
What the environment looked like.
What solution was used.
Whether that solution actually worked.
Whether the environment is still comparable.
I also learned that metadata platforms such as DataHub are extraordinarily valuable to AI agents because they provide something an LLM cannot reliably infer from text alone: a structured representation of the world.
Lineage gives Cortex relationships. Schemas give it structure. Freshness gives it time-awareness. Historical experiences give it precedent.
I also learned to work with platform constraints rather than fighting them. The DataHub SDK's separation of lineage from dataset metadata forced me to build a cleaner abstraction—and that abstraction now makes it easier to support other metadata sources in the future.
Most importantly, I learned that agent autonomy needs boundaries.
The goal isn't to let an AI blindly modify production systems. The goal is to let the agent perform the tedious investigation while keeping humans in control of consequential decisions.
What's next for Project Cortex V1.0
Cortex V1.0 is the foundation for a larger AI Data Reliability Engineer. The current implementation proves the architecture works. The next phase is about depth, richness, and autonomy.
Richer Incident Types
I want Cortex to handle a wider range of failures, including:
Schema drift.
Pipeline failures.
Data freshness violations.
Broken lineage.
Data quality failures.
Contract violations.
Transformation failures.
Each incident type requires different evidence sources and different investigation patterns. The architecture is designed to support this—LangGraph's stateful workflows can be extended with new nodes and routing logic for each incident category.
Richer Experiences
I want Cortex to track more than just "what happened" and "what fixed it."
Future experiences will include:
What worked.
What didn't work.
What was tried and abandoned.
What evidence was considered but ultimately ruled out.
What the confidence level was at each decision point.
What the human reviewer noted during approval.
This means Cortex won't just remember successful fixes—it will remember false starts, dead ends, and rejected hypotheses. That knowledge is just as valuable as knowing what worked, because it helps future investigations avoid wasting time on paths that have already been proven unproductive.
Experiences Reflection
Currently, Cortex adds every experience to ChromaDB unconditionally. That's fine for a prototype, but in production, memory needs maintenance.
I want Cortex to periodically reflect on its own experiences:
Which experiences were most useful?
Which were never retrieved or applied?
Which had low confidence scores?
Which contradicted more recent, better‑verified experiences?
Based on this reflection, Cortex should curate its memory—promoting high‑value experiences and discarding ones that no longer serve the system. This is memory hygiene, and it's what keeps the retrieval layer fast and relevant as the system scales.
Generalized Experience Consolidation
Today, Cortex learns per asset. It can recognize that a schema drift incident on raw_sales was fixed by aliasing a renamed column. But if the same pattern appears on a completely different asset—say, raw_products—Cortex currently treats it as a new, unrelated incident.
I want Cortex to look across its experiences and recognize that:
"Customer ID renamed to customer UUID on raw_sales" and "Product ID renamed to product_identifier on raw_products"
are actually the same structural pattern.
When Cortex sees enough of these, it should consolidate them into a generalized lesson:
"When a primary key column is renamed in a raw layer, the fix pattern is: alias the new name to the old name in the staging model, then update downstream transformations incrementally."
This lesson then becomes portable across any asset, any pipeline, any team. That's the difference between a memory system and a true learning system.
Automated Verification
The next step is connecting Cortex to real pipeline execution and data-quality systems so proposed fixes can be tested automatically.
Instead of:
Propose Fix → Human Says It Worked
I want:
Propose Fix → Apply in Safe Environment → Run Pipeline → Run Data Quality Checks → Verify Result → Promote Experience
This shifts Cortex from a "proposal" system to a "self-validating" system. The human remains in the loop for consequential decisions, but the verification of whether a fix actually solved the problem becomes automated and evidence‑driven.
Visual Investigation Interface
I envision a UI where an engineer can watch Cortex investigate an incident in real time:
Lineage traversal.
Schema comparison.
Memory retrieval.
Evidence weighting.
Diagnosis reasoning.
Fix proposal.
The goal is for Cortex to become something an engineer can watch, question, verify, and trust—rather than a black‑box chatbot.
The UI would also show:
Why Cortex reuses or rejects a previous experience.
What evidence contributed most to the diagnosis.
What alternatives were considered.
What the confidence level is at each step.
My long-term vision is simple:
Cortex should make every incident investigation make the next one easier.
Log in or sign up for Devpost to join the conversation.