Inspiration
DataHub is evolving from a catalog that humans browse into a context layer that AI agents actively depend on.
That creates a new kind of production risk.
Traditional data testing can tell us whether a pipeline ran, whether a schema compiled, or whether a column still exists. But an AI agent depends on much more than that: descriptions, glossary definitions, ownership, lineage, quality signals, query patterns, documentation, classifications, and relationships.
A change can therefore be technically valid while still breaking an agent.
A renamed field can make generated SQL fail. A vague description can cause the wrong dataset to rank first. A changed business definition can silently alter the meaning of a metric. Missing ownership can break an incident workflow. Metadata can even contain instructions that manipulate a tool-using agent.
We started ContextGuard CI around one question:
If organizational context is becoming production infrastructure for AI agents, why aren't we regression-testing it like production code?
That became the core idea behind ContextGuard CI: make organizational context testable, versionable, and safe enough for agents to depend on.
What it does
ContextGuard CI is the pre-merge safety gate for DataHub-powered agents.
It compares a trusted baseline with a proposed candidate change and determines whether that change breaks the workflows that depend on DataHub context.
Instead of evaluating only the final LLM answer, ContextGuard separates failures into independent layers:
- Metadata and context: Did schemas, definitions, owners, glossary terms, classifications, lineage, freshness, or quality guarantees change?
- Retrieval: Can the agent still discover the correct datasets, fields, documents, and relationships?
- Tool use and trajectory: Did the agent use the right DataHub tools, in the right order, within its allowed budget?
- Execution: Does the SQL or other generated artifact actually parse and run?
- Grounding: Are the datasets, fields, relationships, and claims in the answer backed by evidence the agent really retrieved?
- Semantic quality: Is the answer still correct and complete?
- Security: Did untrusted metadata, prompt injection, tool poisoning, or an unsafe action influence the agent?
ContextGuard then produces a CI-style PASS, WARN, or BLOCK decision with reason codes and evidence.
When something regresses, it does more than say "the agent failed." It shows what changed, which scenario was affected, where the failure occurred, and the smallest safe repair.
A repair can be previewed and simulated before anything is changed. Any real DataHub mutation remains outside the evaluation agent's authority and is approval-gated.
Finally, ContextGuard writes a concise audit result back to DataHub and verifies it through read-after-write, turning DataHub into both the source of organizational context and the system of record for what was learned during the review.
The result is a complete:
read → analyze → execute → repair → write → verify
workflow.
How we built it
ContextGuard CI is built in Python 3.11 as a local-first modular application with explicit ports and adapters.
DataHub OSS/Core is the primary metadata and context graph. We integrate with the DataHub Agent Context Kit, with MCP-compatible transport support, so the agent can search DataHub, inspect entities and schemas, traverse lineage, use query context, and retrieve organizational knowledge.
The evaluation agent is intentionally bounded. It receives only the read tools required by each scenario, with hard limits on tool calls, time, retries, and model usage. Write authority is handled separately so an evaluation agent cannot approve its own actions.
Tests are defined as versioned YAML scenario suites. ContextGuard snapshots and normalizes the relevant DataHub context, computes baseline-versus-candidate differences, uses lineage and references to identify affected scenarios, and evaluates only the workflows that matter.
We deliberately made the gate deterministic-first. Schema checks, identity checks, security policies, evidence grounding, and executable results carry more authority than subjective model scoring. LLM-based evaluation can add useful semantic signals, but it is never allowed to become an unexplained single point of truth for a blocking decision.
Generated SQL is validated and executed against a controlled DuckDB sandbox. Run metadata is persisted in SQLite, while snapshots, traces, reports, manifests, and other evidence are stored as immutable, content-addressed artifacts.
The review application uses FastAPI, Jinja2, and HTMX. The UI is evidence-first: reviewers can move from the overall comparison into the context diff, affected scenarios, retrieval evidence, tool traces, generated SQL, execution results, security findings, repair simulation, and write-back verification without relying on hidden chain-of-thought.
The same core services are designed to support the browser, API, and CLI so that the verdict does not change depending on how the run was started.
Challenges we ran into
The biggest challenge was realizing that "Did the agent answer correctly?" is the wrong level of abstraction for regression testing.
A correct-looking answer can hide a broken retrieval path. An agent can select the wrong dataset and still get lucky. Valid SQL can use the wrong business definition. A semantic judge can fluctuate between runs. A tool can report success even when its downstream mutation never becomes visible.
We had to make all of those states independently observable.
Another challenge was determinism. LLMs are probabilistic, but CI gates need to be reproducible. That pushed us toward deterministic checks wherever possible and toward explicit variance and flakiness handling wherever probabilistic evaluation is unavoidable.
DataHub integration introduced another important lesson: capabilities cannot simply be assumed from version numbers. Tool availability, authorization, search/index projections, server capabilities, and transport behavior can differ between environments. ContextGuard therefore treats capability discovery and runtime identity as evidence rather than assumptions.
Security was also more subtle than protecting an API key. The context itself is untrusted input. Metadata descriptions, documents, and even tool descriptions can become an attack surface for a connected agent. That led us to separate trusted instructions from retrieved content and to treat tool use, grounding, authorization, and mutation boundaries as first-class testable behavior.
Finally, making results reproducible forced us to think about evidence as a product feature rather than debug output. If a regression blocks a change, a reviewer should be able to understand exactly why without trusting an opaque model score.
Accomplishments that we're proud of
We're especially proud that ContextGuard CI is not another chat interface over a data catalog.
It treats DataHub context as a software dependency and brings software-engineering concepts—baselines, regression suites, deterministic gates, executable tests, evidence, impact analysis, and safe remediation—to agent context.
Some of the parts we're most excited about are:
- Baseline-versus-candidate context contracts rather than one-off agent benchmarks.
- Layered regression localization, separating metadata, retrieval, trajectory, execution, grounding, semantic, and security failures.
- Deterministic-first CI decisions instead of relying solely on an LLM judge.
- Real executable validation of generated SQL instead of assuming syntactically plausible code is correct.
- Evidence grounding that catches invented datasets, fields, relationships, or unsupported claims.
- Lineage-aware impact selection so changes are connected to the agent workflows they can actually affect.
- Repair preview and simulation rather than jumping directly from diagnosis to mutation.
- Separation of read-only agent authority from approved write authority.
- Verified DataHub write-back, including read-after-write verification instead of trusting a successful mutation response.
- Reproducible evidence bundles that make each gate explainable and auditable.
Most importantly, the project demonstrates a different way to think about metadata in the age of agents: not as passive documentation, but as runtime context that deserves regression protection.
What we learned
The biggest lesson was that context is code-like infrastructure once agents depend on it.
A metadata description, glossary term, lineage edge, ownership assignment, or retrieval ranking may look informational to a human, but for an agent it can directly determine behavior. That means those changes need contracts, tests, provenance, and review.
We also learned that agent evaluation becomes much more useful when it is decomposed. Retrieval quality, tool behavior, execution, grounding, semantic quality, and security answer different questions and should not be collapsed into one score.
Another key lesson was that evidence is more valuable than confidence. A reviewer should be able to see the exact asset, field, lineage path, tool call, execution result, and reason code behind a decision.
And we learned that "success" from an external system is not always a verified outcome. For governed actions, write → read back → verify is a much stronger contract than trusting an acknowledgement.
What's next for ContextGuard CI
The next step is to take the same context-regression model beyond the hackathon's focused single-tenant profile.
We want to deepen CI integration so ContextGuard can automatically evaluate relevant agent scenarios as part of pull requests and metadata-release workflows.
From there, the architecture is designed to expand toward team and enterprise environments with shared persistence, distributed execution, stronger identity federation, external KMS/HSM-backed signing, immutable evidence storage, and larger-scale scheduling and coordination.
We also want to expand the scenario ecosystem. The same framework can protect analytics agents, code-generation agents, incident-response workflows, data-product migrations, ML lineage and training/serving context, and other autonomous systems that rely on organizational knowledge.
Longer term, our goal is simple:
Whenever an AI agent depends on organizational context, changes to that context should be testable before they become production failures.
Log in or sign up for Devpost to join the conversation.