Inspiration
Every team is racing to put AI agents on their data, and the agents keep getting the numbers wrong. Not because the model is weak, but because the metadata the agent trusts is ambiguous, stale, or missing. Ask a fresh text-to-SQL agent for "2025 revenue" and it happily sums a gross, pre-discount column across cancelled orders and hands you a confident, wrong answer. We wanted to fix the context layer that every agent depends on, and prove the fix in numbers rather than vibes.
What it does
Keystone is a context reliability engine for AI data agents, built on DataHub.
- Audit. It scores how ready each dataset is for AI agents (description, documented columns, owner, deprecation, glossary) and ranks the context defects that hurt agents most.
- Profile. It reads the real data behind every table (distinct values, ranges) so its documentation is grounded, not guessed.
- Publish. It writes governed context back into DataHub through the Agent Context Kit: table and column descriptions, ownership, deprecation for a legacy table, lineage for the mart, and a business glossary with precise metric definitions.
- Prove. A downstream text-to-SQL agent reads its context live from DataHub and is scored against ground truth computed from the warehouse, before and after.
The write-back is the whole point. The graph gets richer every run, and every agent that reads it gets more accurate.
Results
Same warehouse, same free 8B model, same 20 analytics questions. The only variable is the context in DataHub.
| Before Keystone | After Keystone | |
|---|---|---|
| Downstream agent accuracy | 45% | 95% |
| Catalog AI-readiness | 31.4 | 100 |
A small, free model jumping 50 points purely because the catalog got trustworthy is the clearest proof we could give that context, not model size, is the lever.
How we built it
Local open-source DataHub as the context graph, a DuckDB warehouse seeded with deliberate ambiguity (a gross-vs-net revenue trap, ISO country codes, a deprecated legacy table, a stale status flag) as the data, the DataHub Agent Context Kit for the write-backs, a free Llama 3.1 8B model via Groq for both the curator and the downstream agent, and a before-and-after eval harness. The UI is Streamlit.
What we learned
Column names are self-documenting only until an agent has to make a decision. The gap between "looks fine" and "an agent can compute this correctly" is exactly the governed context a catalog is supposed to hold. Publishing precise, machine-followable metric definitions into DataHub closed almost the entire accuracy gap.
Challenges
Getting a small model to follow business conventions took real precision in the glossary (spelling out that fulfilled means status in shipped or delivered, that revenue excludes the gross column, that a country is an ISO code). We also worked through DataHub API details: owners are a GraphQL union that needs inline fragments, glossary terms in the new SDK take an id and definition rather than a name, and tags must exist before they can be attached.
What's next
Ship the datahub-context-audit skill upstream, extend the audit to more entity types, and run continuous re-scoring so the graph stays trustworthy as it changes.
Log in or sign up for Devpost to join the conversation.