-
-
product hero screenshot
About the project
DataPilot is a metadata-aware AI agent that turns DataHub context into merge-ready analytics code.
Inspiration
The idea came from a simple frustration: most AI coding tools can write SQL or dbt models, but they usually do it without enough context. In real data teams, that leads to broken joins, wrong field choices, missed governance rules, and output that looks impressive but is not safe to merge.
The hackathon theme made that gap feel especially clear. DataHub already contains the context that agents are missing: schemas, lineage, ownership, quality signals, domains, glossary terms, and governance metadata. We wanted to build something that shows what happens when an agent can use that context before generating code.
Instead of asking an LLM to guess, we built a workflow where the agent first understands the catalog, then generates artifacts a data team could realistically review in a pull request.
What it does
DataPilot takes a plain-English business request like:
Build a weekly customer revenue mart by acquisition channel and country with lifetime value metrics.
It then:
- ranks relevant datasets using DataHub-style metadata
- expands through lineage to gather upstream context
- highlights governance risks like PII exposure or low-trust source tiers
- generates
model.sql,schema.yml,sources.yml, a PR summary, a lineage report, and a DataHub writeback plan
The goal is not just to generate code, but to generate code with enough context to be trustworthy.
How we built it
We built DataPilot as a lightweight Python application with two interfaces:
- a local web UI for quickly demoing the workflow
- a CLI for generating artifacts directly into an output folder
The project includes:
- a metadata ranking and selection engine
- a demo DataHub-style catalog with realistic datasets, lineage, and governance signals
- artifact generators for dbt SQL, YAML, PR notes, and lineage summaries
- an optional live GraphQL adapter for connecting to a real DataHub environment
- sample generated outputs so judges can review quality without extra setup
At a high level, the agent scores candidate datasets based on how well they match the request through names, glossary terms, fields, freshness, and quality tier. In simplified form, the ranking logic looks like:
$$ \text{score} = \text{name match} + \text{schema match} + \text{glossary match} + \text{freshness bonus} + \text{quality bonus} - \text{risk penalties} $$
That score is then combined with lineage expansion and field-coverage checks so the final selection is useful, not just statistically similar.
Challenges we faced
One of the hardest parts was avoiding the usual AI demo trap. It is easy to make a generator that looks good on a happy path, but much harder to make one that behaves like a real teammate.
A few challenges stood out:
- deciding how much context is enough before generation starts
- selecting the right datasets without over-selecting irrelevant ones
- surfacing governance issues early instead of burying them in generated output
- balancing a local, judge-friendly demo with a design that still maps cleanly to real DataHub usage
- making the generated artifacts feel reviewable by humans, not just machine-produced
A big design challenge was credibility. We wanted the output to look like something a data team might actually merge, so we added PR summaries, assumptions, lineage notes, and writeback suggestions instead of stopping at SQL generation.
What we learned
The biggest lesson was that context quality matters more than model cleverness. A simpler generator with good metadata can outperform a more sophisticated model working from incomplete context.
We also learned that agent workflows become much more valuable when they do three things well:
- read context before acting
- explain why they made a decision
- leave behind structured output that the next human or agent can build on
That is why DataPilot does not just generate code. It also documents source selection, assumptions, and governance implications so the work is easier to trust, review, and extend.
What’s next
The current prototype already works locally and supports optional GraphQL connectivity, but the next steps are clear:
- connect directly to a live DataHub environment end to end
- add writeback into DataHub so generated knowledge closes the loop
- support more artifact types like Airflow, Dagster, and ingestion configs
- improve team workflows around PR generation and review
Why this project matters
DataPilot is our answer to the question behind this hackathon: what can agents do when they actually have context?
Our answer is: they stop guessing and start producing work that looks much closer to something a real data team would ship.
Log in or sign up for Devpost to join the conversation.