Winnow — Devpost Project Submission

Project Story

About the Project

Modern enterprise data warehouses are littered with orphaned "zombie" tables—datasets that are no longer queried or maintained, yet continue to quietly generate cloud storage and compute costs month after month. Data engineering teams rarely delete these assets due to fear-driven retention: without clear, real-time visibility into data lineage, deleting an old table risks breaking an active downstream Looker dashboard or production pipeline.

Winnow is an autonomous data governance agent that safely eliminates cloud data bloat. By programmatically inspecting metadata in the DataHub knowledge graph, Winnow identifies isolated Snowflake tables, verifies that they have zero downstream dependencies with 100% precision, and automatically generates an actionable GitHub Pull Request to archive the wasted assets.

What Inspired Us

In enterprise data stacks, manual metadata audits are slow, error-prone, and often neglected in favor of product features. We wanted to build an agentic governance workflow that bridges the gap between identifying waste and taking action, turning passive metadata graphs into automated, cost-saving engineering Pull Requests.

How We Built It

Winnow operates as a deterministic, multi-step governance agent:

Perception Layer: Connects to the local DataHub GMS server (http://localhost:8080) using the acryl-datahub Python SDK to retrieve all registered Snowflake dataset entities.
Lineage Interrogation: Queries DataHub's metadata graph in the DOWNSTREAM direction to calculate the exact degree of dependence ($D_t$) for each table $t$.
Safety Verification: Filters for orphaned assets where downstream relationships equal zero: $$D_t = 0 \quad \forall \quad t \in T_{\text{zombie}}$$
Impact Assessment: Calculates projected monthly cost savings ($S$) using a linear estimation model based on baseline Snowflake storage and compute allocations: $$S = n \times 150$$ (where $n$ represents the total count of verified zero-dependency tables, calculated at $\$150/\text{month}$ per entity).
Action Layer: Formats a structured Markdown report and uses the GitHub REST API to post an automated deprecation Issue/PR directly to the target repository.

Challenges We Faced

Graph Lineage Parsing: Traversing DataHub's metadata relationships dynamically required carefully handling nested graph objects and verifying schema attributes across different SDK versions to prevent runtime exceptions.
Deterministic Precision: Ensuring that zero false positives occur required strictly enforcing zero-downstream constraints rather than relying on heuristic or probabilistic scoring.
Plug-and-Play Resilience: Structuring the script so that missing API keys or inactive Docker containers fail gracefully without crashing local execution for reviewers.

What We Learned

Metadata-Driven Governance: Centralized metadata platforms like DataHub provide rich graph structures that can power fully autonomous infrastructure cleanup workflows.
Deterministic vs. Probabilistic Agents: For mission-critical infrastructure where an accidental deletion breaks production pipelines, deterministic graph interrogation provides the 100% accuracy required where standard LLMs might risk hallucinating dependencies.