About the project Agente Vigía is an AI agent that monitors your DataHub context graph to detect metadata risks before they become silent data failures. It reads lineage, ownership, and governance signals, then surfaces actionable findings — orphaned assets, missing owners, undocumented PII candidates, and untagged sensitive fields — with human-in-the-loop approval before any write-back.
What inspired me Every data team has seen it: a pipeline breaks, a dashboard shows wrong numbers, or an audit finds PII in the wrong place — and the root cause was always visible in metadata, just not in time. DataHub's context graph (schemas, lineage, ownership, governance) is the perfect foundation for an agent that watches for these issues proactively. The hackathon's Agents That Do Real Work category was the exact opportunity to build that watcher.
What I learned DataHub's MCP Server is the fastest way to connect an agent to metadata — just point it at http://:8080/mcp (self-hosted) or the Cloud endpoint and enable mutations for write-back.
DataHub Skills provide ready-made capabilities like datahub-search, datahub-lineage, and datahub-setup, which can be loaded via npx skills add datahub-project/datahub-skills or installed as Claude Code plugins.
Agent Context Kit lets you annotate tools with the datasets they touch and register agents, skills, and tools into DataHub automatically, turning your agent into a first-class catalog asset.
Building a read → act → write loop with human approval gates is critical for trust: the agent proposes, a human confirms, then the agent writes back tags, owners, or documentation.
How I built it Spin up DataHub locally using datahub docker quickstart and load sample data (e.g., showcase-ecommerce or nyc-taxi).
Set up the MCP Server endpoint and enable METADATA_SERVICE_AUTH_ENABLED=true on both datahub-gms and datahub-frontend-react to unlock token generation and mutations.
Install DataHub Skills via npx skills add datahub-project/datahub-skills and run /datahub-skills:datahub-setup to configure authentication.
Build the agent using Python + LangChain (or Google ADK) with the datahub-agent-context package (pip install datahub-agent-context[langchain]). The agent uses skills like datahub-search, datahub-lineage, and datahub-setup to read context, detect issues, and propose fixes.
Implement approval gates: the agent generates a markdown report with findings, requires explicit human confirmation, then writes back to DataHub (tags, owners, descriptions) via MCP mutations.
Test end-to-end with sample datasets, verify findings, and record a demo video showing the full loop.
Challenges I faced Enabling mutations: The "Generate new token" button was disabled until I set METADATA_SERVICE_AUTH_ENABLED=true on both GMS and frontend containers and restarted the quickstart.
Memory constraints on local setup: Running DataHub OSS on a 16GB laptop required adjusting WSL2 memory allocation (memory=11GB in .wslconfig) and understanding that OPENSEARCH_JAVA_OPTS (not ES_JAVA_OPTS) controls heap size.
Balancing automation with trust: Deciding which findings should require human approval vs. auto-fix. I opted for a conservative approach — all write-backs require explicit confirmation — to avoid unintended catalog changes.
What's next Extend to detect lineage gaps (assets with no upstream/downstream) and stale assets (no refresh in X days).
Add ML lineage support to catch training data drift or feature pipeline changes.
Integrate with DataHub's Analytics Agent to enable natural language queries about catalog health.
Built with DataHub OSS (MCP Server, Agent Context Kit, DataHub Skills)
Python + LangChain (or Google ADK)
Docker (DataHub quickstart)
GitHub (public repo with Apache 2.0 license)
YouTube (demo video)
Try it out GitHub repo: https://github.com/marcanogc/Agente-Vigia
Demo video: https://youtu.be/4evCtQQh-ls
Setup instructions: See README.md in the repo for DataHub quickstart, MCP setup, and agent configuration.
Log in or sign up for Devpost to join the conversation.