Inspiration
Modern data teams spend countless hours debugging broken pipelines, tracking schema changes, understanding data lineage, and searching through documentation. While AI agents have become incredibly capable, they often lack the organizational context required to solve real data engineering problems. Without knowledge of schemas, ownership, lineage, and metadata, AI assistants frequently hallucinate or provide incomplete solutions.
Serein was created to bridge that gap. Instead of being another chatbot, Serein is designed to function as an autonomous data engineering companion that understands an organization's data ecosystem through DataHub and helps engineers solve production issues with confidence.
What it does
Serein is an AI-powered data engineering companion built on DataHub's metadata platform.
When a user reports a problem—such as a broken dashboard or failed pipeline—Serein investigates the issue by querying DataHub's metadata graph and MCP Server. It traces dataset lineage, identifies upstream dependencies, detects schema changes, locates dataset owners, and explains the root cause in clear, human-readable language.
Beyond diagnosis, Serein generates production-ready fixes such as SQL queries, dbt models, Airflow configurations, or migration scripts. It can also document findings and write the investigation back into DataHub, ensuring future engineers and AI agents inherit the knowledge instead of solving the same problem repeatedly.
How we built it
Serein combines modern AI agent architecture with DataHub's context platform.
The application uses DataHub's MCP Server and metadata graph as its primary source of organizational context. Large Language Models interpret metadata, reason through dependencies, and generate actionable solutions. A web-based interface enables engineers to interact with Serein using natural language, while integrations with GitHub allow generated fixes to be reviewed and merged into existing development workflows.
By grounding every decision in DataHub's metadata rather than assumptions, Serein produces responses that are significantly more reliable than traditional AI assistants.
Challenges we ran into
The biggest challenge was designing an AI agent that doesn't simply answer questions but performs meaningful engineering tasks. Understanding complex metadata relationships, reasoning across lineage graphs, and converting metadata into production-ready code required careful orchestration between the language model and DataHub.
Another challenge was ensuring that every recommendation remained grounded in organizational metadata instead of relying on model assumptions.
Accomplishments that we're proud of
- Built an autonomous AI companion focused on real data engineering workflows.
- Leveraged DataHub's metadata graph and MCP Server for context-aware reasoning.
- Automated root-cause analysis for pipeline and schema failures.
- Generated production-ready engineering artifacts instead of generic suggestions.
- Designed a workflow that continuously captures and shares organizational knowledge.
What we learned
Building Serein reinforced an important lesson: powerful AI isn't enough. The quality of an AI agent depends on the quality of the context it receives. DataHub's metadata graph provides the missing context that enables AI agents to reason about real-world data systems instead of guessing.
What's next for Serein
Our vision is to evolve Serein into a complete autonomous data engineering teammate.
Future versions will support multi-agent collaboration, proactive monitoring of data quality, automated governance workflows, continuous documentation, intelligent pull request reviews, and deeper integrations with modern data platforms. Ultimately, we want Serein to become an always-on engineering companion that understands, protects, and continuously improves an organization's entire data ecosystem.
Log in or sign up for Devpost to join the conversation.