The Context Agent

We built DataHub ContextPilot because data teams often need answers to simple questions such as which dataset they should trust for a KPI, where a dataset comes from, or which data asset is most relevant for a specific task. The information needed to answer these questions is often spread across metadata, documentation, data assets, and lineage. Our goal was to build an AI assistant that uses this context to provide more useful and trustworthy answers instead of relying only on the language model.

DataHub ContextPilot is a metadata-focused RAG assistant that allows users to ask data questions in plain English. It can help discover relevant data assets, summarize metadata, provide lineage-aware recommendations, generate SQL starter queries, and create useful artifacts such as handoff notes. The responses are supported with citations and relevant context so users can better understand and verify the information provided by the agent.

We built the project using Next.js, React, TypeScript, and Node.js. The backend uses Next.js API routes, while retrieval is handled through a local RAG chunk index using BM25-style lexical ranking. An OpenAI-compatible Chat Completions endpoint is used for generating responses. The application combines retrieved context with the user's question to create a grounded prompt for the LLM. The project is deployed on Vercel, and API keys are stored securely as Vercel environment variables instead of being included in the repository.

One of the main challenges was deciding how metadata, lineage, and retrieved project context should work together inside the agent. We also spent time fixing UI scroll and nested overflow issues, changing the interface from a chart-heavy design to a more focused agent experience, managing GitHub and Vercel deployment configuration, and cleaning local development references from the public project documentation.

During the project, we learned that good context is very important for AI applications. RAG quality depends heavily on how information is divided into chunks and which context is selected for each question. We also learned that metadata and lineage can provide valuable information for understanding data and improving recommendations. Finally, we learned that citations, security, deployment configuration, and documentation are important parts of building a reliable AI application.

Built With

• api
• next.js
• react
• typescript