Ghost

A two-phase offboarding agent for data engineers

Comment

Why it matters

IDC estimates organizations lose $31.5B annually from knowledge lost during employee transitions. Data engineering is disproportionately affected — tribal knowledge is embedded in table names, cron schedules, column choices, and join logic that no schema can capture. DataHub's institutionalMemory aspect literally ships with documentation noting that "tribal knowledge is very important" — Ghost is an agent that operationalizes that claim.

Ghost is a two-phase offboarding agent for data engineers.

What it does

Ghost scans a departing engineer's entire DataHub asset footprint, scores every asset 0–100 on knowledge-gap severity, then conducts an AI interview where Gemini 3.5 Flash uses the DataHub MCP Server as a live tool source — querying schemas, lineage, and usage stats in real time to ask targeted questions instead of generic ones. After the interview, Ghost synthesizes answers into documentation and writes 4 metadata aspects back to DataHub (description, institutional memory, ownership transfer, structured properties), then opens a GitHub PR with the handoff brief and reference SQL.

How we built it

Three services. The frontend is a Next.js app (TypeScript). The backend is a FastAPI service using the DataHub Python SDK for scanning assets and writing metadata change proposals. The AI layer connects Gemini 3.5 Flash to the DataHub MCP Server via the Vercel AI SDK's @ai-sdk/mcp package over HTTP transport, giving the LLM direct tool access to get_entities, list_schema_fields, get_lineage, and 5 other DataHub MCP tools during the interview. GitHub PR generation uses Octokit to create branches and commit the handoff brief. Everything runs locally via running the separate dev servers, or using Docker Compose — DataHub GMS, the MCP server sidecar, the FastAPI backend, and the Next.js frontend.

Challenges we ran into

  • We spent hours debugging "undefined" the MCP tool calls
  • GitHub's Contents API rejects file creation without a sha parameter if the file already exists, which broke PR creation on repeat runs until we added timestamps to file paths.

Accomplishments that we're proud of

Ghost reads AND writes to DataHub. It doesn't just pull metadata — it pushes synthesized documentation back. The MCP integration means interview questions are grounded in live schema and lineage data, not static prompts. The gap scoring algorithm catches real anti-patterns: tribal naming (_FINAL, _BACKUP, _copy), undocumented assets with downstream consumers, missing quality assertions. The whole system works end-to-end: scan → interview → writeback → GitHub PR, with zero manual steps after the engineer finishes answering.

What we learned

Type mismatches between dynamic and static tool schemas in the Vercel AI SDK cost real debugging time. DataHub's Python SDK is mature for reads and writes, but the MCP server is newer and requires careful transport configuration. The lesson here was separating the "tool-enabled" and "tool-less" LLM generation paths with distinct system prompts prevents cascading failures when the MCP server is unreachable.

Built With

Share this project:

Updates