Akashic weaver

Inspiration

Every data team has lived this moment: someone renames a column — first_name becomes given_name, a table gets restructured, a field gets deprecated — and weeks later a dashboard is quietly wrong, or an API starts throwing errors nobody can immediately trace back to the source. The schema change itself takes five minutes. Finding and fixing every place downstream that silently broke because of it can take days, and it almost always falls on a human to manually hunt through repos. DataHub already knows the answer to "what's connected to what" — that's the entire point of a lineage graph. The question we kept coming back to was: why is a human still the one walking that graph by hand? If an agent can read lineage, it can trace the blast radius of a schema change itself. If it can read code, it can fix it. So we built the agent that does both, end to end.

What it does

Akashic Weaver is an autonomous schema migration agent. When a column is renamed and marked Deprecated in DataHub: It reads the deprecation off the source dataset — the old column name, the new one — directly from DataHub's metadata. It traverses the dataset's downstream lineage graph to find every dependent entity, extracting the GitHub repo, file path, and branch stored in each one's custom properties. For each dependency, it fetches the live file from GitHub and hands it to an LLM with a tightly scoped prompt: find every reference to the old column — across raw SQL, ORM calls, and JSON access — and replace it, without touching anything else in the file. It opens a real pull request with the refactored code, a clear title, and a body that explains exactly what changed and why. It writes a last_migration_pr property back onto the source dataset in DataHub, so the next engineer — or the next agent — sees the migration was already handled instead of rediscovering the same problem.

How we built it

The stack is deliberately small: Node.js, @octokit/rest for GitHub automation, the OpenAI SDK for the refactor step, axios against DataHub's GraphQL API for reading, plus its REST ingestProposal endpoint for writing metadata back. Split into four modules by responsibility — datahub.js, github.js, llm.js, index.js — so each piece can be reasoned about independently. To make the DataHub side real rather than mocked, we wrote a seed script that writes actual globalTags, datasetProperties, and upstreamLineage aspects into a live local DataHub instance — loaded from one of the hackathon's provided datapacks — via the same ingest API the official DataHub SDK uses.

Challenges we ran into

Running DataHub's full local stack needs far more RAM than a modest laptop has to spare. Rather than compromise on running a real instance, we moved the whole environment into a GitHub Codespace — turning a hardware limitation into a non-issue, with the side benefit of making the build reproducible for anyone judging it. The other real challenge was scoping the LLM refactor tightly enough to trust it. Getting the prompt to correctly handle raw SQL, ORM query builders, and plain object access as three different shapes of the same change — while leaving everything else in the file untouched — took real iteration.

Accomplishments that we're proud of

Getting the full loop closed — detection, lineage traversal, code fix, PR, and write-back — running against a real DataHub instance and a real GitHub repo, not a simulated one.

What we learned

How much of "agent reliability" comes down to how narrowly you scope what the agent is allowed to touch. The more constrained the task, the more consistently correct the output got. We also came away with a much better feel for DataHub's metadata model — aspects, URNs, the ingest proposal API — than we had going in.

What's next for Akashic Weaver

Support more deprecation types beyond column renames — table splits, type changes, deprecated datasets with a suggested replacement. Move from DataHub's raw GraphQL API to its MCP Server for broader agent-runtime portability. Add a review-gate comment on the PR summarizing every reference changed, so reviewers don't have to reverse-engineer the diff. Batch mode: handle multiple deprecations across a whole lineage graph in one run.

Built With

• datahub
• geminisdk
• graphyql
• mysql
• nextjs
• octokit/rest