Inspiration
We've all seen it happen: a senior data engineer leaves, and a month later someone opens a table with 8 downstream dependents, zero documentation, and a single owner who's no longer around to ask. DataHub is excellent at telling you what exists in a data stack, but it stays silent on why, and that "why" almost always lives in someone's head, or scattered across old commits and PR threads nobody rereads. We wanted to see if that knowledge could actually be excavated automatically, before it disappears, rather than mourned after the fact.
What it does
Lore reads DataHub's metadata graph to score every asset on how much institutional knowledge would be lost if its owner left today, weighing downstream impact, missing documentation, and ownership concentration. From there, one click ("Excavate") sends a Git Miner into a connected GitHub repo's commit history, PRs, and code comments looking for real evidence, which an LLM synthesizes into a plain-language explanation with every claim linked back to its source. High-confidence results write themselves back into DataHub automatically; lower-confidence ones go to a human review queue instead of guessing.
How we built it
FastAPI + React on top of the acryl-datahub Python SDK for the core
read/write pipeline, PyGithub for evidence mining, Groq's Llama 3.3 for
synthesis, and ReactFlow for the dependency graph visualization. We also
integrated DataHub's official MCP Server for natural-language asset search, genuinely, not just as a checkbox: after discovering that
mcp-server-datahub's dependencies hard-conflict with our FastAPI version
at the package level (no version of starlette satisfies both), we built
it as a fully isolated subprocess in its own virtual environment, so it
runs as a real, separate MCP client without touching the rest of the app.
Everything was built test-first once the core pipeline stabilized: 29 automated tests mock every external dependency (DataHub, GitHub, Groq), so the suite runs in about two seconds with zero live services required.
Challenges we ran into
Getting DataHub running locally on Windows was its own saga: a stale WSL
distro stuck mid-uninstall, an outdated datahub CLI whose quickstart
plan pointed at a docker-compose file that no longer existed upstream, a
disk-space-starved image pull that silently corrupted two container images
(MySQL and Kafka both failed with unrelated-looking errors that turned out
to share one root cause), and finally a genuine Windows-only bug in
DataHub's own CLI, urlparse() misreading a drive letter like C: as a
URL scheme, breaking any local file ingestion on Windows. That last one
turned out to be a year-old, still-open, maintainer-labeled bug affecting
multiple other users, so we patched it and opened a real PR upstream
(#19080) rather
than just working around it locally.
Beyond infrastructure, the more interesting challenges were correctness bugs we only found by testing against live data instead of trusting our own assumptions: a write-back path that could silently overwrite a different asset's real pre-existing documentation (two platforms can expose a same-named table, and literal-string evidence matching can't always tell them apart), an LLM that occasionally wrapped its JSON response in a markdown code fence despite being told not to, and a lineage endpoint that crashed outright the first time it encountered a non-dataset entity (a pipeline task) in DataHub's own graph.
What we learned
The most valuable moments in this build weren't the features that worked on the first try, they were the ones that didn't, because they usually revealed something more honest about the problem than a clean success would have. It's easy to build something that looks like it recovers tribal knowledge; it's harder to notice that literal string matching can conflate two unrelated tables with the same name, or that a "confidence score" and a "risk score" measuring completely different things will look like a contradiction to anyone who hasn't seen the code. Building the review queue instead of just trusting every LLM output, and writing regression tests specifically for the bugs we found rather than just the features we intended, ended up mattering more than any single feature did.
Log in or sign up for Devpost to join the conversation.