Here it is in one block, ready to paste:
## Inspiration
At a past internship, I saw pipeline failures where the error pointed at the job that broke, not the upstream change that caused it. Tracing it through lineage by hand took hours. Built-in AI fixes patch the failing line, often just the symptom.
## What it does
Lakehouse Doctor is an AI reliability engineer that lives inside a Databricks workspace. When a job fails, it investigates the whole workspace: the error, the code, table lineage and Delta version history. It finds the true root cause (often a job upstream that succeeded) and the blast radius downstream. It proposes a fix that is checked against the real code, and with one human click it patches the right notebook and reruns the chain.
## How we built it
- **Agent:** a tool-calling agent on a Databricks Foundation Model endpoint. Its tools use the Jobs and Workspace APIs, Delta table history, lineage parsed from every job's code, system tables, and a Genie space.
- **App:** a seven-page Streamlit app on Databricks Apps. It covers live diagnosis, a lineage graph, a diff of the verified fix, Genie chat, costs, and a Chaos lab that breaks real jobs on demand.
## Challenges we ran into
- **The built-in lineage table was empty**, so I built my own lineage from job code and Delta history.
- **The LLM sometimes invented fixes**, so I added a verification step in code that forces it to retry.
- **The app couldn't read system tables** (it needs admin rights), so I exposed them through views I own.
- **Git folders, permissions and table ownership** for the app's service principal took a lot of debugging.
## Accomplishments that we're proud of
- It correctly traced a downstream failure to the exact upstream job, table version, run parameter and line of code.
- Every fix is verified against the real code before it can be applied.
- It builds on Databricks' own tools: Genie is one of its tools, and Genie Code receives the fix to implement.
## What we learned
- AI agents need checks written in code, not just good prompts.
- Delta table history is a powerful tool for forensics.
- I learned how Databricks Apps, service principals and Unity Catalog permissions actually fit together.
## What's next for Databricks - Lakehouse Doctor
- **Watch mode:** diagnose failures automatically and send alerts.
- **Accuracy scorecard:** run a library of chaos scenarios with known answers and measure how often the doctor is right.
- **Wider coverage:** Lakeflow pipeline and data-quality failures, not just jobs.
- **Incident memory:** recognize repeats of past incidents.
- **Cost doctor:** rewrite slow queries and measure the savings.
Built With
- llm
