Inspiration
This is my first hackathon and honestly my first time touching DataHub at all. I found the challenge through Devpost's newsletter and had no idea what a data catalog even was going in. I started reading the docs and got interested in one specific problem: data pipelines break silently. A column stops updating, a stage falls behind, and nobody finds out until someone downstream is looking at a wrong number on a dashboard, I got that after seeing nyc-taxi problem. That felt like I could solve it, which is exactly why I wanted to build for it.
What it does
Ripple AI watches a data pipeline (I used DataHub's own nyc_taxi_pipeline sample dataset, raw_trips flowing into staging_trips flowing into mart_daily_summary, plus a mock CFO dashboard I attached downstream) and when a table falls stale, it doesn't just alert someone. It traces the lineage graph to find where the problem actually started, figures out everything downstream that's affected, generates a real backfill script using an LLM constrained to the dataset's actual schema, runs it, checks that the fix actually worked, opens a real pull request with the fix, and writes an incident plus a warning tag back into DataHub itself so the catalog reflects what happened. Once it's clean, it removes the warning tags and resolves the incident.
How we built it
I started by just poking around DataHub's UI to understand what lineage and glossary terms even looked like. Then I found the nyc_taxi_pipeline dataset had a real planted bug (staging silently falls behind raw) and decided to build against that instead of making up my own fake scenario. I used DataHub's Python SDK and its LangChain tool wrapper to search and trace lineage, built the dependency graph with networkx, and used Groq to generate the actual fix once I had all the real facts (schema, gap, transformation rules) pulled together. Later I pushed more of the logic into DataHub itself, adding custom properties like timestamp_column, sla_days, and repair_strategy directly onto the datasets, so the agent reads its own rules from the catalog instead of me hardcoding them in Python.
Challenges we ran into
So many. The first big one was realizing DataHub had ingested both a clean version of the dataset and the broken one under separate instances, and my code kept accidentally grabbing the clean one, which meant it looked like nothing was wrong when something clearly was. I had to learn to always double check which instance I was actually querying instead of trusting the first search result. I also had an off by one bug in my backfill date range that would have reinserted a day that wasn't actually missing. My detection logic for one of the two planted bugs kept giving false positives because I was comparing a date to its neighboring rows in a sparse dataset instead of comparing it against the actual source data for that exact day. And at one point my LLM generated a script using the wrong table's transformation logic entirely, aggregation rules meant for the summary table applied to the row level table, which would have crashed immediately since the columns didn't even exist. Every one of these I only caught because I stopped and actually ran a query to check instead of assuming the code was right. Btw it was exhausting.
Accomplishments that we're proud of
Getting the whole loop to actually close end to end. Not just detecting a problem, but tracing it to its root cause through a real lineage graph, generating a fix that's scoped to the exact broken table's real schema, applying it, verifying the fix actually worked by rerunning the same check, and opening a real PR automatically. I'm also proud that the deterministic parts (the SQL checks, the lineage traversal, the date math) stayed deterministic the whole way through, and the LLM only ever had one job, writing the fix script from facts I'd already confirmed myself.
What we learned
I learned how a metadata catalog actually works under the hood, lineage, glossary terms, assertions, custom properties, and how to query all of it programmatically instead of just clicking around a UI. I also relearned something that has nothing to do with DataHub specifically: never trust that something works just because it ran without an error. Almost every real bug in this project was invisible until I manually queried the data and checked it against what I expected. Datahub is really something else I didn't even know such tool existed. I'll be letting others know what kind of jobs it can do.
What's next for Ripple-AI
Right now it's scoped to one specific rule, freshness staleness, on one dataset shape. I'd want to generalize it to more failure types like row level data quality issues, and let it discover its own SLA thresholds and transformation logic straight from DataHub's glossary and custom properties instead of me writing any of that by hand. I'd also want the fix generation step to handle multi table dependencies where a single upstream break requires coordinated fixes across more than one downstream table at once.
Log in or sign up for Devpost to join the conversation.