Keel

Keel:AI-powered trust layer for data pipelines. Scores assets on hygiene, detects incidents, investigates root causes, and routes smart alerts to the right teams -with human approval always required.

Comment

Inspiration

We started with a simple frustration: data quality breaks things silently. A dashboard that worked yesterday suddenly shows wrong numbers, but nobody knows why. The data team is scrambling, the business is confused, and by the time anyone figures out what happened, decisions have already been made on bad data.

We wanted to build something that catches these problems early and tells you exactly what's wrong and who should fix it.

What it does

Keel watches your data pipelines like a careful guardian. Every time data flows through, it checks:

  • Is this data fresh? (Was it updated recently?)
  • Is this data stable? (Does the structure change constantly?)
  • Does someone own this data? (Who's responsible if it breaks?)
  • Is it documented? (Will the next person understand what it means?)

When something goes wrong—a quality check fails, data gets stale, an important field disappears—Keel figures out why it happened and who needs to know. Then it sends a message to the right team in the right way: engineers get technical details, business teams get a simple heads-up.

How we built it

We built Keel as a layer on top of DataHub (a metadata platform that knows everything about your data). Here's the journey:

  1. Started simple: Just scoring assets on basic hygiene. Does this data have an owner? When was it last updated?

  2. Added intelligence: Built a trust engine that understands lineage—if upstream data breaks, we know exactly what downstream will fail.

  3. Made it smart: Added an AI agent (powered by Qwen) that can actually investigate what went wrong. It reads your lineage graph, checks prior incidents, looks at the actual data rules, and makes recommendations.

  4. Added notifications: Integrated Slack so alerts go directly to the right people. Engineers see technical details. Marketing/Sales teams see "we're on it" messages in plain English.

  5. Built the safety: Made sure humans always approve before anything is flagged or fixed. No autonomous "fixes" that could make things worse.

Challenges we ran into

DataHub complexity: DataHub has a lot of metadata. We had to figure out which parts actually matter for trust scoring, and when to read vs. when to write.

LLM reliability: AI is great at investigation but can hallucinate. We built fallbacks so if the LLM fails, notifications still go out with sensible defaults.

Notification routing: Getting the right message to the right person is harder than it sounds. A table might matter to Engineering (they own it) and Marketing (they use it for reports) in completely different ways. We had to design department-aware messaging.

Human approval without friction: We wanted to keep humans in the loop—no autonomous fixes—but didn't want to make the approval process so heavy that people ignore alerts. We had to find the right balance.

Accomplishments we're proud of

  • End-to-end incident management: From detection → investigation → notification → approval → action, all in one system
  • Audience-aware messaging: Slack messages that make sense to both technical and non-technical teams
  • Durable audit trails: Every flag, every decision, every approval is logged in DataHub so you can track what happened when
  • Zero autonomous decisions: The system recommends, humans decide. Always.
  • Real-time updates: The dashboard, lineage graph, and activity feed all update live when something changes
  • Works with existing DataHub: No new infrastructure. We integrate with what you already have.

What we learned

  1. Data quality is an ops problem, not just a tech problem. You need governance, audit trails, and clear ownership—not just algorithms.

  2. Context matters more than perfection. A slightly wrong alert routed to the right person is better than a perfect alert that nobody sees.

  3. People fear automation. When we first built "auto-resolve" features, teams didn't trust them. Making everything human-approved actually builds more confidence.

  4. Metadata is messy, but valuable. Half your battle is just having clean lineage and ownership. Once you have that, scoring is easy.

  5. Demo scenarios are underrated. Being able to show "inject a failure here, watch it cascade downstream, send alerts here" teaches more than a thousand slides.

What's next for Keel

  • More sophisticated root cause analysis: Not just "this check failed" but "because this upstream column changed type"
  • Auto-remediation suggestions: "Try rolling back the pipeline", "Maybe impute this column"
  • Cross-org federation: Share data quality insights between teams without sharing the actual data
  • Predictive alerting: "This usually fails on Mondays, be ready"
  • Custom trust models: Let orgs define what "trust" means for their specific business

The big picture

We believe data quality shouldn't be something you discover in a crisis. It should be something you monitor, understand, and fix proactively. Keel is the system that makes that possible.

Built With

Share this project:

Updates