Regression Hunter AI

Inspiration

Every engineer has seen this: you change a field type, remove a column, or refactor a function. Unit tests pass ✅. PR merges. A day later, a nightly ETL job silently breaks, a revenue dashboard shows wrong numbers, or an ML model starts eating garbage features — because something downstream depended on that code, and nobody knew.

Unit tests validate that your code works. They can't tell you what breaks three hops away in a data pipeline, dashboard, or model you've never seen. We wanted to close that gap — by grounding an AI agent not in guesses, but in the actual metadata graph of the company.

What it does

Regression Hunter AI is a context-aware agent that predicts, explains, and prevents regressions before a pull request merges. It listens for GitHub PR events, parses the code diff at the AST level to find changed functions and entry points, then queries DataHub's lineage graph to find every downstream table, dashboard, and ML model that depends on that code.

If a change touches a Tier-0 or payment-critical asset, the risk score is automatically escalated. An AI reasoning agent generates a grounded hypothesis of what could break, a test agent writes and runs a verification test in an isolated sandbox, and the results are published both as a GitHub PR comment and written back into DataHub as a RegressionAssessment entity with tags and proposed assertions.

How we built it

The system is a monorepo of small, focused services: pr-listener → repo-worker → diff-analyzer → code-intelligence → context-orchestrator → risk-engine → reasoning-agent → test-agent → publisher.

The core of DataHub integration lives in a 4-level code-to-asset mapping strategy: explicit @datahub_urn annotations, OpenLineage/dbt manifest metadata, AST-extracted SQL/Kafka references, and DataHub semantic search as a fallback. The context-orchestrator queries DataHub's MCP server for get_lineage, get_entities, and search, then builds an EvidenceBundle that grounds every downstream LLM call in real graph data — not hallucinated guesses.

For write-back, the publisher creates custom DataHub entities, saves markdown reports, tags at-risk assets, and proposes schema assertions — closing the loop from code change to graph update.

Challenges we ran into

Grounding an LLM reliably in a metadata graph without letting it hallucinate relationships was the hardest part — we solved this by forcing every reasoning step to cite specific EvidenceBundle fields rather than free-form guessing.

Untrusted input was another core challenge: PR descriptions and diffs come from anyone, so before any data reaches the LLM, it passes through a prompt injection classifier and a secret/PII redaction engine. Generated tests also needed to run safely — we built an AST-guarded sandbox that blocks dangerous modules (os, subprocess, eval, exec) and runs with no network access.

Finally, making the system usable without a live DataHub instance (for local dev and demos) required building stub adapters (StubDataHubGraphPort, StubDataHubContextPort) that mirror the real interface exactly.

Accomplishments that we're proud of

57/57 unit and integration tests passing across 11 modules
Full bi-directional DataHub integration: reads lineage/ownership/criticality, and writes back custom entities, tags, and assertions
A defense-in-depth security layer: prompt injection detection, PII/secret redaction, AST-sandboxed test execution, and RBAC-gated API access
An offline standalone mode that runs the entire pipeline end-to-end with zero external dependencies

What we learned

Grounding matters more than model choice. A smaller model with a well-built EvidenceBundle from a real lineage graph outperforms a bigger model working from raw code alone — because it's reasoning over facts, not pattern-matching on code style.

We also learned that write-back is what turns a "read-only assistant" into a real part of the data governance workflow — tagging at-risk assets in DataHub means the next engineer who touches that table sees the warning too, not just the original PR author.

What's next for Regression Hunter AI

Real-time Slack/Teams alerts to downstream asset owners when their pipeline is touched
Auto-remediation: generating backward-compatible schema patch PRs for broken dbt models
Native AST parsers for Go, C#, Java, and TypeScript beyond Python
Training a calibrated risk model on historical incident data instead of static policy floors