Regression Hunter AI

Context-aware AI agent that predicts downstream regressions before code merges using DataHub's metadata graph.

Comment

Inspiration

Every engineer has seen this: you change a field type, remove a column, or refactor a function. Unit tests pass ✅. PR merges. A day later, a nightly ETL job silently breaks, a revenue dashboard shows wrong numbers, or an ML model starts eating garbage features — because something downstream depended on that code, and nobody knew.

Unit tests validate that your code works. They can't tell you what breaks three hops away in a data pipeline, dashboard, or model you've never seen. Industry studies consistently show that defects escaping to production cost 10–100× more to fix than defects caught pre-merge — and the most dangerous defects are the ones no test suite can see, because they live in dependencies, not in the changed code.

We wanted to close that gap — by grounding an AI agent not in guesses, but in the actual metadata graph of the company.

Why this fits CUTC: Transform. Regression Hunter AI transforms development from reactive firefighting to proactive prevention: regressions are predicted, explained, and neutralized before code merges — and the knowledge is written back into the organization's data catalog so the protection compounds over time.

What it does

Regression Hunter AI is a context-aware agent that predicts, explains, and prevents regressions before a pull request merges.

End-to-end pipeline

┌──────────┐  ┌───────────┐  ┌────────────┐  ┌──────────────┐  ┌─────────────────┐
│ 1. GITHUB│─▶│ 2. LISTEN │─▶│ 3. SNAPSHOT│─▶│ 4. DIFF+AST  │─▶│ 5. CONTEXT      │
│ PR event │  │ webhook + │  │ repo-worker│  │ diff-analyzer│  │ context-orchest.│
│          │  │ idempot.  │  │ tarball    │  │ code-intel   │  │ EvidenceBundle  │
└──────────  └───────────┘  └────────────┘  └──────────────┘  └────────┬────────┘
                                                                        │
┌──────────┐  ┌───────────┐  ┌────────────┐  ┌──────────────┐  ┌────────▼────────┐
│ 9. PUBLISH│◀─│ 8. TEST   │◀─│ 7. REASON  │◀─│ 6. RISK      │◀─│ lineage +       │
│ GH comment│  │ sandboxed │  │ evidence-  │  │ calibrated   │  │ owners +        │
│ + DataHub │  │ verify    │  │ bound LLM  │  │ score 0–100  │  │ criticality     │
│ write-back│  │ tests     │  │ hypothesis │  │ policy floors│  │                 │
└──────────┘  └───────────┘  └────────────┘  └──────────────┘  └─────────────────┘

Concrete walkthrough

  1. PR opened: an engineer changes apply_discount() in pricing.py.
  2. AST intelligence: the changed fully-qualified symbol billing.pricing.apply_discount is extracted.
  3. Code-to-asset mapping (Level 1): a @datahub_urn annotation maps it to urn:li:dataset:snowflake.billing.discounts.
  4. Lineage expansion: DataHub MCP returns the downstream chain: dbt.marts.finance_revenue → looker.dashboard.exec_revenue (marked Tier-0).
  5. Risk engine: Tier-0 in lineage triggers a policy floor → ESCALATE, calibrated score 87/100.
  6. Reasoning agent: generates a grounded hypothesis, citing EvidenceBundle fields: "the change alters the discount cap; the exec revenue dashboard aggregates discounted amounts".
  7. Test agent: writes a property-based test asserting the cap invariant and runs it in an AST-guarded sandbox.
  8. Publisher: posts a structured risk report as a GitHub PR comment and writes a RegressionAssessment entity back into DataHub, tagging @RegressionHunter:AtRisk on impacted assets.

4-level code-to-asset mapping

Level Source Example Confidence
1 Explicit annotations @datahub_urn, datahub_assets.yaml Exact
2 Pipeline metadata OpenLineage events, dbt manifest High
3 AST extraction SQL table refs, Kafka topics Medium-high
4 Semantic search DataHub search fallback Best-effort

How we built it

The system is a monorepo of small, focused services and shared domain packages:

apps/
├── api-gateway/          # FastAPI REST/SSE with RBAC & auth
├── pr-listener/          # GitHub webhook receiver + idempotency validator
├── repo-worker/          # source snapshotting & tarball storage
├── diff-analyzer/        # file delta & line-range analysis
├── code-intelligence/    # AST parsing, symbol extraction, FQN resolution
├── context-orchestrator/ # DataHub impact neighborhood & EvidenceBundle builder
├── risk-engine/          # calibrated scoring & policy-floor enforcement
├── reasoning-agent/      # evidence-bound LLM reasoning (state machine)
├── test-agent/           # test-gap planning & isolated sandbox runner
└── publisher/            # GitHub comment/check + DataHub write-back
packages/
├── contracts/  ├── domain/  ├── datahub-adapter/  ├── github-adapter/
├── llm-gateway/ ├── code-graph/ ├── risk-model/  └── observability/

DataHub integration (bi-directional):

  • Read: MCP tools get_lineage, get_entities, search for column lineage, owners, quality, criticality.
  • Write-back: custom RegressionAssessment entities, markdown report documents, @RegressionHunter:AtRisk tags, proposed schema assertions.

Grounding over guessing: every LLM reasoning step must cite specific EvidenceBundle fields; free-form speculation is structurally impossible.

Offline standalone mode: StubDataHubGraphPort / StubDataHubContextPort mirror the real interfaces exactly, so the entire pipeline runs end-to-end with zero external dependencies for demos and CI.

Security architecture

[ Untrusted PR data ] ─▶ ( Prompt-Injection Classifier ) ─▶ ( Secret/PII Redaction )
                                                                   │
                                                                   ▼
[ LLM Reasoning ] ◀────────────────────── [ Safe EvidenceBundle ] ◀┘
        │
        ▼
[ Generated test patch ] ─▶ ( AST Security Parser ) ─▶ ( Isolated Sandbox ) ─▶ [ Results ]
                              blocks: os, subprocess, socket,
                                      eval, exec, __import__
  1. Prompt-injection defense — untrusted PR text is scanned and wrapped in <untrusted_content> markers.
  2. Secret/PII redaction — API keys, tokens, AWS creds, emails, SSNs masked before any LLM call.
  3. AST-guarded sandbox — generated tests are AST-validated, run in temp dirs with NO_NETWORK=1.
  4. RBAC-gated API — JWT/API-token auth; unauthorized triggers rejected with 403.

Challenges we ran into

  • Hallucination control: grounding an LLM in a metadata graph without invented relationships. Solution: mandatory citation of EvidenceBundle fields at every reasoning step.
  • Untrusted input: PR diffs come from anyone. Solution: injection classifier + redaction engine before the LLM boundary.
  • Safe code execution: generated tests are untrusted code. Solution: AST guard blocking dangerous modules + network-less sandbox.
  • No live DataHub in dev: Solution: stub ports mirroring the real interface 1:1, enabling full offline runs.
  • Mapping ambiguity: the same table referenced in raw SQL vs dbt. Solution: the 4-level strategy with confidence-weighted merging.

Accomplishments that we're proud of

Metric Result
Test suite 57/57 passing across 11 modules
Architecture 9 microservices + 8 domain packages
DataHub Full bi-directional integration (read + write-back)
Security 4 independent defense layers
Offline mode 100% of pipeline, zero external dependencies
  • Defense-in-depth security: prompt-injection detection, PII/secret redaction, AST-sandboxed execution, RBAC.
  • Deterministic risk engine: calibrated 0–100 scoring with policy floors for Tier-0 and payment paths.
  • Write-back loop that turns a read-only assistant into a governance participant.

What we learned

  • Grounding beats model size. A smaller model with a well-built EvidenceBundle from a real lineage graph outperforms a bigger model on raw code — it reasons over facts, not style.
  • Write-back is the moat. Tagging at-risk assets in DataHub means the next engineer sees the warning too — protection compounds.
  • Determinism + LLM = production trust. Calibrated scores and policy floors make the agent's behavior auditable, not vibes-based.
  • Security is architecture, not a feature. Treating every PR as hostile input from day one shaped the whole pipeline.

What's next for Regression Hunter AI

Phase Horizon Deliverables
1 Q4 2026 Slack/Teams alerts to downstream owners; Go AST parser
2 Q1 2027 Auto-remediation PRs (backward-compatible schema patches); Java/C# parsers
3 Q2 2027 Calibrated ML risk model trained on historical incidents

Long-term vision: every pull request in every data-driven company is automatically checked against the organization's live lineage graph — so "silent regression" becomes a historical term.

Built With

Share this project:

Updates