Regression Hunter AI — Autonomous Pre-Deploy Defense Fleet

Inspiration

The inspiration came from a painful, familiar pattern: an engineer renames a database field in a payment microservice. Unit tests pass, CI is green, the PR merges — and three days later the executive revenue dashboard shows $0 and a Kafka pipeline silently drops records. Nothing failed at deploy time; the blast radius only became visible downstream. We wanted an agent that catches this class of failure before merge — the moment the code still exists only in a pull request.

What it does

Regression Hunter AI is an autonomous multi-agent defense fleet that intercepts GitHub pull requests and defends downstream data assets:

• Blast radius quantification — AST analysis across 5 languages (Python, TypeScript, Java, Go, C#) extracts code symbols, which are bound to enterprise assets via the DataHub lineage graph (through MCP, Model Context Protocol), with column-level lineage.
• Grounded reasoning — a Reasoning Agent powered by Google Gemini 3.5 Flash produces root-cause hypotheses that must cite evidence; unverified claims are flagged, never hallucinated.
• Policy-driven risk scoring — a policy floor elevates any breaking change on a regulated payment path with no regression test to 85/100 CRITICAL.
• Autonomous test synthesis — regression tests are generated and validated in an isolated sandbox.
• Institutional memory — every reasoning trace is persisted to a Google Cloud Firestore Memory Bank, with cross-session recall of past regressions.
• Write-back — sticky PR reports, DataHub assertions, and a Next.js impact dashboard.

How we built it

A Python monorepo of decoupled services (pr-listener, repo-worker, diff-analyzer, code-intelligence, context-orchestrator, risk-engine, reasoning-agent, test-agent, publisher). The LLM Gateway abstracts providers behind one interface: Gemini 3.5 Flash via the google-genai SDK (Gemini API), with a deterministic mock provider for offline testing. The Memory Bank uses Google Cloud Firestore on the free Firebase Spark plan — no billing account required — with a local mirror fallback. A CI/CD pipeline (cloudbuild.yaml) deploys to Google Cloud Run with scale-to-zero. 61 unit, integration and security tests cover the pipeline.

Challenges we ran into

• Feature-schema mismatch: the context builder emitted feature names the risk scorer didn't read (has_tier0_or_regulated vs has_tier0, missing weighted_downstream), so real blast radius scored 0/100. We added backward-compatible derivation in RiskFeatures.from_bundle — the demo now honestly scores 85/100 CRITICAL.
• LLM output robustness: live Gemini sometimes returns numbers as strings or omits confidence entirely. We built defensive coercion instead of trusting the schema.
• Cross-session recall silently broke once Firestore was active — search only looked at the local cache. We made recall query Firestore with a local-mirror fallback.
• No Google Cloud billing account: we adapted to the Firebase Spark free tier and proved real cloud deployment via Firestore + Cloud Shell.

Accomplishments that we're proud of

• End-to-end autonomy: PR webhook → AST → lineage traversal → Gemini reasoning → sandbox test execution → published report, with zero human hand-holding.
• The demo output makes real vs mock visibly distinguishable: live provider, model, latency (~11s) and token counts, plus a unique grounded hypothesis per run.
• A policy engine that turns "breaking change + regulated data path + no test" into an 85/100 CRITICAL verdict — deterministic, auditable, explainable.

What we learned

Grounding beats prompting: forcing the agent to cite lineage-graph evidence eliminates hallucinated blast-radius claims. Untrusted PR content must be sanitized before it reaches the LLM. And the entire Google Cloud fleet can run on free-tier infrastructure.

What's next for Regression Hunter AI — Autonomous Pre-Deploy Defense Fleet

• Agent Registry & Agent Gateway on the Gemini Enterprise Agent Platform
• Model Armor integration for inline prompt-injection guardrails
• Multi-repo, multi-org registries with cross-team agent discovery
• OpenTelemetry-native reasoning traces in the dashboard

Built With

• cloud-run
• datahub
• docker
• fastapi
• firestore
• gemini-3.5-flash
• github-actions
• google-cloud
• google-gemini
• google-genai-sdk
• model-context-protocol
• next.js
• pytest
• python
• react