diffgate

Change-aware security regression and release gating for AI agents

Comment

Inspiration

AI agents are moving from answering questions to taking real actions: updating records, calling privileged tools, sending data, and triggering business workflows.

That creates a gap in today's CI/CD security checks.

A pull request can pass code review, unit tests, SAST, and dependency scanning while still changing an agent's security behavior — by adding a new tool, changing authorization logic, or giving model reasoning a new path to sensitive business state.

We wanted to answer a different question:

Which security guarantees could this agent change affect, and can we prove they still hold before release?

Running a full agent-security assessment on every PR would be too slow and noisy. DiffGate makes that assurance change-aware: test deeply when needed, replay known security regressions when relevant, and stay out of the developer's way when a change cannot affect a protected guarantee.

What it does

DiffGate is a change-aware CI/CD security release gate for AI agents.

For each pull request, DiffGate decides how much security assurance the change requires:

                    PULL REQUEST
                         |
                         v
               What security changed?
                         |
          +--------------+--------------+
          |              |              |
     NEW / CHANGED     KNOWN PROTECTED   NO PROTECTED
   SECURITY SURFACE      CHANGE          IMPACT
          |              |              |
    Fresh assurance    Replay stored     Skip deep
      + replay         regression       assurance
          |              |              |
          +--------------+--------------+
                         |
                Verify real evidence
                         |
                  +------+------+
                  |             |
                ALLOW         BLOCK

The hackathon reference application is a Google ADK Procurement Agent with a sensitive update_bank_account capability.

Its security rule is deliberately simple:

Without valid approval, the supplier bank account must not change.

Before remediation, an unauthorized request changed the protected synthetic business state. DiffGate observed the mutation and blocked the release.

After remediation, the same unauthorized behavior was replayed. The bank account remained unchanged, the regression passed, and the release was allowed.

DiffGate does not trust the candidate agent's claim that an action was safe. It verifies the resulting protected state independently.

For new security surfaces, Gemini helps reason about the semantic change and the defensive validation required. Deterministic DiffGate code controls execution, verifies the evidence, and makes the final release decision.

Gemini reasons. Deterministic enforcement decides.

How we built it

DiffGate separates the application under test from the system deciding whether it may ship.

A base-controlled GitHub Actions workflow analyzes the PR using registered Google ADK capability schemas and reviewed security context.

If deep assurance is required, the exact candidate SHA is:

  1. built with Google Cloud Build,
  2. stored in Artifact Registry,
  3. deployed as a private Cloud Run candidate,
  4. exercised against controlled synthetic Firestore state.

The Procurement Agent uses Google ADK and Gemini 3.5 Flash through Vertex AI.

The candidate produces structured execution evidence, including protected state before and after the action. That evidence is evaluated by the separately deployed trusted DiffGate control plane.

DiffGate returns PASS, FAIL, or INCONCLUSIVE. Required FAIL or INCONCLUSIVE results block release.

Security validations are persisted as regressions, so once an important agent failure is discovered, future relevant releases can automatically prove that it has not returned.

We also use Secret Manager, Workload Identity Federation, GitHub Actions, Python, and Docker. Candidate PR code never replaces the trusted release-decision logic.

Challenges we ran into

The hardest challenge was deciding what to trust.

It would have been easy to let an LLM identify a risk, run a test, interpret the result, and then declare the release safe. Instead, we separated probabilistic reasoning from deterministic enforcement.

The second challenge was proving behavior rather than trusting responses. An agent saying "the operation was denied" is weaker evidence than verifying that the protected business record actually stayed unchanged.

The third challenge was making agent-security testing practical inside CI/CD. Running everything on every PR creates latency, cost, and developer fatigue. That led to DiffGate's three selective-assurance paths: fresh + replay, replay only, or no deep assurance.

Accomplishments that we're proud of

  • Built an end-to-end GitHub PR → security impact → selective assurance → release decision workflow.
  • Demonstrated a real before/after security regression: unauthorized mutation → FAIL/BLOCK, then same request after remediation → PASS/ALLOW.
  • Implemented all three assurance modes: fresh + replay, replay only, and no-impact.
  • Tested the exact candidate SHA, not an ambiguous shared development version.
  • Used Gemini 3.5 Flash + Google ADK for meaningful security reasoning while keeping the final verdict deterministic.
  • Verified actual protected business state rather than relying on model output as the security oracle.
  • Kept candidate code outside the trusted decision plane and used short-lived Google Cloud authentication instead of stored service-account keys.

The hackathon implementation intentionally goes deep on one executable assurance profile:

update_bank_account → supplier.bank_account → scoped approval → controlled Firestore state

DiffGate does not pretend unsupported security scenarios are safe — relevant unknowns fail closed.

What we learned

Behavioral evidence beats model confidence. For consequential agent actions, verify what actually happened to the protected system.

Security findings should become regressions. Once an important invariant is discovered, future releases should keep proving it.

Security impact should determine testing depth. A sensitive capability change should receive more assurance than a documentation or presentation change.

LLM reasoning and security enforcement need different trust levels. Gemini is useful for understanding semantic change. It should not be the final authority deciding whether a security invariant passed.

What's next for diffgate

The next step is generalized onboarding.

Instead of one deeply modeled Procurement Agent, DiffGate should support many agents and build an Agent Security Graph from capabilities, permissions, source/runtime observations, developer-reviewed business invariants, and historical security regressions.

That would let DiffGate automatically select assurance for broader classes of sensitive actions such as:

  • financial operations,
  • data export,
  • privileged APIs,
  • customer-data access,
  • identity and authorization changes,
  • external communication.

Over time, each agent could accumulate its own security memory: the guarantees that previous testing, reviews, and incidents taught us must never regress.

The goal is simple:

Make behavioral security regression a normal part of shipping AI agents.

Built With

  • adk
  • cloud-build
  • cloud-run
  • gcloud
  • gemini
  • vertex
Share this project:

Updates

Submission history