Polygraph

Each run's signed evidence, replayed: agent claims vs the verifier's verdicts, and exploits before and after the patch.
A real pipeline where an agent lied: the Builder claimed every test passed, and verify:tests checked the actual report and blocked the run.
The same exploits are replayed on the build that ships. A fix counts only if the exploit fails. One didn't, so it's blocked.
Red Team attacks a deliberately vulnerable twin live on Google Cloud Run and finds all 7 seeded weaknesses.

Inspiration

Code generation is basically solved. The real blocker for hands-off agents is trust. Nobody gives an AI agent production access because an agent can claim that all tests pass or a vulnerability is fixed, whether or not it's true, and humans rarely check every claim.

We wanted agents to carry their entire change to production, backed by a system that cannot be fooled by what they say. A polygraph for software delivery.

What it does

Polygraph is autonomous delivery that can't lie: AI agents do the work, and evidence decides what ships.

You open an issue and assign it to our GitLab Duo flow. A Planner agent writes acceptance criteria and a risk label, while a Builder agent writes the code and tests and opens a merge request. Instead of a summary, every agent ends its step with a structured claim, such as "127 tests pass, 0 fail, on commit abc123."

A deterministic verifier, not an LLM, re-derives every number from authoritative evidence that the agent cannot write to: JUnit reports, the container registry, the Cloud Run API, Cloud Monitoring, and live exploit replays. If a claim contradicts the evidence, Polygraph blocks the run, records a caught lie, and automatically opens a follow-up issue.

Fixes are proven, not asserted. Every commit builds a hardened image and a deliberately vulnerable pre-patch twin, both deployed to Cloud Run staging. The Red Team playbook attacks the twin using SQL injection, XSS, IDOR, missing authentication, stack-trace leaks, vulnerable dependencies, and prompt injection. The same exploits are replayed against the build intended for release, and must fail. Every regression test must also fail on the twin and pass on the patched version, rejecting tests that do not actually prove the fix.

The verifier tests itself. Every pipeline plants nine known lies before evaluating the change. If even one gets through, the run is blocked.

Autonomy is earned. A hash-chained trust ledger scores every agent across runs. An agent caught lying goes on probation, and its subsequent deployments require human approval even if every technical gate passes. After a clean streak, it earns unattended production access back. We demonstrated this with real CI runs: the Builder lied, its next deployments were held, and after five clean runs, it deployed autonomously again.

Signed, public proof. Each run's evidence pack is signed keylessly using Sigstore Cosign, verified in a separate job, posted on the issue and merge request, and replayed as an animation on a GitLab Pages dashboard.

Production on Google Cloud. Polygraph promotes the exact verified image digest to Cloud Run, then verifies the live deployment through the Cloud Run API and Cloud Monitoring, including five out of five successful health probes and error-rate checks. Failed deployments trigger automatic rollback.

Polygraph covers all nine DevSecOps stages:

Stage	What runs
Plan	Duo Planner writes criteria and risk; `verify:plan` checks the live issue.
Create	Duo Builder opens the MR and posts claims; `verify:create` checks the MR.
Verify	`verify:tests` re-derives claimed numbers; `verify:canary` tests against nine planted lies.
Package	Builds the hardened image and pre-patch twin; verifies registry labels.
Secure	SAST, dependency and secret scanning, plus Red Team vs. Blue Team exploit replay.
Release	`policy:autonomy` evaluates earned trust before promoting the exact verified digest.
Configure	Terraform provisions Cloud Run, Artifact Registry, and keyless Workload Identity Federation.
Monitor	`verify:deploy` checks the Cloud Run API and Cloud Monitoring, with automatic rollback and scheduled drift monitoring.
Govern	Signed evidence, verifier fingerprints in every evidence pack, the trust ledger, and automatically generated follow-up issues.

Who it's for: Platform and security teams that want to hand the post-code delivery lifecycle to AI agents but need verifiable evidence before trusting them with production access.

How we built it

GitLab Duo Agent Platform: We built two agent flows:

PCP Deliver: Planner → Builder, triggered when an issue is assigned.
PCP Blue Team: Triage → Blue Team | Pipeline Fixer, triggered by a failed pipeline and bounded to three rounds.

Both flows are generated from versioned prompt files and validated against GitLab's flow schema in CI. The verifier and exploit replay are also exposed as MCP tools, allowing agents to request checks without ever computing the verdict themselves.

GitLab CI/CD: A gated pipeline where work jobs never judge their own results. Each verify:* job reads the agent's claim alongside authoritative evidence and fails closed whenever evidence is missing, contradictory, or ambiguous. Demo switches—including planted lies, incomplete patches, and human approval—are implemented as pipeline inputs.

The verifier: We implemented pure-Python checks, a versioned catalogue, claim and evidence JSON schemas, planted-lie self-tests, and a hash-chained ledger stored in the project's package registry. The verifier has 136 tests with approximately 92% test coverage.

Google Cloud: Cloud Run hosts a staging pair for each merge request and the production service. Artifact Registry stores container images, and Cloud Monitoring provides deployment telemetry. Everything is provisioned using Terraform. Workload Identity Federation provides three separate identities without stored keys: a production deployer accessible only to protected-branch jobs, a staging deployer, and a read-only verifier.

Demo application: We built a small FastAPI team-notes service with seven seeded vulnerabilities, making the exploits real, reproducible, and suitable for demonstrating both successful fixes and incomplete patches.

Challenges we ran into

Our own system caught our bugs. Initial seeded pipeline runs failed for the wrong reasons: a demo input leaked into a test, and the incomplete-patch switch reached the test job, causing the tests gate to block a run that the exploit replay should have blocked. We fixed both issues and added regression tests.

GitLab rejected our flow prompts. Our initial claim marker used an HTML comment, which Duo does not permit in developer prompts. We moved claims into a fenced pcp-claim block and added a test validating every prompt against GitLab's rules.

Cloud Run reserves /healthz. Its frontend returned its own 404 for our health probes, causing the Red Team's safety check to refuse to attack a target it considered unavailable. We moved our health endpoint to /health.

Workspace limitations. As Developers, we could not pass CI variables to manual pipelines, so we implemented demo switches as GitLab pipeline inputs. Code-owner approval was unavailable in our workspace, and we explicitly documented this limitation in the README rather than claiming a capability we did not have.

First keyless production deployment. Our deployer identity initially could not read from the container registry. Granting the necessary read-only role resolved the issue without introducing stored credentials.

Accomplishments that we're proud of

A real, hands-off Duo run: Issue #6 was planned and implemented by agents, resulting in merge request !5. The pipeline then read and verified the Builder's own claims.
A caught lie in real CI: The Builder claimed zero failures, but the report showed one. Polygraph flagged the contradiction as a caught lie, blocked the run, and automatically opened a follow-up issue.
An incomplete patch caught against a live deployment: Exploit replay against a real Cloud Run build detected that the vulnerability remained exploitable: "V1 STILL OPEN".
Earned autonomy demonstrated on real runs: An agent on probation was held for human approval, completed five clean runs, regained its autonomy, and deployed independently.
A self-testing verifier: Every run checks the verifier using planted lies, while the evidence is signed and independently verified.
55+ visible pipelines: Our project includes promoted, caught-lie, incomplete-patch, held, and agent-triggered runs, demonstrating the system across multiple scenarios.

What we learned

Don't grade agents on their prose; grade them on claims you can check. Once outputs become structured claims, deterministic verification becomes straightforward, and dishonest or incorrect claims become measurable.

A gate is only as good as the evidence behind it. Passing unit tests does not guarantee that the shipped artifact is secure. That is why we attack the actual built image, compare it with a deliberately vulnerable twin, and verify the result against live deployments.

Trust should be earned and lost over time, like a colleague's. Production access should not be a static permission. Agents should earn autonomy through demonstrated reliability, lose it when they fail, and regain it through a clean track record.

What's next for Polygraph

Bring Polygraph to any repository. Everything except the exploit playbook is project-agnostic, so the system can support projects with tests and container images. Our next step is a pluggable playbook that generates probes from SAST findings and replays them against vulnerable and patched builds.

Expand the evidence catalogue. We plan to introduce more claim types, including performance budgets, database migrations, and compliance checks, each backed by its own deterministic evidence verification.

Publish to the GitLab AI Catalog. We want to make our agent flows reusable by other teams and strengthen the trust boundary with code-owner approval on protected branches.

Our goal is simple: agents should not need to be trusted because they sound convincing. They should earn production access because independently verifiable evidence proves they deserve it.