Hold the Line

One GitLab Duo flow catches a voice agent booking after "wait, no", opens a native Incident, writes the failing test and fix, and proves pass, fail, pass across three Cloud Run revisions.

Comment

Inspiration

A shorter greeting is a normal product change. On a voice agent that can book, cancel, or message, that change can also close the microphone before the caller finishes "wait, no." Unit tests stay green, and a judge that scores polite wording will still accept the booking.

Hold the Line grades the tool calls, not the wording. One GitLab Duo flow ships the greeting change. A separate verifier, with no model in the verdict, replays one frozen set of 30 calls on three image digests.

What it does

Creating work item 10 was the only human action. It asks to shorten the Saturday greeting from 2,200 ms to 900 ms and says the consent rules stay as they are. That started session 9567205.

  1. Plan. Before the first code commit, the flow posted the frozen contract and seal 5ca32408dc0ca5fada8b9f77d91a287c45689cffe9ada345155225d5b1bdd625.
  2. Create. The flow opened merge request !5 and committed the 900 ms greeting (c3bbdc00).
  3. Verify. The verifier built 30 calls from that image digest. On call 9 the simulated caller said "wait, no" at 990 ms. The agent still called book_appointment at 2,500 ms. Pipeline 2952600808 failed.
  4. Monitor. The same session opened Incident 11 with that trace.
  5. Repair. The flow committed a regression test (0efe1046). Pipeline 2952624503 failed on that test. A later commit (3febe5fd) kept listening open through the consent window.
  6. Release. Pipeline 2952628581 replayed the same calls on three digests: pass, fail, pass, plus a distinct holdout pass. The verifier promoted revision hold-the-line-00025-cr8 and the flow closed the Incident. Production traffic is 100% on that revision.

How we built it

The flow is .gitlab/duo/flows/hold_the_line.yaml, started by the work item. The voice agent, consent ledger, and timing generator are Python in src/hold_the_line/. They use timing seed 20261014. SHA-256 mixes that seed with the image digest. The booking ledger is simulated. No model decides pass or fail.

The source pipeline tests, scans, and publishes an image by digest. It cannot promote. Promotion is a second public project, hold-the-line-verifier, pinned at gate-v1. It deploys the digest to Cloud Run with GitLab OIDC, calls the tagged URL, and grades the tool calls it sees. The image's own pass flag is ignored. The certificate is release verified-2952633485.

Challenges we ran into

The flow that edits the agent must not be able to edit the rule it is graded on. The contract is sealed, and CI rejects a mismatched hash.

A bot cannot start a second flow. Detection, the Incident, the regression test, the timing fix, and the resolution stay in the one session the work item started.

The image under test cannot grade itself. The verifier replays the Cloud Run tag and checks that the URL is serving the digest it just deployed. Certificate verified-2952633485 names baseline sha256:f69e2b19e229e34a3e1d14ea2ec6a2c8dae36f23b629721a6b73bd71ae910ed7, regressed sha256:965daae9bf7e72bc5abc2350c973af778ec23b59999bad71f6042ace862f28bd, and repaired sha256:9d01b87f7f1a6d353608012807a9912433ed26a999b9b0164b37f9aa3dd2cb00.

Google's token exchange rejects a subject longer than 127 characters, so the workload-identity subject is the CI job id.

Accomplishments that we're proud of

Session 9567205 finished the loop with no later human edit: one work item, one flow, Incident 11 opened and closed, and three commits on merge request !5.

All nine stages leave a GitLab artifact: the work item, the commits, the failed and passing pipelines, the image digest, the scan and SBOM, the Terraform plan, the contract-hash job, Incident 11, and the verification certificate.

Production serves the repaired revision at 100%. The baseline and this run's regressed revision remain on their own URLs.

What we learned

An agent is useful at reading a failed trace and writing the next two commits. It is the wrong place to decide pass or fail. The verdict belongs to a sealed check that replays the same calls on the bytes that were deployed.

Built With

Share this project:

Updates

Submission history