Offbrief

A release gate for AI coding agents: recorded traffic replayed against stable (twice) and the candidate, plus per-run honeytoken markers, catches what the agent's own tests never will.

Comment

Inspiration

A coding agent ships its change together with its own passing tests. That is the problem I kept running into: the tests, the diff and the fix all have the same author, so "the pipeline is green" says very little about what the service now does under real traffic.

I built the synthetic incident the whole project is organised around. An agent adds a notes field to a patient record on my own demo API. Its unit tests pass. The recorded traffic, replayed against the candidate, shows the field also leaving toward a destination the change never declared. Nothing in a diff review, a unit test written by the same agent, or a static scan is positioned to see that. The one thing that is, is production-shaped traffic flowing through the candidate while something independent watches the wire. That observation became the name and the tagline.

What it does

Offbrief is a release-quality gate for changes that coding agents make to patient-api, my own synthetic demo API (FastAPI + SQLite, Faker records, nobody real). Everything runs in my own GitLab CI jobs and my own Google Cloud project; the egress check is a data-leak-prevention regression test on my own code inside a sandbox I own, and nothing targets third parties.

The loop is hands-off in the hackathon's own words ("you set the intent and walk away ... You push code or create an issue, and the next time you check in, it's live"): the human opens one issue labelled offbrief::request, and the next human-visible event is a summary note on that issue ending with Human actions in this loop: 1. In between:

  1. An intent flow derives a change contract (routes, fields, allowed partner destinations, SLO) from the issue; the warden freezes its SHA-256 in a hash-chained ledger before any code exists.
  2. A builder flow opens one merge request inside the agent-writable area (app/patient_api/**, app/tests/**).
  3. The warden starts a conductor pipeline on main (never the MR's own CI). One sandbox job runs the candidate and two copies of stable on an internal network, replays the same recorded traffic through all three, runs the candidate's own tests, and records every byte the candidate sends out, DNS included. The replay is three-way on purpose: whatever differs between the two stable copies is noise, whatever differs only on the candidate is behaviour. Every PHI-shaped value in the per-run snapshot is a honeytoken marker derived from a key generated inside the job, so a leak is recognised in raw, base64, hex, URL and DNS-label encodings without the candidate being able to tell markers from data.
  4. The evidence lands on the MR as a note; a judge flow reads contract and evidence with GitLab tools only and can add blockers, never remove any.
  5. The gate is an AND of twelve deterministic checks (state_ok, head_ok, paths_ok, contract_ok, evidence_ok, controls_ok, egress_ok, replay_ok, judge_ok, scans_ok, pin_ok, budget_ok), fail closed. On FAIL a fixer flow pushes to the same branch and the evaluation starts again from scratch, up to the attempt limit.
  6. On PASS the warden merges with the gate record hash as a commit trailer. The deploy pipeline copies the exact evaluated image digest to Artifact Registry (no rebuild), deploys a baseline and the candidate as no-traffic revisions, checks staging, runs a canary at 10 % and 50 % with a client-side Mann-Whitney / Hodges-Lehmann decision, then promotes or rolls back. A rollback opens an incident that a responder flow annotates.
  7. A release v1.<iid>.<k> carries a behavior receipt (offbrief.receipt/1) that anyone can recompute offline with python -m offbrief receipt verify; the ledger head, the evidence digest and the contract digest are all in it.

How I built it

  • Five Duo Agent Platform flows (flows/): offbrief-intent, offbrief-builder, offbrief-judge, offbrief-fixer, offbrief-responder, plus a deterministic fallback judge. The YAML is generated from flows/spec/flows.yaml and flows/prompts/*.md by flows/build.py, which refuses anything the custom flow schema or my own rules forbid (no run_command outside coding_environment: full, no note tools for builder and fixer, exact-token routers with default_route). Flows never start flows; evidence travels by reference (note ids and digests), never inline.
  • The warden (warden/): one Cloud Run instance, ticked every minute by Cloud Scheduler, deployed only by the human's tofu apply. It reads GitLab with the PAT (Secret Manager, never in CI), starts flows through the Flows API, links every artifact to its flow session by run key and time window (never by author, because of composite identity), composes the gate, merges, serves the deploy pipeline every ledger-derived receipt fact (offbrief.deploy-task/1), brokers a 60-minute deployer token only for a recorded merge, and writes the only state it has: an append-only, hash-chained ledger in Cloud Storage. It never runs candidate code (app/ is not in its image).
  • The conductor pipeline (.gitlab/ci/conductor.yml): sandbox:three-way (one privileged dind job, COPY-only candidate build, pip install --require-hashes), then replay:stable-vs-stable, replay:stable-vs-candidate, egress:honeytoken-markers, secure:candidate-scan, secure:gitleaks, evidence:bundle and evidence:report as small unprivileged jobs.
  • The deploy pipeline (.gitlab/ci/deploy.yml): pin:verify, GitLab SAST / secret detection / container scanning / IaC-SAST templates, package:promote-digest (skopeo copy by digest), configure:tofu-validate, deploy:baseline, deploy:candidate, verify:staging, canary:10, canary:50, promote or rollback, release:receipt, release:create, govern:summary, govern:log-hygiene. A skeleton deploy mode exists as an honest fallback and is recorded in the receipt.
  • Guardrails (offbrief/pin/, warden/history.py): everything outside the agent area is pinned by a SHA-256 over path:blob_sha, recomputed from the GitLab tree API before every write; every first-parent commit on main must be a recorded merge or a recorded maintenance push. What a Developer-role service account can still do (push to main, auto-merge, create packages and releases with CI_JOB_TOKEN) is detected, not prevented: the warden freezes, opens one tamper incident and refuses every later action (docs/threat-model.md).
  • Infrastructure as code (infra/platform, OpenTofu): Workload Identity Federation (no service-account keys), seven service accounts with narrow roles, Artifact Registry with a keep-tagged cleanup policy, secret containers without versions, three Cloud Run services (patient-api, pa-stubs, offbrief-warden) and one job (traffic-gen), two Scheduler jobs, the ledger and receipts buckets, a monitoring dashboard and an alert-only budget. CI only runs tofu fmt -check and tofu validate; it never reads the state.
  • Engine and benchmark (offbrief/, benchmark/): replay classification with noise cancellation, marker scanning with an Aho-Corasick automaton, canary statistics, receipt build/verify, a 24-fixture benchmark that runs with no LLM and a per-fixture podman sandbox.

Challenges I ran into

Candidly, in docs/challenges.md:

  • Triggers fire only on human actions and flows cannot chain, so the chain is a scheduler-ticked service, not a flow.
  • The PAT is the only credential that can drive the loop; it lives in Secret Manager, is versioned by hand, never reaches CI or the OpenTofu state, and its re-pin window is documented rather than hidden.
  • Composite identity: MRs (and possibly notes) are attributed to the human account, so authenticity is a session link in the ledger, never an author field.
  • On an --internal container network --dns is ignored by both engines I tried; the orchestrator mounts its own resolv.conf so that DNS-label exfiltration is observable.
  • GitLab scoped labels sharing a key are mutually exclusive; the state transitions kept displacing my markers until every independent marker got its own scope.
  • Credits: the pool is unknown until the first session, so the design spends LLM credits only where a model is needed and caps every loop.
  • Maintainer-only features (CI/CD variables, protected branches and tags, catalog sync) shaped the design toward "pinned in the tree" and "detected, not prevented".
  • Platform behaviour the documentation does not settle is marked ASSUMPTION(D1-x) in the code with its fallback, and probed on day one.

Accomplishments that I'm proud of

From docs/benchmark.md (24 fixtures with green unit tests, seed-v0 corpus, first 300 of 1 000 requests, podman sandbox, whole run 375.8 s; Wilson 95 % intervals, wide on purpose with 18 and 6 fixtures):

Row Recall on B ∪ C ∪ D ∪ E False positives on A Misses
1. unit tests only 0/18 = 0.0 % [0.0 %, 17.6 %] 0/6 all eighteen
2. P vs C, no noise cancellation 11/18 = 61.1 % [38.6 %, 79.7 %] 0/6 five egress fixtures, two guardrail edits
3. legacy Diffy thresholds 11/18 = 61.1 % [38.6 %, 79.7 %] 0/6 the same seven
4. Offbrief deterministic, no judge 17/18 = 94.4 % [74.2 %, 99.0 %] 0/6 = 0.0 % [0.0 %, 39.0 %] c6-raw-ip-miss (the documented raw-IP limitation)
  • Row 1 is the thesis: every fixture ships green unit tests, and unit tests alone catch none of the eighteen regressions.
  • Four undeclared-egress fixtures that replay cannot see (c1 proxy bypass with trust_env=False, c2 DNS-label exfiltration, c3 base64 body, c5 marker in a query string) are caught only by the sandbox egress check; c4 (a disallowed PHI class sent to a declared partner) is caught by both replay and egress; the one miss is c6, the raw-IP case I document as a Tier 1 limitation.
  • Two guardrail tampers (.gitlab-ci.yml, build/requirements.lock) are rejected by the path policy before a conductor ever runs; removing the determinism hooks is reported as hooks_missing.
  • A 12-check deterministic gate, a hash-chained ledger, a behavior receipt that verifies offline, five flows, a nine-stage evidence matrix generated per run, and 2,489 tests (2,478 passed, 11 skipped at commit 60f381f).
  • Run A went from the issue to the promoted canary in about 45 minutes, at an estimated 29 credits (the warden's per-hop estimates), and its summary reads "Human actions in this loop: 1"; with the judge in the loop (benchmark row 5, 13 fixtures on the live platform) the full gate caught 9 of 9 regressions and passed all 4 benign changes.

What I learned

  • "Who wrote the evidence?" is a better question than "did the tests pass?". Unit tests, code review and the fix come from the agent; replayed traffic, per-run markers and a canary do not.
  • Noise cancellation earns its keep against candidates that evade the determinism hooks: with the hooks in place the two stable copies agree byte for byte and keys_raw == keys_after_cancel (61 → 61 keys across the benchmark).
  • Under a Developer role, honesty beats bravado: say detected and tamper-evident, and make every consequence observable in a ledger a stranger can verify.
  • In GitLab CI, stage order (not needs) is what gates production jobs: a job with needs starts as soon as its needs finish, whatever stage it sits in, so deploy:baseline got its own stage (.gitlab/ci/README.md).
  • Evidence by reference is not a style choice; a flow goal that embeds a report would hit the job environment limit.
  • Local podman and GitLab dind differ in small ways that matter (DNS on internal networks, SELinux labels, bind-mount roots); recording each finding in build/README.md was worth more than any abstraction.

What's next

  • Run more loops on real change requests, and grow the recorded corpus with them, so the replay keeps covering what production traffic actually does.
  • Publish the replay-gate CI/CD component in the CI/CD Catalog so other projects can put the same replay and egress gate in front of their own agent-made changes.
  • After judging: revoke the PAT, destroy the Google Cloud project, and keep the fork for later work.

Links

Built With

  • artifact-registry
  • claude
  • cloud-scheduler
  • cloud-storage
  • docker
  • faker
  • fastapi
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • google-cloud
  • google-cloud-run
  • httpx
  • mitmproxy
  • opentofu
  • podman
  • pytest
  • python
  • scipy
  • secret-manager
  • skopeo
  • sqlite
  • workload-identity-federation
Share this project:

Updates

Submission history