One Approval

Inspiration

A prompt tweak to an AI app can pass every test and quietly make answers worse. Meanwhile every release still needs a person to read five browser tabs before clicking deploy. We took a small RAG app (Doc Chat) that shipped with no CI, no tests, no evals and no scans, and asked what the shortest honest path from issue to production looks like when agents write the change.

The obvious answer is "issue, agent writes an MR, agent reviews, pipeline, deploy, a human clicks approve at the end". We wanted to answer the harder question: when does the human have to look at all? Our answer is a file in the repo, not the agent's opinion and not a fixed rule.

What it does

An issue becomes a draft merge request through the GitLab Duo Agent Platform. CI classifies the change using release-policy.yml, then runs only the checks that class needs (lint, tests, build, a RAG quality eval, security scans). After a merge, the image is built, scanned, deployed to staging, smoke-tested, and a production pipeline is generated from the same policy: automatic for a docs change, waiting for one approval for anything else.

Each merge request also gets an evidence pack: risk class and rule, a table of checks with links to the jobs, the eval score against its baseline, and which flow or person did each step.

Why Supervised: the pipeline and the policy clear the gates on their own (classification, lint, tests, build, eval, scans, evidence pack, staging deploy and smoke test); a person approves the outcome, which is the production deploy. Starting each flow is a person's click because flows do not chain at our access level; we do not claim the flows run end to end by themselves.

How we built it

• GitLab Duo Agent Platform: the Developer Flow turns an issue into a draft MR; the Code Review Flow reviews it. AGENTS.md gives them standing rules (they refused to edit a test to get green). Flows are started by a person: we tested triggers, the Flows API and tokens, and flows do not chain at our access level (see limits).
• GitLab CI/CD: merge request pipelines, a main pipeline, and a generated child pipeline for production whose manual or automatic gate comes from the policy (failing closed if the policy is unreadable). Path-based skipping keeps docs-only changes cheap.
• The policy file and classifier: release-policy.yml plus scripts/classify.mjs, with tests. The highest class among the changed files wins; an unknown file counts as ui.
• Eval gate: 10 questions about a sample handbook, deterministic pattern scoring (no model as judge), compared with a stored baseline, cached to keep cost small.
• Security: SAST, secret detection and dependency scanning on every MR; SBOM and image scan (Trivy) on main.
• Google Cloud: Cloud Run (staging and production), Artifact Registry, Secret Manager, and Workload Identity Federation so CI holds no long-lived key. Google issues deploy credentials only to jobs started by the approver's GitLab login.
• Monitor: a pipeline schedule probes production every 6 hours.
• The app: Doc Chat (Next.js, Claude, OpenAI embeddings, Supabase pgvector), with an upload switch, size and rate limits, and a preloaded sample so judges can ask a question in one click.

What we added on or after Oct 5

Everything except the original Doc Chat chat, upload and retrieval code: the release policy and classifier; CI pipelines; the eval harness; the evidence-pack builder; Cloud Run deployment with keyless auth; app guards; scanning, SBOM and the monitor; 158 tests; the docs. The original project is github.com/pravoobi/doc-chat (MIT).

Challenges we ran into

• Flows do not chain. GitLab says triggers need a human. We tested it: the MR-created trigger fired for a human's MR in 4 seconds and not for a flow's. A flow starting another flow through the API was refused (HTTP 400), and a token that could start flows from CI would also let a pipeline merge as the approver. So a person starts each flow.
• No protected environments or code-owner enforcement at our access level (the organizers confirmed). The production approval is enforced by Google Cloud instead, which admits deploy credentials only for the approver's login.
• Our own gate was wrong. The eval failed three times on a correct refusal worded "I don't see any information", because the scorer did not know that phrase. We found it by reading the answers, fixed the scorer in its own MR with the real answers as tests, and kept the baseline.
• A real vulnerability. The new image scan found a CRITICAL remote code execution in the Next.js version production was running. Fixed by upgrading (13 findings to 3, none CRITICAL), after four tries at a lockfile that works on Linux.

• Windows and CI disagree about lockfiles; local checks passed what CI rejected.

• The eval does not test what production runs. The app answers with Sonnet; the eval uses Haiku and its own in-memory retrieval, so it covers the prompt and chunking but not the chat route or retrieval code. Found in our own security review and stated, not hidden.

• A security review of our own demo found a rate limit that a made-up header defeated, unchecked chat input, a classifier that missed moved files, and public/** classed as documentation. All fixed, each with a test that fails on the old code, and verified on the live app.

Accomplishments that we're proud of

• Three real end-to-end runs, kept public as the demo set. Approvals the policy requires: docs 0, ui 1, ai 1.
• A docs-only change reached production in about 6 minutes with no manual job.
• The Developer Flow was seen keeping a gate more than once: on issue #7 it refused to weaken a test and asked, and on every MR it reported exactly what it could not run (the eval, the build) and left that to CI.
• The scan found and we fixed a CRITICAL RCE in our own demo. We say so in the README.
• Everything is reproducible from public merge requests and pipelines.

What we learned

• Reviewing your own project like a stranger would pays off: the live rate limit looked right in the tests and was bypassable with one header.
• Deterministic gates make agents trustworthy: the flows acted on pass/fail from scripts, and said when they could not verify something themselves.
• A phrase-based eval is reproducible but brittle to rewording, and it was too coarse to see a 17 times smaller chunk size (still 10/10). That is a limitation we state, not hide.
• An AI reviewer's approval must never count as the human decision: the Code Review Flow approves MRs as an AI account.
• Platform limits worth knowing: flows run on Node 20 with no API keys, so CI is the only place the eval runs; schedule variables could not be set at our role.

What's next for One Approval

Not built yet: the intake, security-triage, release-captain and watchdog custom flows; an incident issue and rollback MR for the monitor; the GitLab Pages story page; posting the evidence pack as an MR comment. The image scan is report-only and should become a gate on fixed CRITICAL findings. The eval needs multi-sentence questions. One HIGH finding (brace-expansion) is open.

Built With

• artifact-registry
• claude
• claude-(anthropic)-via-the-vercel-ai-sdk
• code-review-flow
• dependency-scanning)
• developer-flow
• environments
• gitlab-ci/cd-(merge-request
• gitlab-duo-agent-platform
• gitlab-security-scanning-(sast
• google-cloud-run
• main-and-dynamic-child-pipelines
• next.js
• node.js
• oidc-`id-tokens`)
• openai-embeddings
• pipeline-schedules
• secret-detection
• secret-manager
• supabase-(postgres-and-pgvector)
• trivy
• vitest
• workload-identity-federation