testrazor: issue to production, hands-off

Open an issue and walk away. A GitLab Duo flow ships it through a test gate it can trust, security scans, staging and verified production, or reports BLOCKED.

Comment

Open an issue. Walk away. It ships, or it tells you why not.

Inspiration

A hands-off agent is only as safe as the green check it merges on. Full suites are slow, and a flaky test turns a correct change red at random. An unattended agent then stalls, or retries until red goes away, and that retry loop is how a real bug gets merged. testrazor (our MIT CLI) was built to make green trustworthy, so we gave it to an agent.

What it does

You assign an issue to the testrazor issue-to-production GitLab Duo flow. Nobody touches anything after that.

  1. Create: the flow writes the change and a test that fails without it, and opens one merge request.
  2. Verify: testrazor impact runs only the tests the change reaches, twice. A test that both passed and failed is quarantined; a real failure always blocks.
  3. Secure: GitLab SAST and Secret Detection. High, Critical or any secret blocks.
  4. Release: a protected main pipeline merges only the exact SHA that passed every gate, deploys staging, checks it serves that commit, promotes the same deployment to production and checks again.
  5. Monitor: if production does not serve the merge commit, it rolls back on its own.
  6. Report: the flow verifies production itself and posts a SHIPPED or BLOCKED report on the issue.

All nine lifecycle stages

Stage What runs
Plan The issue is the intent; the flow restates its acceptance checks
Create Code plus a test that fails without it, on duo/issue-<iid>, one merge request (resumed, never duplicated)
Verify testrazor-gate: impact selection, two runs, flake scoring, gate; JUnit in the MR widget
Secure GitLab SAST (Semgrep) and Secret Detection, then security-gate; a missing report fails
Package Site artifact with version.json and gate.json, built only if the gates passed
Release merge-mr on protected main with a sha= guard, staging, promotion of that same deployment, GitLab release
Configure staging and production environments, protected variables, pinned agent image (.gitlab/duo/agent-config.yml)
Monitor Smoke checks require the expected commit SHA and PASS, not just HTTP 200; automatic rollback
Govern Edit allow-list, no test weakening, 3-attempt fix budget, audit report on the issue

How we built it

  • GitLab Duo Agent Platform custom flow (.gitlab/duo/flows/issue-to-production.yml), published in the AI Catalog, triggered by assign and mention, following docs/AGENT_RUNBOOK.md.
  • GitLab CI/CD: testrazor gate with JUnit in the MR, SAST, Secret Detection, a blocking security gate, staging and production environments, smoke checks, rollback and a GitLab release per deploy.
  • Guardrails: the flow may edit only src/, test/, README.md and docs/; it cannot touch CI, the gates, the quarantine or the lockfile; 3 fix attempts, then BLOCKED. Deploy and merge tokens are protected variables that MR pipelines never see.

Challenges we ran into

The recorded run (issue #1, 2026-10-27) hit two real problems. Our first MR pipeline failed on our own CI config; the flow rebased onto the fix. Then GitLab refused the flow's merge (HTTP 403: flow tokens cannot merge). The flow did not look for a way around it. It posted BLOCKED with the exact error. We added a pipeline-side merge with a SHA guard, and the flow resumed the same merge request. Final result: 32 tests run twice, 0 flaky, 0 real failures, production verified serving 4f7e605a, release deploy-13.

Proof you can check

  • Issue #1: BLOCKED report at 11:22 UTC, SHIPPED report at 11:57 UTC, both posted by the flow's service account.
  • MR pipeline 2994097853 and main pipeline 2994109303 (staging, production, release deploy-13, rollback skipped).
  • curl https://testrazor-transcend.vercel.app/version.json shows the serving commit; the page's "This deploy" card shows it too.
  • evidence/run-issue-1/manifest.json: SHA-256 of every automation file, every pipeline and job status, and six assertions read from the GitLab API and the live URL (all true).

Honest limits

  • testrazor ran all 10 test files on this change because README.md changed and cannot be mapped to tests (fail-safe). For the code files alone it selects 1 of 13, measured locally.
  • Flow tokens cannot merge, so the merge job uses the owner's protected, masked token; the flow and MR pipelines cannot read it.
  • Path B: testrazor (CLI, import graph, flake scoring, gate) predates Oct 5. The flow, runbook, pipeline, gates, merge path, deployment and evidence were all built on 2026-10-27. The feature shipped in the run (impact --changed-from) was written by the flow.
  • Deployed on Vercel. No Google Cloud bonus claimed.

What we learned

A blocked release is a valid outcome. The most useful thing the agent did was stop and say why.

Built with

GitLab Duo Agent Platform (custom flow, AI Catalog, triggers, composite identity), GitLab CI/CD, GitLab SAST, GitLab Secret Detection, GitLab environments and releases, TypeScript, Node.js 22, Vitest, Vercel.

What's next

A pipeline-event trigger so a failed main pipeline opens an incident, and a fine-grained merge token.

Links: GitLab project · Deployed app · Explainer · Issue #1 · Evidence · Original MIT project (pre-hackathon). Built on 2026-10-27 (Path B); not deployed on Google Cloud.

Built With

  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • gitlab-sast
  • gitlab-secret-detection
  • node.js
  • typescript
  • vercel
  • vitest
Share this project:

Updates

Submission history