Gauntlet

• 

  The Flows page with the Builder and Breaker enabled.

• 

  The first failed pipeline, with the SQL injection in the evidence-gate log.

• 

  The Builder session on the tax issue, where it refused to change code without a reproduction.

• 

  The README Results table

• 

  The merge request security report showing the injection and Medium findings as fixed.

• 

  The evidence packet comment reading READY TO MERGE.

• 

  The Builder session on the coupon issue, showing its Changes list.

• 

  README on main, with the Results section.

Inspiration

AI coding tools can write a fix in minutes, but nobody can trust it yet. A fix might pass the tests the AI wrote for itself, miss an edge case, or add a security hole, and a team has no record of what was checked before the change shipped. I wanted a pipeline that makes AI-written code earn trust before it merges.

What it does

Gauntlet is a GitLab pipeline that checks code before it can reach the main branch. Every merge request runs unit tests, adversarial tests, GitLab's static security scan (SAST), and secret detection. The code deploys to a staging environment, and a smoke test checks it. A gate job then posts an evidence packet on the merge request listing the test results, the security findings, and the smoke test. The gate blocks the merge if a required check is missing, the smoke test fails, or the scan finds a High or Critical issue. If the scan report is missing, the gate fails closed.

The project is built around two GitLab Duo Agent Platform custom flows. A Builder agent works on one issue at a time, changes the app code, and opens a merge request with "Closes #N". A Breaker agent writes adversarial tests for a merge request. Both flows are enabled and ran during the project, and a human reviewed each merge before it landed.

How I built it

The demo app is a small Python pricing and invoicing service with four seeded bugs: a discount boundary error, tax rounding and return-type errors, a coupon that can make a total negative, and an SQL injection. The pipeline is defined in .gitlab-ci.yml. It uses GitLab's SAST and secret detection templates with rules that run the scans on merge requests. The gate is a Python script that reads the test reports, the scan report, and the smoke test result, then writes the packet through the GitLab API. The agent instructions live in agents/, and the Builder and Breaker prompts were written from those files.

Challenges I ran into

The first pipeline failed before any job ran because of a YAML problem in an older commit. The security scans did not run on merge requests at first, so the gate passed a merge with "Report not available." I changed the gate to fail closed. I also had two copies of the project, and pushing an older one overwrote the fixes until we caught it in a failed pipeline and repaired it through a merge request.

The Builder hit a limit I did not expect: its tools can read and write repository files, but it cannot run Python. When I asked it to fix the tax issue, it refused to change code without a reproduction, which was the right call. Its earlier session also reported that a bug was already fixed, which was true only after a commit that wasn't its own. The hidden benchmark caught both gaps.

Accomplishments that I am proud of

The gate caught a real SQL injection on its first run. Security scans now run on every merge request, and the report shows the injection and three Medium findings as fixed, with nothing new. A Builder agent fixed the coupon bug and its merge request touched only the expected files. The Builder also refused to make an unsupported change, and it correctly reported on another issue that the fix was already present.

What I learned

A passing pipeline can still miss things. Starter tests passed on code that contained all four bugs. Adversarial tests written from the issue text caught all four. A gate is only as good as the reports it reads, so we made it fail closed rather than trust a missing report. Agents need evidence, not just instructions, and I learned to check every "fixed" claim against code and tests.

What's next for Gauntlet

Build the Triage agent so issues arrive with acceptance criteria and risk levels. Decide whether to add the three checks the hidden benchmark still tests (rejecting negative quantities, and case-insensitive coupon codes) as separate issues, rather than fitting them into existing ones. Give the Breaker the ability to commit its own tests to a branch with a path restriction, so no human copy step is needed.

Built With

• duoagent
• gitlabapi
• gitlabci/cd
• gitlabsast
• python
• sqlite
• unittest