INSPIRATION
AI writes code faster than people can review it. Security scanners warn about possible problems, but they can miss real ones, like a user reading someone else's data. We wanted a merge to be earned: it has to survive an attack first.
WHAT IT DOES
Trial by Fire stops insecure code from merging, and then helps fix it.
- Attack tests act like a hacker against a small notes API: SQL injection, reading another user's note, missing login, a default admin key, and setting yourself as the owner.
- Merge gate: when a merge request is opened, GitLab CI runs the attacks. If one gets in, the pipeline fails and the rule "Pipelines must succeed" blocks the merge.
- Auto-fix help: a trigger on "pipeline failed" starts GitLab's Duo Fix CI/CD Pipeline flow by itself. It reads the failed job, names the failing test and the root cause, and posts an inline suggested fix on the code.
- Verification: a person applies the suggestion, the same attacks re-run, and the pipeline goes green. The merge unlocks.
- Anti-cheat gate: a second job (attacks-intact) fails any merge request that deletes or weakens the attack tests, so a pass means something.
HOW WE USED GITLAB TO AUTOMATE THE POST-CODE LIFECYCLE
- GitLab CI/CD: unit tests, SAST, Secret Detection, the red-team attack job, the anti-cheat gate and a report job.
- Merge request rule: "Pipelines must succeed" as the merge gate.
- GitLab Duo Agent Platform: a flow trigger (pipeline failed) running the Fix CI/CD Pipeline flow, and the Security Analyst agent for review in Duo Chat.
- Lifecycle stages touched: verify, secure, govern, and create (the suggested fix). We do not claim package, release, configure or monitor.
AUTONOMY LEVEL
Assisted. The Duo flow starts automatically, but a person applies the suggestion. The pipeline re-run then verifies it.
EVIDENCE
Three demo merge requests, all public in the repository:
- MR 1, SQL injection: GitLab's scanner flags 1 high finding, and the attack job fails.
- MR 2, an access-control bug (IDOR): the scanner reported no new vulnerabilities, but the attack job fails. The Duo suggestion was applied, and the pipeline passed.
- MR 4, mass assignment: the attack job fails, the Fix flow suggests removing two lines, and applying them turns the pipeline green.
These are examples, not a measured detection rate.
WHAT DID NOT WORK (STATED PLAINLY)
- The attack tests are written by us, not by an AI. We drafted custom Duo flows meant to write them (the flows and agents folders), but they did not produce reliable results, so we do not claim them. GitLab staff diagnosed an early issue (a missing project ID and a completion instruction). After fixing it, our simple test flow worked, but our red-team flow still did not produce a test.
- Duo's fix flow could not run tests in its own sandbox (no network for pip). Its suggestions are unverified until our pipeline re-runs them.
- No deployment and no measured benchmark. The target app is a small prop, and the automation is the product. The carbon (SCI) script in the repository is an estimate with stated assumptions.
WHAT WE LEARNED
- A passing scanner is not proof of safety. A test that actually attacks the app catches bugs a scan can miss.
- Guardrails matter for agents: attack tests must be protected from edits, so "it passes" means something.
- Reading the platform's own docs and asking its maintainers solved a problem in hours that guessing had not.
WHAT IS NEXT
Make an AI agent write the attack tests from the diff, with the anti-cheat gate protecting them, and measure how many seeded bugs each approach catches.
Built With
- agent
- duo
- fastapi
- gitlab
- gitlab-ci-cd
- gitlab-duo-agent-platform
- pytest
- python
- sast
- secret-detection
