ProofFirst – Exploit-Verified Security Remediation

No fix without proof: an AI agent on GitLab Duo proves each vulnerability in CI, fixes it, and hands the evidence to a human for approval.

Comment

Inspiration

Security scanners flood teams with alerts. Some are duplicates, some are false positives, and real bugs still slip through, so developers learn to ignore them.

As a bug bounty hunter, I follow one rule: never report a bug without a proof of concept. Triage teams reject anything that can't show impact. ProofFirst brings that rule into CI/CD: an agent must prove a vulnerability is real before anyone spends time fixing it.

What it does

You assign an issue to the ProofFirst service account. One agent on the GitLab Duo Agent Platform then runs the whole post-code security loop:

  1. Triage: reads the SAST report and the code, removes duplicates and false positives, and flags what SAST missed.
  2. Prove: maps each candidate to a vetted, non-destructive PoC template, fills it in with sed, and pushes it to its own branch. CI goes red: the vulnerability is proven.
  3. Fix: applies the smallest correct fix without touching the tests. CI goes green: the fix is verified.
  4. Release: opens a merge request with both pipelines, CVSS 3.1 scores and a reviewer checklist. A human approves and merges.
  5. Package and govern: the fixed app is built and pushed to the Container Registry, and the issue closes with a full audit trail. A guardrail prevents the agent from ever writing to main.

Result on the demo app: 6 findings in → 3 proven, 1 false positive, 1 duplicate, 1 missed by SAST (a path traversal the scanner never reported).

The red pipeline captured real impact: the SQL injection leaked 3 rows including the admin account, the injected marker command really ran on the server, and ../app.py was served with HTTP 200.

How we built it

  • GitLab Duo Agent Platform: a custom flow (flows/prooffirst-flow.yml) with one agent, triggered by assigning an issue to its service account.
  • Vetted PoC template library (security/poc_templates/), one template per CWE (89, 78, 22). The agent never writes exploit code itself.
  • GitLab CI/CD: SAST (Semgrep), Secret Detection, pytest, and a Docker build pushed to the Container Registry.
  • A deliberately vulnerable Python/Flask demo app as the target.

Challenges we ran into

  • Multi-agent handoffs: my first design used 4 agents (triage, PoC, fix, report). Across 7 iterations, agents after the first one kept receiving empty input. A single agent with clear phases ran reliably end to end.
  • Exploit generation: the model wouldn't freestyle exploit code, even for my own demo app. A library of vetted templates solved it, and turned out to be the safer design: every PoC is reviewable in advance.
  • Safety: in one early run an agent pushed straight to main. That led to the guardrail: the agent checks its branch before every commit and push, and only a human merges.
  • No packages in the agent sandbox: so CI became the proof. A red pipeline followed by a green one is evidence anyone can check.

Accomplishments that we're proud of

  • A fully working loop: issue → triage → 🔴 proof → 🟢 fix → MR → human merge → packaged image → issue closed.
  • The agent found a real bug that SAST missed, and correctly set aside a duplicate and a false positive.
  • Every claim in the merge request is backed by a CI pipeline that anyone can open.

What we learned

Proof beats opinion. A scanner's guess and an LLM's guess are both just guesses; a failing test that turns green after the fix is evidence. I also learned that small, well-scoped agent designs and vetted building blocks are more reliable than long chains of agents.

What's next for ProofFirst

More templates (SSRF, IDOR, XSS, deserialization), a Release stage with tags and release notes, a post-deploy health check, and running ProofFirst automatically on every SAST pipeline.

Evidence (all public): MR !2 · red pipeline · green pipeline · issue #9

Built With

Share this project:

Updates

Submission history