Inspiration Most security bugs aren't new. They're the same mistake made again in a different file, months after someone fixed the first one. The lesson from each fix lives in one person's head or in a postmortem nobody reads. I wanted the codebase itself to remember.
What it does
When a developer merges a fix and mentions OnceBitten, three Duo agents take over:
- read_bite reads the fix MR and works out what bit us (class, CWE, the vulnerable lines).
- forge_mark writes a Semgrep rule shaped like the bug (a Bite Mark) and proves it with a deterministic prover: the rule must flag the vulnerable commit, stay silent on the fix, lose matches across the fix, and stay precise on today's code. A rejected rule is revised. It then fixes every sibling the proof lists and runs the tests.
- publish opens a merge request with the mark, the proof certificate and the sibling fixes.
A human reviews the outcome and merges: the only approval in the loop. From then on, the Twice Shy gate fails any MR that re-adds the pattern and says when and where it bit before, and the Scar Map (GitLab Pages) counts every bite remembered and every re-bite blocked from real pipeline history. A custom Ask OnceBitten chat agent answers "why was my MR blocked?".
In this project: a SQL injection fix (!1) became a proven mark and fixes for two siblings, one of them written as an f-string (!2). A later feature that repeated the mistake (!3) passed its own tests but was blocked by Twice Shy, and GitLab's own Fix Pipeline agent read that memory and proposed the same fix as !1.
How we built it
- GitLab Duo Agent Platform: a custom flow (flow registry v1) with three AgentComponents,
inline prompts and scoped tool sets; a mention trigger on the flow's service account;
.gitlab/duo/agent-config.ymlto install Semgrep and the prover in the flow runner; a custom chat agent; a repo skill. - The prover (Python): git + Semgrep, no model. It produces a Markdown proof certificate. Every number shown anywhere comes from it, never from the LLM.
- GitLab CI/CD: tests on every pipeline, the Twice Shy gate on MRs with a JUnit report in the MR widget, and a Pages job that builds the Scar Map from the repo and the jobs API.
- Demo codebase: Tally, a small Flask + SQLite invoicing API with deliberately planted bugs (disclosed in the README).
Lifecycle stages
Create (agent branch, mark, sibling fixes, MR) · Verify (proof against real commits, tests, gate) · Secure (CWE-tagged detectors, sibling remediation) · Govern (Twice Shy gate, proof certificates, service-account audit trail) · Monitor (Scar Map from pipeline history) · Configure (marks as policy-as-code, flow environment config) · Plan (mention-driven work, session logs). Not touched: Package, Release.
Challenges
- The flow's default image ships
python3.12/pip3.12only; the first run failed withpip: command not founduntil the setup script was fixed. - A flow can't be woken by its own pipeline (only humans fire triggers), so verification lives inside the agent's own loop (prove → revise) and in deterministic CI, not in a second flow.
- Making the proof robust: guard-style fixes (adding a check without changing the dangerous line) and siblings with identical text required matching findings by disappearance, not by line.
- Semgrep needs a writable home directory, which agent sandboxes don't have.
Accomplishments
- The first real run proved a taint-mode rule against the real fix and fixed two siblings in under three minutes, with no human edits.
- GitLab's built-in Fix Pipeline agent used OnceBitten's memory without any integration work.
What we learned
Agents are most trustworthy when they're paired with a deterministic judge. The LLM is good at generalising a bug's shape; git and Semgrep are good at proving it.
What's next
- A benchmark against real public CVE fix commits.
- A scheduled re-hunt of every mark across all projects in a group.
- Marks for other languages and CodeQL as a second prover backend.
Log in or sign up for Devpost to join the conversation.