Custos — Always-on GitLab Repo Custodian

Custos — a GitLab custodian that grows into your team's senior: Gemini plans & calls GitLab tools, 4 personas review, then it self-evolves.
Live review on a real MR: agentic triage + the GitLab MCP tools it called, 0–100 risk, and BLOCKs on a secret, SQLi & command injection.
Live dashboard: both lanes, the Semgrep 30+ language SAST engine, and a feed separating GitLab MCP tool calls from applied Custos skills.
Self-evolution: active/induced/quarantined skills + the two-ledger Acceptance-vs-Quality split — learning from what truly breaks.
Senior Score 91/100 — recall, calibration & anti-distraction on a labeled golden set. 'Senior' as a measured trajectory, not a claim.
Proactive lane: an hourly sweep of already-merged code auto-files deduped, prioritized GitLab issues — no PR required.
Inside one skill: when it applies, its steps, and its two-ledger confidence — the unit Custos tunes from reactions & post-merge outcomes.
Every diamond is a real gate in the code - MCP fallback, dedupe, post-merge truth check. Violet = agentic core + self-evolution.
The self-evolution loop: post-merge truth (x3) -> Gemini induces a rule -> golden FP-gate -> quarantine -> graduate -> Senior Score 91.

Inspiration

Today's AI code-review bots (Copilot Review, CodeRabbit, GitLab Duo) are passive. They wake up when a PR opens and forget the code the moment it merges. Review, triage, and fixing live in separate tools — so OSS maintainers and small-team tech leads repeat the same chores every week: reviewing incoming PRs, finding post-merge bugs in already-merged code, filing issues, and writing fix MRs by hand.

A good senior does the opposite. They own repo health end-to-end and adjust to the team's standards over time. Custos solves that thesis on top of GitLab:

(a) Autonomous operation — review and post-merge watch run without a human in the loop (two lanes).
(b) Goal → plan → act → human oversight — given a merge request, it plans for itself, selects and runs the GitLab tools it needs, and decides from what it observes. The final merge decision stays with a human.
(c) Evolution — it takes the maintainer's reactions (merge · 👍 · follow-up commits) and post-merge outcomes as learning signals to recalibrate its own standards.

It gives you that Sunday afternoon back.

What it does

Custos lives inside your GitLab repo as two lanes:

Reactive Lane — when an MR opens, four personas (Security Guard 🔒 / Performance Hunter ⚡ / Test Fanatic 🧪 / Team Lead 👔) analyze it in parallel within 60 seconds, posting a 0–100 risk score + inline comments. The GitLab CI pipeline status folds into the score.
Proactive Lane — a scheduled sweep scans already-merged code, dedupes new defects, and auto-files prioritized GitLab issues. No PR required.

Trust is earned incrementally through three permission modes (dry-run → comment-only → full), with an instant kill switch.

Incident memory — Custos remembers past-incident (incident-labeled) merge requests and automatically recalls a "⚠️ related to incident #N" warning on new MRs that touch the same area. It's the smallest working form of self-evolution.

Auto-Fix Lane (issue → automated fix MR) is on the roadmap — the safeguards and GitLab client methods are done; the live merge path is deferred past the deadline.

How we built it

The agent — decomposing a goal into a plan it executes

The four personas are built as parallel deterministic checkers (same input → same result, measurable).
On top of them, agentic triage runs live. Google ADK's LlmAgent (Gemini) decides — over GitLab's native MCP server — which read tools to call: diffs, pipeline status, commits, notes, file contents, code search, and GitLab's SAST report.
The plan → act → decide loop is surfaced at the top of the MR comment and on the dashboard: it plans what to do, calls tools, and routes from observations (which persona to emphasize, how deep to review).
Agency proven by measurement: different inputs change Gemini's tool choices — a security MR pulls diffs + SAST report, a CI-broken MR pulls diffs + pipeline. Confirmed live in production — 77% agentic over GitLab's MCP (no fallback on these tool-selection runs). An agent that judges, not a fixed script.
A clear boundary: vulnerability findings are produced by deterministic checkers (measurable on a golden set). What the agent does is plan, select tools, and route → adaptive without losing accuracy.

Multi-language static analysis — Semgrep

SAST runs Semgrep across 30+ languages (Python, JS, TS, Go, Java, Ruby…) in a dedicated Cloud Run service, producing deterministic findings; it falls back gracefully to a local run when the service is unavailable.
Coupled with evolution: Custos synthesizes new deterministic detection rules (regex) from past incident patterns, adopting one only after it passes zero false positives on a golden-negative set → the detection engine itself grows with the team.

GitLab as the substrate

Production runs live over GitLab's native MCP server (/api/v4/mcp) (native MCP / community MCP / in-process REST selectable as transports, with fallback).
It drives the GitLab workflow directly: review (read) · folds the CI signal into the risk score · writes issues and inline comments.
Governance (maintainer Approval Rules) and hands-free merges (Merge-When-Pipeline-Succeeds) are client-complete; the live path ships with Auto-Fix on the roadmap.

Inference · memory · backend

LLM: Vertex AI Gemini — prefers the Gemini 3 family (Gemini 3.1 Pro for analysis · Gemini 3.5 Flash for triage) on Vertex AI's global endpoint (where 3.x is served), with automatic graceful fallback to the GA Gemini 2.5 family (H-M-13) · text-embedding-005 for embeddings. The Cloud Run service + Firestore run on europe-west1.
Skill Library: seed skills are embedded and searched via Vertex AI Vector Search; on cold start (small library) it falls back to brute-force + LLM rerank.
Backend: FastAPI on Cloud Run (Python 3.13) + Firestore. Source code snippets are never persisted — a finding stores only the message + file:line.
Frontend: a landing page and dashboard SPA (React) served from the same Cloud Run — 2-lane status, Senior Score, evolution trajectory, PR detail, daily digest.
Slack: one-way per-incident alerts + a daily Block Kit digest.

Data sources

Our own seed skill library + a labeled golden set (for measuring persona precision/recall) + runtime GitLab MR/diff/CI data. No external datasets or crawling — the only learning signal is the maintainer's reactions (as aggregate counts).

Self-evolution — learning from "truth," not "consent"

We made the self-evolution stack real in code + tests, not slideware.

Truth spine — merged doesn't mean correct. A real post-merge incident is the truth signal: it recalibrates the confidence of the skill that raised (or missed) that warning. One truth signal ≈ 3 plain acceptances.
Two-ledger — quality signals and acceptance signals are recorded separately. Where they diverge is the evidence that "acceptance ≠ correctness."
Induction — only incident-validated patterns graduate into new skills (golden false-positive gate + candidate quarantine).
Incident recall — related incidents are recalled via a 1-hop graph expansion.

→ So Custos measures itself getting more accurate, not just more agreeable.

Security

Webhooks: HMAC constant-time signature verification + idempotency dedupe.
Triple recursion guard: the bot's own events can't re-trigger an infinite loop — blocked at the bot-account, metadata, and idempotency layers.
Prompt-injection isolation: persona LLM input is isolated as untrusted data; the system prompt is immutable.
PII minimization: maintainer identity (name/email) is never stored — reactions are aggregate counts only. Source code isn't persisted either. Slack output auto-masks code, mentions, and secret prefixes.
Least-privilege bot: the bot account holds Developer rights only (no Maintainer) and can't push to protected branches directly.
Vertex AI data governance: explicitly opted out so customer data isn't used for model training.

Challenges we ran into

Recursion loops — the path where a bot-created MR re-enters the webhook and triggers itself, cut with a triple guard.
Cold-start retrieval — with few seed skills, vector-search accuracy is low, so we added a brute-force + LLM-rerank fallback.
Cost control — Proactive sweep downgrades to the Flash model, and a file-hash cache skips re-analyzing unchanged files.
Agent ↔ Cloud Run connectivity — the async Firestore client couldn't establish connections on Cloud Run; switching to a sync transport fixed it (webhook ACK 30s hang → 0.4s).
Adaptive yet measurable — the agent only selects tools and routes; findings stay deterministic. Holding that boundary gave us both debuggability and agency.

Accomplishments we're proud of

Live on a real GitLab repo — when a human pushes an MR, a 4-persona review + the agent's plan trace appear within 60 seconds.
Deterministic synthesis — same input → always the same risk score (weighted average + veto).
Deep GitLab platform use — review (read) · CI-signal folding · issue/comment writing, all driven over native MCP.
A measurement backbone — a labeled golden set measures persona precision/recall/noise. "Becoming senior" is shown as a measured trajectory, not marketing — offline rule-engine recall 0.96, Senior Score 91/100. Even the bot's own false positives are captured as golden negatives and pinned by a gate.

What we learned

Multi-agent needs deterministic synthesis — even with LLM personas, the synthesis must be deterministic (weighted average / veto) to be debuggable.
The essence of an agent = the LLM *choosing its own tools* — starting from a deterministic pipeline and layering agentic tool-use as an orchestration layer was the right call.
Self-evolution has to be honest — chasing acceptance alone only makes a bot more agreeable. Without anchoring to truth (post-merge outcomes · golden set), you optimize for consent.

What's next

Auto-Fix Lane — turn filed issues into automated fix MRs. The safeguards (risky-file blocking, safe-patch-only MWPS, maintainer approval gate, prompt-injection defense) are designed and implemented; live integration is post-deadline. Invariant: writes / auto-fixes only ever go through a human-approval path.
Agent action orchestration — let the agent autonomously select the write tools above, too.
GitHub backporting (this submission is GitLab-only per the rules).