LLMSlim Budget Guard

GitLab Duo agents review, secure, release and deploy LLMSlim, a context-compression library: they guard its token budgets and read compressed pipeline digests, not raw logs.

Comment

Inspiration

LLMSlim compresses the context that AI agents send to models. Its failures are quiet: a change that keeps 5% more tokens, drops a number from a retrieved document, or lets untrusted RAG text outrank a system instruction still passes ordinary unit tests and still ships. Code review, security scanning and releases for the project were manual, and nothing compared a change against its base on the library's own benchmark. We wanted the life after code to be guarded by agents that understand what this library promises.

What it does

  • MR Guardian (GitLab Duo flow): when a merge-request pipeline finishes, four agents read a compressed digest of the run, tie each token-budget or quality regression to the lines that caused it, triage security findings, and post one review note with labels. Only the reviewer agent can write.
  • Budget gate: benchmark:budget-guard runs LLMSlim's offline benchmark on the MR head and its merge base in isolated worktrees and blocks on 20 metrics: safety (provenance, must-keep and priority violations must stay 0), retention of instructions, entities, numbers and negations, tokens shipped, tool-retrieval quality, and latency (warning only).
  • Pipeline Doctor (flow): on failed pipelines it finds the root cause, comments a fix or opens a small fix MR for a human to merge; after a failed production rollout it opens an incident issue with the SLO numbers and the rollback result.
  • Release Captain (flow): derives the next version from Conventional Commits, opens the release MR with the changelog; on merge, CI tags, creates the GitLab Release and publishes with OIDC trusted publishing.
  • Issue Planner (flow): labels new issues and writes acceptance criteria that include the budget invariants.
  • Deploy and monitor: the live service runs free on Vercel (deploy:vercel: the same request handler and report page as the container image, then the same smoke test). The pipeline also contains a keyless Google Cloud Run deployment (Workload Identity Federation): a no-traffic revision with its own URL, a smoke test, a manual production gate, a two-minute SLO probe and automatic rollback. It is ready, but not active without a billing account.
  • Report page: the deployed service shows the latest run, budget verdict, agent decisions and SCI estimate, using only numbers from the project's own CI artifacts, plus a live compression demo.
  • Governance: a tool policy grants each write tool to one flow component; CI fails if a flow breaks it. Flows can never dismiss vulnerabilities.

How we built it

  • GitLab Duo Agent Platform: four custom flows (flow registry v1, ambient), triggers on pipeline, MR, mention and work-item events, three custom chat agents, Code Review Flow instructions, AGENTS.md, an Agent Skill, an agent-config.yml, and LLMSlim exposed as an MCP server.
  • GitLab CI/CD: 10 stages, SAST, secret detection, dependency and container scanning templates, pip-audit, CycloneDX SBOM, licence policy, JUnit and Cobertura reports, metrics reports, environments, resource groups, release keyword, OIDC id_tokens.
  • Python standard library for the service and the automation (no web framework), React and Tailwind for the report page. Live on Vercel's free Hobby plan (function region arn1, Stockholm); the Cloud Run configuration targets europe-north1.
  • The key idea: duo:context-bundle reads every artifact once and compresses failing logs with LLMSlim itself, so each agent reads at most about 1,800 tokens instead of the raw reports. On a local run of the full test suite: 650 digest tokens instead of 6,304 (89.7% fewer), not counting 140,797 tokens of JUnit/Cobertura XML. On GitLab pipeline 2940868316 the digest was 693 tokens out of 63,332 raw (98.9% smaller, XML included).

Challenges we ran into

  • Custom flows accept a restricted subset of the flow registry (ambient only, no top-level names, no model or response schemas), so we wrote a linter that checks those rules and the tool policy in CI before a flow is pasted.
  • Benchmark results depend on the installed optional dependencies, so a fair MR-vs-base comparison has to run both revisions in the same job and refuse to compare if Python imported the wrong copy of the library.
  • Triggered flows cannot be fired by bots, and pipeline payloads are large; we kept agent inputs small by pointing them at the digest.

Accomplishments that we're proud of

  • The library guards its own budgets with its own compression.
  • Google Cloud and PyPI use OIDC, with no stored keys. The only stored secret is the masked VERCEL_TOKEN for the free deployment target.
  • A rollback path that is deterministic CI, with the agent explaining it afterwards rather than deciding it.

What we learned

  • Custom flows are a deliberately small subset of the flow registry (ambient only, no top-level names, no model or response schemas). Checking those rules in CI, with the tool policy, caught mistakes before a flow was ever pasted into GitLab; GitLab's own validation then rejected max_cycles and two tools that chat agents can't use.
  • Triggers shape the design: flows fire on pipeline, MR, mention and work-item events but not for bots, so each flow has to be small, safe to run twice, and cheap to start.
  • What the agents read matters as much as how they are prompted. Pointing every agent at one compressed digest (693 tokens on our pipeline, instead of 63,332 raw) kept their inputs small and the same across flows.
  • SCI made us separate what we measured (job durations, tokens read, latency) from what we assumed (grid intensity, power coefficients, an LLM energy figure with a 5x range), and that split is the useful part.
  • With OIDC for PyPI and Google Cloud, the only stored secret is the masked Vercel token.

What's next

Run the budget guard on the full benchmark mode nightly, add a supervised auto-fix for formatting failures, and publish the flows to the AI Catalog.

Path B: what was added during the submission period

The original project (github.com/Thanatos9404/llmslim, MIT) had GitHub Actions tests, a manual release process and no deployed API. Everything below is new work in the GitLab project, on the branch hackathon/duo-agent-automation (commits from 13 Oct 2026 IST onward): four GitLab Duo custom flows and their triggers (MR Guardian, Pipeline Doctor, Release Captain, Issue Planner), three custom agents, a tool-permission policy enforced in CI, the GitLab CI/CD pipeline (tests, MR token-budget gate, packaging, security scanning, SBOM, licence policy, releases with trusted publishing), the LLMSlim-compressed pipeline digest for agents, an SCI estimate per pipeline, the report page and its live deployment on Vercel (deploy:vercel), and a Google Cloud Run deployment pipeline with staged revisions, a production gate, SLO monitoring and automatic rollback (ready, not deployed without a billing account). Library changes on GitHub between 5 and 12 Oct (v0.8.0, v0.8.1) are part of the imported baseline and not claimed. Details: HACKATHON.md.

Built With

  • cyclonedx
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • llmslim
  • mcp
  • pip-audit
  • python
  • react
  • semgrep
  • software-carbon-intensity
  • tailwindcss
  • vercel
  • vite
Share this project:

Updates

Submission history