Glass Box

AI agents review changes in a GitLab pipeline, a CI gate decides how much approval each needs, and a person merges. Autonomy level: Assisted.

Comment

Glass Box puts AI agents into a GitLab delivery pipeline and wraps them in checks, so a human keeps the final say. One agent reviews each change and tries to break it. A CI gate reads that review and decides how much human approval the change needs. A second agent writes up what a person still has to decide. The autonomy level is Assisted: the agents propose, and a person decides.

Inspiration

I've contributed to open source, including CBOMkit's SonarCryptography scanning, where I helped add C/C++ and JavaScript/TypeScript support. On every team I've seen, writing the code was never the only slow part. Review, security scanning, approval, deployment and the check that production is really healthy are where the time goes. They're also where trust, more than effort, limits what you can hand to a machine.

Most agent demos show an agent writing a fix. I wanted to build the thing that stands next to it: a gate that reads what the agent says, decides how much a person still needs to check, and leaves a link for every step.

Who it's for

Small teams that want AI help reviewing and shipping code, but don't want an agent approving or merging its own work.

What it does

When a merge request is opened, I ask the Skeptic to review it. It reads the diff, tries to break the change, and posts up to five ways it could fail, ending with a RISK: low, medium or high line.

risk-gate, a CI job, reads that verdict. It only trusts a note from the Skeptic's own service account, and it prints how much human approval the change needs. Then I ask the Reporter for a status note that quotes the verdict and lists what a person still has to decide. I merge.

After the merge, the pipeline runs the tests, builds a container image and an SBOM, runs GitLab's SAST, secret detection and dependency scanning, and deploys to Google Cloud Run with Workload Identity Federation, so there is no stored key. A smoke test then checks the live service, and a schedule runs it again every hour.

Why this matters

Unit tests only catch what someone thought to test. I made eight small seeded changes to the app: five that introduce a real problem and three that are harmless. My 10 unit tests failed on three of the five unsafe changes. Two passed every test: removing the cap on stored links, and generating short codes with Math.random. Those are the changes a second reader, a person or an agent, is there to notice.

I have run the Skeptic on two of the eight so far. On the removed link cap (which passes all 10 tests), it named the problem as its first finding, unbounded memory growth, and rated the change RISK: high. On a pure rename it found no behavior change and rated it RISK: low. The other six are not run yet, so this is two data points, not a score. The changes and a script that reproduces the test results are in the repo's benchmark/ folder.

How it works

┌──────────────────────────────────────────────┐
│ 1. Merge request                             │
│ I open it and mention the Skeptic            │
└──────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────┐
│ 2. GitLab Duo Agent Platform (comments only) │
│ Skeptic: reads the diff, tries to break it,  │
│   ends with RISK: low, medium or high        │
│ Reporter: writes what a human must decide    │
└──────────────────────────────────────────────┘
                        │ verdict as an MR note
                        ▼
┌──────────────────────────────────────────────┐
│ 3. risk-gate (CI job)                        │
│ reads the Skeptic's verdict, prints DECISION │
│ fails the pipeline only on RISK: high        │
└──────────────────────────────────────────────┘
                        │
                        ▼
┌──────────────────────────────────────────────┐
│ 4. A person decides and merges               │
└──────────────────────────────────────────────┘
                        │ merge to main
                        ▼
┌──────────────────────────────────────────────┐
│ 5. Pipeline on main                          │
│ tests > image + SBOM > SAST, secrets, deps   │
│ > deploy to Cloud Run (no stored key)        │
└──────────────────────────────────────────────┘
                        │ after the deploy
                        ▼
┌──────────────────────────────────────────────┐
│ 6. smoke-prod                                │
│ checks the live service after each deploy    │
│ and every hour (schedule runs only this)     │
└──────────────────────────────────────────────┘

A hash-chained log (evidence/log.jsonl) records notable events. A script writes it, and it isn't part of the live flow.

Who does what

Who or what What it does What it can't do
Skeptic (my Duo custom flow) Reads the diff and posts a review ending in a RISK: line Merge or edit anything
Reporter (my Duo custom flow) Posts a status note quoting the verdict and listing what a person must decide Merge or edit anything
risk-gate (CI job) Reads the Skeptic's verdict and prints a DECISION; fails only on RISK: high Judge the code itself
Planner Agent (GitLab's own) Planned issue #4 and, after I approved, created child tasks #5 to #8 It isn't my code
Fix CI/CD Pipeline flow (GitLab's own) Diagnosed a failed job and suggested a retry It changed no code
Me Open the merge request, mention the agents, approve, merge

Why Assisted

In my tests a person mentioned the agent to start it, and a person merged. The agents propose and report, and I approve each change. I haven't shown that a schedule or a bot can start a flow without a human, so I'm not claiming a higher autonomy level.

How I run a review: I open a merge request, comment @ai-skeptic-gitlab-ai-hackathon please review, retry risk-gate if it ran before the review arrived, comment @ai-reporter-gitlab-ai-hackathon please report, then decide and merge.

The nine stages

Stage Where
Plan GitLab's built-in Planner Agent planned issue #4 and created child tasks #5 to #8
Create app/, flows/, merge requests (for example MR !6)
Verify unit-tests, infra-validate, the Skeptic
Package package-image, generate-sbom
Secure sast, secret_detection, dependency_scanning
Release deploy-cloudrun, rollback-cloudrun
Configure infra/*.tf, Workload Identity auth in .gitlab-ci.yml
Monitor smoke-prod and the hourly schedule
Govern risk-gate, flows/reporter.yaml, policy/autonomy.yml, evidence/log.jsonl

Verify it in 5 minutes

No login is needed for the live service. Each row is one claim and one link.

Claim What you'll see Link
The service is live {"status":"ok"} /health
The Skeptic found a real bug HEAD requests counted as visits; fixed in 6c0821b MR !6
The injection test A hostile comment, the Skeptic's refusal, RISK: medium MR !14
The Skeptic on a seeded change The removed link cap, named as unbounded memory growth, RISK: high MR !24
The benchmark Eight seeded changes; my tests miss two of the five unsafe ones benchmark/
Keyless deploy works deploy-cloudrun passed pipeline 2950729078
Post-deploy check works SMOKE PASS job 17199574841
Infra check works infra-validate passed MR !13
Planning happened Child tasks #5 to #8 issue #4
Auto-diagnosis A timeout called a network failure, retry suggested session 9530923

How we built it

The app is a small Node.js URL shortener with no dependencies. It's the stage for the automation, not the point. It has unit tests, a non-root Docker image, and a .gitlab-ci.yml for the pipeline above. risk-gate is a short Python script with its own tests. OpenTofu creates the Cloud Run service (capped at one instance), an Artifact Registry repo, two service accounts and the Workload Identity setup.

I used Claude Code to write much of the code, the scripts and the CI config, and other AI tools (Codex and DeepSeek) while brainstorming and planning. The GitLab Duo agents do the review and the reporting inside the pipeline. I did the GitLab and Google Cloud setup and every approval myself, and I checked each claim on GitLab before writing it into STATUS.md.

Design decisions

Decision Why
The agents can't merge or edit Both flows only post a comment. I wanted the approval to stay with a person, so they never got a merge tool
The gate trusts one author risk-gate only accepts a verdict from the Skeptic's own service account, so a copied "RISK: low" from anyone else is ignored
The gate fails only on RISK: high A low or medium verdict prints a decision but doesn't block the pipeline. I wanted the gate to guide a person, not replace one
Keyless deploys The Workload Identity provider only accepts tokens from this project's main branch. The deploy account has developer rights on this one Cloud Run service, write access to one Artifact Registry repo, and permission to act as the runtime account
One instance, request-based CPU State lives in memory, so the service is capped at one instance. 256 MiB of memory needs request-based CPU on Cloud Run
The schedule runs only the smoke test Scheduled pipelines switch the other jobs off, so the hourly check stays cheap

Built with

Layer Tools
Agents GitLab Duo Agent Platform: two custom flows, plus the built-in Planner and Fix CI/CD agents
Pipeline GitLab CI/CD, SAST, secret detection, dependency scanning, an SBOM
Cloud Google Cloud Run, Artifact Registry, Workload Identity Federation
Infrastructure OpenTofu
App and scripts Node.js, Python, shell
Coding help Claude Code

Challenges we ran into

Learning GitLab itself

GitLab itself was the hardest part at first. I hadn't worked with merge request pipelines, protected branches, custom flows or Duo's triggers before. The docs for the general flow framework and for custom flows disagreed, and my first YAML was rejected for fields the general docs call optional. I mixed up a branch pipeline with a merge request pipeline: on MR !13 the branch pipeline failed on a runner timeout while the merge request pipeline passed, and it took me a while to see they were two different things. I also didn't expect the gate to run before the Skeptic had commented.

What went wrong

What happened What I did
risk-gate ran before the Skeptic had commented on MR !14 and printed DECISION: no Skeptic review yet. It doesn't run again when a review arrives Retried the job by hand
I started rollback-cloudrun and cancelled it after 48 seconds. It had already moved traffic back one revision Fixed it with gcloud run services update-traffic. I never saw the job finish
A Reporter session ended as failed (dropped) A retry worked. Nobody knows why it dropped
My first infra config was rejected: "Total memory < 512 Mi is not supported with cpu always allocated (unthrottled)" Switched the service to request-based CPU
On Oct 22 I merged a deliberately unsafe route (eval on a query parameter) into main, trying to trigger a RISK: high verdict. It deployed as revision 00010 and ran about 30 minutes Reverted it. The logs show three requests to it, all with the harmless value expr=1, from one device. I should have used a throwaway branch

I also haven't shown that a schedule or a bot can start a flow. In my tests a human mention started both agents.

Limits

Limit Detail
The gate trusts the Skeptic risk-gate follows the Skeptic's verdict. A prompt injection that changed that verdict would change the gate's decision, so the person who merges is the backstop
Deploys move all traffic at once deploy-cloudrun sends 100% of traffic to the new revision and then runs the health check. If the check fails, the new revision is already serving and rollback is manual
The link store can be filled POST /shorten is public with no rate limit, so a client can fill the 10,000-link store, and it stays full until a restart
In-memory state Links and visit counts are lost on a restart, so the service is capped at one instance
Public stats GET /stats/:code has no login, and bot visits are counted (HEAD is not)
The SBOM job is not pinned generate-sbom uses aquasec/trivy:latest, so the result can change between runs
The Skeptic can read widely Its only write is one comment, but its read tool can reach whatever its service account can read
Infrastructure state is local OpenTofu state lives on one machine, not in a shared backend
The evidence log It is backfilled and not tamper-proof: someone who can rewrite the whole file can recompute the chain, and cutting lines off the end goes unnoticed
The injection test One case against two agents. It's not proof they resist injection
Scope I've only run this on one app

Accomplishments that we're proud of

  • A real bug caught before merge. On MR !6 the Skeptic found that HEAD requests were being counted as visits. I fixed it, and on production a HEAD request now leaves the count at zero while a normal request raises it to one.
  • A reproducible gap in my own tests. Two of five seeded unsafe changes pass all 10 unit tests, and a script in benchmark/ reproduces that. On one of them, the removed link cap, the Skeptic named the problem and rated it RISK: high.
  • An injection test that held up once. I added a hostile comment aimed at AI reviewers to MR !14. Neither agent acted on it, and the Skeptic returned RISK: medium. After a retry, risk-gate printed DECISION: human approval required (RISK: medium).
  • Keyless deploys. The pipeline deploys to Cloud Run from GitLab with no stored Google credential.
  • Every step leaves a link. Almost every claim in this write-up points at the merge request, pipeline or job that proves it.
  • Honest limits. The project says what it can't do, and what went wrong, in the same place it says what worked.

What we learned

About the agents

  • A RISK: low verdict doesn't mean there are no bugs. The Skeptic rated MR !6 low while it was catching a real one.
  • Agent flows can fail with no clear reason, so a retry plan is part of the design.
  • Check the schema before writing a flow. GitLab's custom-flow rules rejected fields that the general framework docs list.

About GitLab and the Duo Agent Platform

  • Once I understood the merge flow, I came away impressed by how much sits in one place. A merge request, its pipeline, the agents' comments, the jobs and the deploy all link to each other, so a claim can point at the exact thing that proves it. I didn't have to wire that together myself.
  • Duo flows run as their own service accounts, and every run is a visible session linked to a CI job. That made it possible to build a gate that only trusts one author, and to keep an audit trail without extra tooling.
  • Triggering an agent by mentioning it on a merge request fits the way review already works. It also kept a person in the loop, which is why this is an Assisted project.
  • GitLab's built-in agents are useful on their own. The Planner Agent asked for my approval before it created tasks, and the Fix CI/CD Pipeline flow diagnosed a failed job and recommended a retry instead of changing code.
  • Keyless deploys with GitLab's identity tokens and Workload Identity Federation meant I never stored a Google credential. That was worth the setup time.

What's next for Glass Box

These are not done yet.

  • Run the Skeptic on the other six seeded changes, and report the full result.
  • Turn the two seeded changes my tests missed into unit tests.
  • Deploy new revisions with no traffic until the health check passes.
  • Wire the evidence log into CI so it isn't appended by hand.
  • Confirm the hourly schedule runs with nobody touching it.
  • Make risk-gate run again when a late review arrives.
  • Try the loop on a second project.

Links

Built With

Share this project:

Updates

Submission history