Scrutent AI

Checks what AI-generated merge requests and releases claim against real evidence — the diff, the CI logs, the security scans and the live service — and only lets what is proven ship.

Comment

Inspiration

AI agents can now write a whole merge request in minutes, and each one comes with a confident description: “adds validation,” “fixes CVE-2024-…,” “fully tested.” We noticed that reviewers, ourselves included, mostly take those sentences on trust. Code review checks the code. Nobody checks the claims. As agents write more of our software, the description becomes the riskiest part of a merge request, because it’s the part a human actually reads. If generating code is solved, what’s left is trust. We wanted a system where every claim in a merge request has to come with evidence, and where an AI agent can propose a verdict but never decide it.

What it does

Scrutent AI connects to any GitLab repository. When a merge request’s pipeline finishes, it collects the evidence itself from the GitLab API: job results, every JUnit test case, coverage, the diff, and the SAST, secret detection, dependency and container scan findings. It stores that evidence where repository users can’t edit it.

A custom GitLab Duo agent flow, the Claims Auditor, reads the MR description, splits it into individual claims, and matches each one to evidence, citing the exact test case or line of code. Then Scrutent’s own deterministic code reconciles the agent’s report against the stored evidence and sets a required scrutent commit status, so nothing merges until its claims hold.

When proof is missing, a Test Writer agent commits one focused test per claim. Scrutent only accepts a claim test that fails on the base commit and passes on the branch:

$$\text{accepted}(t) \iff \neg\,\text{pass}(t,\ \text{base}) \;\land\; \text{pass}(t,\ \text{head})$$

A test that passes either way proves nothing, so it’s rejected as vacuous. When a claim is false, a Fixer agent proposes the smallest fix. It refuses to commit any change it couldn’t test.

After merge, verified claims go into a hash-chained ledger. An hourly Cloud Scheduler job re-checks them against production and opens an incident when one stops holding. All of this is governed by a .scrutent.yml file in the repository, which sets:

the autonomy level how many fix rounds agents get the severity that blocks a merge which accounts may report the paths an agent can never auto-merge

How we built it

A TypeScript monorepo with three shared packages: core: schemas, the reconciler, the security diff and the ledger gitlab: the REST client, evidence collection and audit state cli: the CI commands GitLab Duo Agent Platform: custom flows for the Claims Auditor, Test Writer, Fixer and triage, written in flow-registry YAML. Every pipeline produces evidence in a stable scrutent.evidence/v1 format, and every auditor report uses scrutent.report/v1. Dashboard: a Next.js app. Users sign in with GitLab OAuth, connect any repository they maintain, and Scrutent installs its own webhook with a secret for that connection. Tokens are sealed with AES-256-GCM, and the OAuth flow uses PKCE. Google Cloud: Cloud Run for the dashboard and a demo orders API Firestore, Secret Manager, Artifact Registry and Cloud Scheduler Workload Identity Federation, so GitLab CI deploys with no stored cloud keys Terraform to provision all of it A dogfooding pipeline: Scrutent audits its own merge requests, and every release it audits its own README. The demo video: made with Remotion, narrated with ElevenLabs, and built entirely from live screens of real merge requests.

Challenges we ran into

GitLab’s prompt security scanner. Our first flows were rejected because the prompts contained HTML comments, which we’d used as machine-readable markers. We redesigned report detection around a schema field inside a fenced JSON block, and rewrote every prompt without HTML or angle brackets. Agents in a sandbox. The first auditor runs failed before they started, because the default agent image has no Node tooling. Agents also can’t install dependencies in their sandbox. The Fixer handled this well: it prepared a fix but refused to commit code it couldn’t test. Using Scrutent on itself found real bugs. Running it on real merge requests surfaced five bugs: The orders API’s JUnit report was never kept as an artifact, so its tests were missing from the evidence. Claim tests were run with the wrong path, so every one counted as failing. Security scanners never ran in MR pipelines, so the evidence reported 97 findings as “fixed”. The reconciler downgraded a valid contradiction because the cited line wasn’t in the diff. The Deployments page said “never deployed” because GitLab’s environment list omits the last deployment.

Each fix went through Scrutent’s own audit before merging.

Multi-tenant OAuth. GitLab rotates refresh tokens, so one shared grant would break as soon as two repositories refreshed at once. Each connected repository gets its own grant, and refreshes are serialized with a lease. New tools. This was our first time deploying to GitLab and Google Cloud. Getting Workload Identity Federation, protected CI variables and Cloud Run’s background processing right (it needed --no-cpu-throttling) took a lot of trial and error.

Accomplishments that we're proud of

The full loop runs on real merge requests: On MR !5, all three claims were verified in one round, each tied to a passing test. On MR !1, the Test Writer committed four claim tests. On MR !2, a false claim (“accepts up to 200”) was marked contradicted, citing the exact line that still caps the limit at 100, and the merge was blocked. The agent never holds merge rights. It proposes, deterministic code decides, and every override is shown with its reason. Scrutent caught bugs in itself. Five real bugs were found by dogfooding and fixed through its own audit. It works for any GitLab user, not just us. It’s multi-tenant, with per-repository encrypted grants, one-click connection, and live data everywhere (no mock data). 148 tests across the packages, the API and the dashboard.

What we learned

Agents should propose, code should decide. The most reliable design was letting the LLM do what it’s good at (reading intent and finding evidence) and having deterministic code make the final call. Evidence has to be out of reach. If an agent or a contributor can edit the evidence, the verdict is worthless. Scrutent therefore collects evidence itself, and it only accepts reports from a trusted agent account. Dogfooding is the best test. Every serious bug we found came from pointing Scrutent at its own merge requests and asking whether the evidence really backed the claims. Honest “unknown” beats false confidence. Saying “not scanned” instead of “fixed”, or “unverified” instead of “verified”, is what makes the green checks believable.

What's next for Scrutent AI

Agents that can run tests. Give the Test Writer and Fixer an image with the project’s toolchain, so the Fixer can commit tested fixes and close the loop without a human. Claim tests in any language. The Test Writer already reads each repository’s own test setup. Next is making claim-test checking in CI work beyond JavaScript and TypeScript. Publish the flows. Release the Claims Auditor in the GitLab AI Catalog, so any team can enable it alongside the dashboard. Audit more than merge requests. Check claims in release notes, changelogs and issue resolutions against the same evidence and ledger. Track record per author. Show how often each author’s claims (human or agent) hold up, so teams can see whose promises are reliable.

Built With

Share this project:

Updates

Submission history