Honest Reviewer

A GitLab Duo agent that checks what a merge request says about its testing — "all tests pass", "no vulnerabilities" — against the pipeline's evidence, and blocks the merge when the claim isn't backed.

Comment

Inspiration

"All tests pass." Reviewers read it in merge request descriptions and commit messages every day and mostly take it on trust. Coding agents now write a growing share of those sentences, and they have every reason to sound done. But a pipeline can be green while tests fail behind | tail, while every test is skipped, while only one file of the suite ran, or while the test job never ran at all. Before building this I measured the same failure in models: a benchmark of 48 coding-session logs where the model must report what was actually verified. The gap between "says verified" and "was verified" is real, and it belongs in the review, before the merge.

What it does

On every merge request pipeline, Honest Reviewer:

  1. Collects evidence (honesty-evidence CI job): every JUnit report of the pipeline, grouped into the test suites the repository defines, plus GitLab SAST's report — written to evidence.json.
  2. Checks the claims (honesty-check CI job): finds claims about verification in the MR title, description and commit messages — "tests pass", "verified", "full suite", "safe to merge", "no vulnerabilities" — and tells them apart from statements that only look like claims ("not tested yet", "should pass once…", questions, WIP, unticked checklist items, quoted examples). Each claim is checked against the suites the merge request actually changes and gets a fact with the numbers behind it: supported, contradicted ("2 of 12 demo tests failed"), unsupported ("every demo test was skipped", "no test report for demo"), stale, or not checked. A partial run is caught against main's latest evidence ("5 demo tests ran here vs 12 on main"), a stated count is checked as written ("All 13 tests pass" with one skipped), and a security claim is checked against SAST ("semgrep found 1 vulnerability (1 high): eslint.detect-eval-with-expression in demo/src/promo.ts:35").
  3. Delivers the verdict (GitLab Duo Agent Platform flow, pipeline-event trigger): reads the pipeline, fetches review.md and verdict.json, skips stale or already-reviewed commits, and posts the review verbatim — a plain note when the merge request is honest; submit_mr_review with requested changes when it is not, which blocks the merge ("Merge blocked: 1 check failed"); and a follow-up issue assigned to the author.

The model never writes its own assessment: the verdict comes from deterministic code in CI that anyone can read and rerun. The agent's job is the part that needs judgment about the workflow — finding the right pipeline, job and merge request, checking freshness, not repeating itself, and acting in GitLab.

How we built it

  • TypeScript on Node 24, no build step and no runtime dependencies; 252 tests (node --test), run in CI with JUnit.
  • .gitlab-ci.yml: unit and demo test jobs → GitLab SAST (semgrep, also in MR pipelines) → honesty-evidence → honesty-check (MR pipelines) → package-image (main).
  • Suites in .honest-reviewer.json; main's evidence fetched as the baseline with CI_JOB_TOKEN.
  • Duo flow flows/honest-reviewer.yml (flow registry v1, ambient) with tools gitlab_api_get, get_job_logs, get_merge_request, list_all_merge_request_notes, create_merge_request_note, submit_mr_review, list_issues, create_issue; enabled with a pipeline-events trigger (Passed, Failed).
  • Reusable as a CI/CD component (templates/honest-reviewer.yml, tag v0.1.0); the checker image is published to the project's Container Registry.

DevSecOps stages covered (8 of 9)

Plan (follow-up issue) · Create (review on the MR) · Verify (CI evidence and claim check) · Package (image in the Container Registry) · Secure (security claims vs GitLab SAST) · Release (requested changes block the merge) · Configure (CI/CD component, suite config) · Govern (evidence, verdict and review kept 90 days, exposed in the MR). Monitor is not covered.

Challenges we ran into

  • The reviewer caught itself first. On its own merge requests it read a commit message that listed claim names in parentheses, and later a number next to a quoted example, as claims. We added a "mentioned, not claimed" rule; its next review of itself was honest.
  • Totals hide traps. A demo app's "every test skipped" disappeared inside 200+ passing tests of the reviewer itself, so claims are now checked per suite, against the suites the merge request changes.
  • Flow configuration. An HTML comment in the prompt is rejected as unsafe, and an early save silently kept the template; reading the session's runner log showed what really ran.
  • Its own SAST findings. GitLab SAST reported 10 medium regex findings in the reviewer. Three were fixed by rewriting, untrusted merge request text is now capped with hostile-input tests, and the remaining constant patterns are marked as reviewed with the reason next to each — 0 findings now.

Accomplishments that we're proud of

  • Seven demo merge requests, each a different trap, each answered by the flow with the right verdict; a green pipeline whose merge is blocked because its description wasn't true.
  • Every number in a review can be traced to a JUnit or SAST report in the same pipeline.

What we learned

Honest review is mostly bookkeeping: which commit, which suite, which report. Once that is deterministic, an agent can be trusted with the rest of the workflow.

What's next

Lint and build evidence (so "typecheck passes" can be checked too), a dashboard of claim accuracy per project and per author (the Monitor stage), and checking claims in agent-written merge requests before a human is asked to review.

Try it

include:
  - component: gitlab.com/gitlab-ai-hackathon/transcend-october-2026/11742884/showcase/[email protected]

Built With

Share this project:

Updates

Submission history