Bifocal

Two QA agents, one blind to the code, every bug machine-reproduced before a human sees it.

Comment

Inspiration

Coding agents made changes cheap. Checking them didn't get cheaper. Small teams and maintainers of self-hosted apps have no QA team. An AI agent told to "QA this" produces confident bug reports that nobody can quickly confirm or dismiss. We wanted agentic QA a maintainer could trust: every claim checkable, every fix proven, and a human deciding what matters.

What it does

Bifocal tests every merge request twice, on a fork of linkding, a self-hosted bookmark manager:

  1. The pipeline builds the change and deploys two apps on Cloud Run: the MR head and its merge base.
  2. A QA Lead agent reads the diff and the docs, picks a risk tier (skip, light, standard, deep) and writes test charters.
  3. A white-box agent reads the code and writes failing tests. In parallel, a black-box agent gets only the user documentation and the running app. It has no checkout and no tools except one note. It writes scenarios round by round, and our runner executes them in CI and reports back.
  4. Deterministic verification, with no model involved. Every test is re-run and every scenario replayed on both apps. Findings that don't reproduce are dropped. The rest are labelled introduced or pre-existing.
  5. A report agent dedupes and ranks. Each verified finding becomes an issue, and the MR gets one report note.
  6. The merge gate stays shut while introduced findings are untriaged. A human labels each one fix, won't fix, not a bug or duplicate.
  7. Mention the implementer agent and it pushes a fix. QA replays the original reproduction, not the implementer's own tests. QA-written tests can't be edited or skipped. Then the gate opens and the human merges.
  8. After merge: staging, then a regression replay, then a no-traffic production canary replayed before promotion, with automatic rollback, then a GitLab Release listing the QA-verified fixes. Uptime alerts become GitLab incidents, and the scenarios replay against production every night.

Path B, what we added to linkding: upstream linkding has unit and e2e tests on GitHub Actions, and nothing after that. During the hackathon we added:

  • agentic white-box and black-box QA with machine verification and human triage;
  • an implementer agent with re-verification;
  • security scanning and lint;
  • per-MR review and baseline apps, staging, and production on Cloud Run with canary and rollback;
  • QA-verified release notes, uptime incidents and nightly synthetic monitoring;
  • all infrastructure as code.

How we built it

  • GitLab Duo Agent Platform:
    • five custom flows, published to the AI Catalog from main by CI;
    • started through the Flows API and polled through GraphQL;
    • a Mention trigger for the implementer;
    • a custom agent-config.yml image;
    • pinned tools per flow.
  • GitLab CI/CD:
    • nearly 40 jobs across build, test, secure (SAST, secrets, dependency and container scanning), review apps, QA, staging, production, and a nightly schedule;
    • GitLab-managed OpenTofu state;
    • Environments, Releases and incidents.
  • Google Cloud (europe-west9):
    • Cloud Run for the review, baseline, staging and production apps and for the alert relay, all scaling to zero;
    • Cloud SQL, Artifact Registry, Secret Manager and Cloud Monitoring;
    • keyless CI authentication through Workload Identity Federation;
    • all of it OpenTofu in infra/.

Platform constraints shaped the design:

  • Flows can't run in parallel or start each other, so a CI orchestrator starts them.
  • run_command is a free-form shell, so the black-box agent gets no shell at all. Instead it writes scenarios that CI executes.
  • Flow jobs can't read CI variables, so every secret stays in CI jobs.

Challenges we ran into

  • Black-box isolation on a public repo. Any read tool can reach the code. Our answer is an agent with no tools except one note, plus a check in every run that reads its catalog definition back.
  • Making agent output verifiable. We turned white-box claims into tests and black-box claims into scenarios, both replayable without a model.
  • Seeing a bug isn't reporting it. On two seeds, the black-box agent's scenarios failed on the defect, but it declared itself done without reporting them. Now every failure nobody has claimed gets a final findings round.
  • Measuring honestly. We pre-registered answer keys by hash and kept baselines on mirror MRs, so no config could read another's output. We also published our misses.

Accomplishments that we're proud of

We benchmarked Bifocal on 16 merge requests (12 seeded defects and 4 clean controls, in three rounds; rounds 2 and 3 were pre-registered by SHA-256), against GitLab's Code Review Flow and a single all-access agent:

  • No false claims from any config on the clean controls.
  • Seeds caught: Code Review Flow 11, single agent 12, Bifocal 10. Bifocal's two misses were defects outside the diff. We fixed the planning step, and it caught all three fresh seeds in round 3.
  • Real bugs nobody planted: Bifocal reported them in 5 MRs, the single agent in 2, Code Review Flow in none. Each came with a failing test or a replayable scenario.

Bifocal also found real pre-existing linkding bugs. One of them, an HTTP 500 from a non-numeric or anonymous ?bundle= parameter, went through the full loop and is fixed on main. Bifocal's QA of that fix then caught a regression the fix itself introduced, which was fixed and re-verified before merge.

Sustainability, measured with the Green Software Foundation's SCI: 25.8 gCO₂e per MR (range 7.5–109.6) for Bifocal, against 2.4 for Code Review Flow and 3.0 for a single agent. We report that trade-off openly.

  • Risk tiers cut light and skip runs by 60–90%.
  • Verification, staging, canary and the nightly monitor replay saved scenarios with no LLM tokens.
  • The review apps scale to zero in a 16 g/kWh region, so Cloud Run is under 0.03 g per run.

What we learned

  • Verification beats trust. Once every claim must reproduce on a machine, false positives disappear, and the human's job becomes a decision, not an investigation.
  • A code review is the right tool for small diffs. It catches seeded bugs as well, for a tenth of the carbon. Bifocal earns its cost elsewhere: in evidence, in bugs nobody planted, and in scenarios that keep guarding staging, canary and production.
  • Separating what an agent can see (code vs. docs-only) gives genuinely different findings, not the same review twice.

What's next for Bifocal

  • Repeated runs to report variance.
  • A security probe pack for deep runs.
  • Packaging the CI and flows as a CI/CD component, so another project only needs a qa/config.yml.

Try it: https://linkding-prod-549827948874.europe-west9.run.app (login demo / bifocal-judge-demo) · Repo: https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/2676619/bifocal · Method and numbers: bench/RESULTS.md, qa/sci/SCI.md

Built With

Share this project:

Updates

Submission history