MutantGate (TranscendGate)

Diff-targeted AST mutation testing, dual-execution test synthesis, Cloud Run canary gating, and SCI carbon accounting for the GitLab Duo Agent Platform.

Comment

Inspiration

AI coding tools have made it effortless to get a Merge Request up to 100% line coverage in seconds. The problem is that line coverage only tells you a line of code ran during a test. It does not tell you whether any assertion actually checked the boundary logic or security guards on that line. You can delete an authentication check or flip >= to >, and a surprisingly high number of green CI pipelines will still pass.

While digging through the gitlab-org/gitlab issue tracker (#263424 on mutation testing in specs and #236248 on diff-scoped CI scanning), we noticed why teams rarely run mutation testing in CI: mutating an entire repository takes forever and burns through GitLab Runner compute minutes. We wanted to see what happens if a GitLab Duo agent only mutates the exact lines touched in a Merge Request diff, writes the missing tests itself, and refuses to propose any test unless it can prove the test catches the bug.

What it does

MutantGate is a GitLab Duo Agent and Model Context Protocol (MCP) server that audits Merge Request diffs, hardens weak test suites, and gates Cloud Run deployments across seven GitLab DevSecOps stages:

  • Diff-Scoped Mutation Testing: Instead of mutating a whole codebase, it parses the unified git diff of a Merge Request and injects five CWE-mapped faults (SEC_BYPASS, ROR, LCR, AOR, and SBR) strictly on the modified lines.
  • Dual-Execution Test Synthesis: When a mutant survives the existing tests, our counter-example solver searches for a boundary input where the original function and the mutated function disagree. Before suggesting a test, it runs the test twice in isolated child processes: the test must fail against the mutated code and pass against the original code.
  • Seven-Stage GitLab Artifact Generation: A single run produces confidential GitLab Issue payloads (with optional live creation via the GitLab Issues REST API), MR Suggested Change blocks, a native JUnit XML report (gl-mutation-junit.xml), a GitLab SAST v15.0.0 vulnerability report (gl-sast-report.json), a Cloud Run canary release gate, live HTTP canary health checks with automatic gcloud traffic rollback commands, and a SHA-256 governance attestation.
  • Green SCI Carbon Accounting: It calculates how many CI runner minutes and grams of CO2 (ISO/IEC 21031:2024 Software Carbon Intensity) are saved by scoping mutation analysis to the MR diff instead of the full repository (cutting 270 baseline mutants down to 9 on a typical MR, a 96.67% reduction).

How we built it

We wrote MutantGate in pure Node.js 20+ ESM with zero external runtime dependencies, relying on built-in modules (node:test, node:child_process, node:vm, node:crypto, and node:http).

To integrate natively with the GitLab Duo Agent Platform, we configured .gitlab/duo/agent-config.yml for runner setup, .gitlab/duo/mcp.json for automatic MCP tool discovery, .gitlab/duo/mr-review-instructions.yaml and AGENTS.md for code review rules, and .gitlab/duo/flows/transcend-pipeline.yml using the Flow Registry v1 schema (environment: "ambient", chaining seven AgentComponent stages). The MCP server (src/mcp-server.mjs) exposes six tools over both stdio and Streamable HTTP (POST /mcp), and the web workbench runs on Google Cloud Run alongside a multi-stage .gitlab-ci.yml pipeline.

Challenges we ran into

Our first version looked great on our demo file and immediately fell apart as soon as we pointed it at arbitrary modules and real GitLab CI runners.

First, a naive regex mutator will happily mutate >= inside a code comment or a log string, producing useless mutants. We had to build a character-level masker (maskCommentsAndStrings) that blanks out single-line comments, block comments, and string literals while preserving exact column offsets before applying any operator.

Second, our initial counter-example test synthesizer assumed target functions were synchronous. When we tested it on export async function modules, the runner compared unresolved Promises instead of return values and couldn't generate await assert.rejects(...) assertions. We had to rework the child-process harness to await async exports and handle rejected promises cleanly.

Third, when our GitLab CI pipeline ran inside a minimal node:22-alpine container, --auto-diff crashed with spawnSync git ENOENT because Alpine doesn't ship with the git binary installed by default. We added a filesystem fallback so the CLI can still locate and audit modified source files in stripped-down CI containers. We also caught a bug where our default demo canary error metrics were leaking into custom file runs, causing healthy external modules to trigger false Cloud Run rollbacks until we separated live HTTP canary probing (--canary-url) from demo telemetry.

Accomplishments that we're proud of

  • Enforcing a strict Dual-Execution Proof gate so the agent cannot hallucinate a passing assert.ok(true) test just to make coverage numbers look good.
  • Building an automated 8-phase "Break and Restore" verification script (npm run verify:break) that intentionally injects bugs into each of our own core modules, verifies that our test suite fails, restores the exact file bytes by SHA-256 hash, and confirms all 30 tests pass again.
  • Getting the full loop running end to end on a real GitLab repository and Merge Request, emitting native JUnit and SAST v15.0.0 artifacts alongside a working stdio and HTTP MCP server.

What we learned

We learned that using an agent to write tests only works if you treat the agent's output as untrusted until proven otherwise by an execution harness. Combining deterministic boundary search (CEGIS) for fast counter-example discovery with an LLM fallback for complex domain inputs gave us repeatable results without burning tokens on simple off-by-one bugs. We also learned how much cleaner GitLab CI integrations become when your tool outputs standard schemas like JUnit XML and GitLab SAST v15.0.0 directly, since GitLab's UI already knows how to render them in Merge Requests.

What's next for MutantGate

Right now our lexical and token mutator targets JavaScript and TypeScript modules. Next, we want to add diff-scoped mutation adapters for Python (pytest) and Go (go test) behind the same MCP tools, and package our .gitlab-ci.yml jobs as a reusable GitLab CI/CD Component so any repository can include it in three lines of YAML without copying scripts.

Built With

Share this project:

Updates

Submission history