GHOST USERS: tomorrow's users test every merge request

CI is green, yet carton orders vanish from stock. GHOST USERS replays synthetic customer journeys on every MR, shrinks the failure to one reproducer, and only accepts a Duo fix it can verify.

Comment

Inspiration

Most production incidents in order and inventory systems are not crashes. They are correct-looking code doing the wrong business arithmetic for a customer segment nobody unit-tested: wholesale buyers ordering in cartons, discounts that stack in the wrong order, tax on a zero-value line. Ordinary CI stays green. We wanted every merge request to meet "tomorrow's users" before real ones do, and we wanted an AI agent to be able to fix what they find without anyone having to trust the agent's own claim that it worked.

What it does

  1. Replays synthetic users on every MR. A GitLab pipeline starts a job-scoped preview of the app and replays a deterministic cohort of 842 synthetic customer journeys over real HTTP. An independent oracle checks committed state (inventory, net, tax, order state), not the API's own answers.
  2. Fails the MR on business regressions that ordinary CI misses. In the demo seed, unit tests, contract tests and the security policy all pass; only preview-replay fails: 819 passed / 23 failed / 0 errors on inventory_delta.
  3. Minimizes the failure. The failing journey (5 cartons × 24 pieces: stock moved by −5 instead of −120) is reduced by actual replay to one reproducer: 1 carton × 2 pieces, inventory 2, expected delta −2.
  4. Guards the handoff. assess-failure decides whether a failure may go to an agent: allowed target branch, business-only failure, pinned trusted revision, one attempt per pipeline. It returns READY, SKIPPED (for example STALE_PIPELINE once the MR head has moved) or ERROR, so the same failure is never handed out twice.
  5. Lets GitLab Duo repair it. The GitLab Duo Developer flow receives the evidence and an edit boundary (ghost_users/application.py only) and pushes a normal, non-forced commit to the MR branch.
  6. Verifies the repair independently. verify-repair reads the public GitLab pipelines, jobs and artifacts, checks the patch is a direct-parent, application-only change by the Duo actor, that all five jobs passed, and that the byte-identical 842-journey cohort now passes. Only then does it emit VERIFIED, with a hash-bound certificate and anchor. A model's own "I fixed it" never counts.

How GitLab Duo is used

GitLab Duo is the repair agent in the loop. In two separate loops on draft MRs !6 and !7 the Duo Developer flow read the failing job's artifacts, diagnosed the carton-to-piece conversion, and pushed a one-line fix under the duo-developer-gitlab-ai-hackathon service account (commits 95ad783 and 2a9b816). The MR pipelines that followed ran as the Duo actor and passed 5/5; verify-repair then returned VERIFIED for both. Duo's own reply on the MR is candid about what it could not check in its sandbox; GHOST USERS is the part that does not have to take that reply on trust.

Autonomy level

Claimed: Supervised. Target: Hands-off.

Today each loop starts with one human @duo-developer-gitlab-ai-hackathon mention on the MR; everything after the mention (diagnosis, patch, push, CI, independent verification) runs without a human touching code. An automatic failed-pipeline → guard → Duo handoff is designed and pushed for review (draft MR !8, docs/duo-pipeline-handoff.md) but is not activated; activating a trigger needs explicit approval.

How we built it

  • Python 3.13, standard library only: no packages, no cloud account, no credentials needed to run it locally.
  • SQLite-backed preview app; HTTP replay with redirects and ambient proxies disabled.
  • Independent oracle and fingerprinting of failures by invariant and unit family.
  • Delta-debugging style minimizer (optional-action deletion plus bounded integer reduction) that only accepts reductions confirmed by real replay.
  • Hash-chained evidence ledger (ledger.jsonl + head) and sanitized artifacts; raw payloads and credentials are rejected.
  • assess-failure guard and verify-repair verifier that read public GitLab evidence only, fail closed and pin a reviewed baseline independently of the triggering event.
  • GitLab CI with five jobs: conventional, foundation-contracts, python-policy (AST security policy), preview-replay, foundation-evidence.
  • 174 unit/contract tests.

Results

Measure Result
Duo repair loops with VERIFIED 2 / 2 (MR !6 loop 01, MR !7 loop 02)
Seeded regressions detected by replay 12 / 12, each on its catalogued invariant
Same regressions caught by the ordinary retail smoke test 0 / 12
False positives on the reference app 0 / 12
Headline cohort 842 journeys, SHA-256 09ff6213…5b800, identical before and after repair
Loop 02 timeline failure → Duo fix → passing pipeline → VERIFIED, about 35 minutes

Benchmark details: docs/benchmark-20261022.md (hosted repeat: pipeline 2976208593).

Challenges

  • Not trusting the agent. The verifier had to be independent of Duo, of the MR author and of the triggering event: pinned trusted revision, exact cohort bytes, direct-parent and file-scope checks, actor checks, and fail-closed transport.
  • Stale and duplicate handoffs. A failure is only actionable for the MR head that produced it; the guard records one claim per pipeline and refuses stale ones.
  • Platform reality. Duo's sandbox ran Python 3.9 while CI targets 3.13, so its local unit-test run was unreliable. CI and verify-repair are the arbiters, not the sandbox.
  • Honest scope under a deadline. Native trigger activation needs approval, so we report Supervised rather than claim Hands-off.

Accomplishments we're proud of

  • Two real Duo repairs that an independent verifier, not a human reviewer, accepted.
  • A failure that every ordinary check missed, reduced from 842 journeys to a single two-piece reproducer an agent could act on.
  • Everything is reproducible from public GitLab evidence with one command.

What we learned

The interesting part of an AI repair loop is not the patch; it is the contract around it: what the agent may touch, what evidence it gets, and what has to be true before anyone believes it.

What's next

  • Activate the guarded failed-pipeline trigger (Hands-off), with a negative control.
  • More distinct defects and repair/verification rates across all 12 cases.
  • Public review apps instead of job-scoped previews; cohorts generated from real, privacy-filtered traffic shapes instead of fixtures.

Try it out

From a checkout of codex/benchmark-12 with Python 3.13+ (no installs):

python -W error::ResourceWarning -m unittest discover -v
python -m ghost_users replay --regression unit_conversion --output artifacts/failing-candidate
python -m ghost_users replay --regression none --output artifacts/reference
python -m ghost_users evidence --output artifacts/my-foundation-run
python -m ghost_users verify-repair --before-pipeline 2976430799 --after-pipeline 2976470578 --trusted-revision 4d36b50913b2a8c23140a4c70e182259f47b5ed1 --output artifacts/verify-loop02

The first replay exits 1 (819/23/0) and writes minimization.json; the reference exits 0. verify-repair needs internet access to gitlab.com and prints VERIFIED. Each --output must be a new directory.

Evidence links

Known limits

  • All journeys are deterministic synthetic fixture data, not customer telemetry.
  • Both VERIFIED loops use the same carton seed: this shows repeatability, not breadth. Breadth comes from the 12-case detection benchmark, where repair rates are not yet measured.
  • Each loop was started by one human mention (Supervised); automatic handoff is pending.
  • The preview lives only for its CI job; there is no public Review App.
  • The ledger is hash-chained but not signed; its anchor must be kept outside the evidence.

Built With

  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • python
  • sqlite
Share this project:

Updates

Submission history