Fixed For You

A customer reports a bug. The fix reaches them first — verified on their own requests in production — and only then everyone else. Built on GitLab Duo, running on Google Cloud.

Comment

Inspiration

Every issue-to-production pipeline stops at "deployed". The person who reported the bug is still waiting, and nobody has checked that their requests work now — the ticket was closed on a merge. We wanted a loop that closes on the reporter's own traffic, and tells them.

What it does

Fixed For You runs Kutt (an open-source URL shortener) on Google Cloud, with a ring around it on GitLab. The customer's report is the only human act. From there:

  1. Plan — the customer writes on the app's Contact support page. A deterministic conductor files a public GitLab ticket in their words, adds them to it by email, finds their account and posts their failing production requests from GitLab Observability. A GitLab Duo intake flow compares evidence and report: reproduced → fix; nothing in production → it asks the customer one question and writes no code.
  2. Create — a Duo fix flow writes a regression test that replays the customer's request and a fix behind the feature flag ring-<issue>, and opens the merge request.
  3. Verify / Secure — unit tests; a negative control (the test must fail on the merge base, pass with the flag on, fail with it off); a charter check (agent changes only in server/ and test/ring/); SAST, secret detection, dependency and IaC scanning, and a security gate that stops any new high or critical finding — all on an unprivileged runner. If GitLab Duo Code Review asks for changes, the merge request never merges: the conductor closes it and the next attempt starts from the reviewer's comments. Otherwise it merges itself.
  4. Package / Release — one image per commit, container-scanned, SBOM, keyless signature and attestation; staging, smoke test, production on the same digest; a release.
  5. Configure — the flag goes on for one user list: the reporter (plus the ring's probe account). A stop drill switches it off and on in production and checks that the old behaviour comes back and goes away.
  6. Monitor — the conductor counts the reporter's own requests on the flagged path, from HMAC-signed spans. Enough succeed → verified. One fails → flag off, incident, next attempt.
  7. Govern — a Duo reply flow drafts the answer; a gate refuses trace ids, internal hosts, secrets, other customers and links. The customer gets it by email. Then 25% → 100% while the fleet stays healthy, a cleanup merge request removes the flag and the old code path, and the ticket closes with its record: attempts, CI minutes, credits, grams of CO2e for this one resolved report.

Done is not "merged". Done is when it works for the person who asked.

The live runs (25 Oct 2026, all public)

  • Ticket #3, Marco — his newsletter tool's API calls succeeded but returned no short link. From his report to the fix verified on 7 of his own requests: 80 minutes; told 2 minutes later; everyone after; flag removed in !6. Nobody touched it. 13.76 credits, 39 CI minutes, ~31 g CO2e.
  • Ticket #4, Dana — production showed no failing request for her, so the ring asked her one question and wrote no code.
  • Ticket #5, Aiko — a bug still open upstream (Kutt #975: a taken custom name answers 500). Duo Code Review asked for changes on the first fix (!7); the conductor closed it and attempt 2 (!8) started from the review, passed every gate, met the real 500 in the stop drill with the flag off, and was verified on 6 of her requests. Told, rolled out, flag removed.

Every ring's full trail — including the internal steps — is on the ring ledger.

How we built it

  • Conductor (ring/, TypeScript on Node 24, 235 tests): a pure state machine plus effects behind seams. It runs every two minutes as a Cloud Run job under a Cloud Storage lock, and hourly from a GitLab pipeline schedule. It is the only thing that touches production or the customer.
  • Agents: three GitLab Duo custom flows (intake, fix, reply), published to the AI Catalog from the repo, with their skills in skills/. Their notes are pinned to internal; they hold no production credentials.
  • App: Kutt v3.1.0 + OpenTelemetry with the end user on every span and signed verification fields, a feature-flag client for GitLab's Unleash API, and the Contact support page.
  • Google Cloud (deploy/terraform/): Cloud Run (app, conductor, synthetic traffic), Cloud Scheduler, Artifact Registry, Secret Manager, Cloud Storage (SQLite replicated by Litestream), two CI runners on Compute Engine, Workload Identity Federation — no cloud keys anywhere.
  • Synthetic customers: twenty people who do not exist use the app every five minutes; their client checks are signed spans. A person writes their reports — that is the one human act.

Challenges we ran into

  • GitLab Service Desk's incoming email never created a ticket for us (two projects, several senders, no bounce), while its outgoing email worked — so customers write on a support page, the conductor files the ticket and adds them with /add_email, and replies still reach their inbox.
  • Google's front end marks most Cloud Run requests not-sampled in its traceparent, so the default sampler dropped ~90% of our spans; the ring could not see the reporter's requests until we traced every request.
  • The first stop drill could not sign in. The ring did what it should — handed the ticket to a person (incident #2) — and we built the probe account it uses now.
  • GitLab's merge request list reports a lazily computed merge status ("unchecked"), so the conductor first missed a review's request for changes; it now reads the reviewers' own states.

Accomplishments that we're proud of

  • Two customer rings closed end to end with no person in between — one of them after an agent reviewer rejected another agent's fix and the loop recovered on its own — and one refusal that wrote no code.
  • Every gate fails closed: a missing scan report, a stop drill that doesn't restore the old behaviour, a reply that mentions another customer, an exhausted credit or attempt budget — each stops the ring and hands it to a person.
  • A per-report carbon figure on every closed ticket.

What we learned

"Merged" is a proxy. The moment you verify against the reporter's own traffic, the pipeline has to be honest about sampling, flags, auth and timing — and most of the engineering went there, not into the agents.

What's next

Multiple flags per ticket, reporter-scoped canaries for config changes, and running the same ring on a second open-source app.

Path B: what we added

Original project: thedevs-network/kutt v3.1.0 (MIT). Everything after git log v3.1.0..main is ours, built during the Submission Period: the ring, the Duo flows and skills, the CI/CD pipeline (Kutt had Docker build workflows and no tests), telemetry, feature flags, the support page, Terraform and the deployment, the synthetic customers and the site. Before/after: docs/what-we-added.md. What we claim, and at what confidence: CLAIMS.md.

Sustainability

SCI per resolved customer report (ISO/IEC 21031, SCI for AI), published on every ring: ring #3 = 30.6 g CO2e (range 8.5–62), CI 1.0 g; ring #5 (two attempts) = 44.6 g, CI 1.5 g. CI and app run in europe-west9 (16 gCO2eq/kWh vs 579 in us-east1); rollback is a flag switch, not a revert pipeline; no fix flow runs for a report production never saw. Method and limits: docs/sustainability.md.

Built With

Share this project:

Updates

Submission history