ShakeFix: Shake Your Phone, AI Agents Ship the Fix

Her pharmacy app counted refills up, not down. She shook her phone and said what was wrong. 18 min later GitLab Duo agents fixed, proved and shipped it, zero humans. She confirmed it.

Comment

Inspiration

For apps people can't afford to get wrong. 43% of US adults 65+ take five or more prescription drugs (CDC), and adverse drug events send more than 1.5 million Americans to the emergency room every year (CDC). The people who depend most on a pharmacy app are the least able to report what went wrong in it.

Picture Margaret, 78 (our demo patient). She refills her heart pills and the app says she has more refills left than before, so it will never tell her to ask her doctor for a new prescription. Reporting that today takes steps to reproduce, a screenshot, an app version and someone who believes her. So she calls her daughter, who emails support, and the ticket ends as "cannot reproduce", like 17% of all bug reports (MSR 2014).

ShakeFix makes it one gesture. She shakes her phone and says what went wrong; that is the whole bug report. GitLab Duo agents turn her taps into a failing test, fix it, prove the fix changed only what it should, and ship it. In our live run (case #18) that took 18 minutes 34 seconds with zero human touches. Then her phone asks "Fixed ✓. Is it working for you now?", and her 👍 is what closes the case.

What it does

ShakeFix turns a phone shake into a verified production fix:

  1. Shake. The SDK sends the user's last 30 actions, console errors, network calls and a redacted screenshot. She can also say what went wrong: ElevenLabs Scribe turns her voice note into an "In their words" section on the issue.
  2. Triage. The intake (Cloud Run) opens a GitLab issue, and one GitLab Duo custom flow runs four agents: Triage Nurse, Surgeon, Attending and Discharge. The replay becomes a Playwright test that must fail on main.
  3. Surgery. The Surgeon makes the smallest change that turns that test green, inside policy limits.
  4. Second Opinion. CI deploys main and the fix side by side on Cloud Run, replays real user flows on both, and diffs screens, API responses and database writes. Each change is labelled intended or unintended. Fix Confidence (0–100, GO / HOLD / BLOCK) decides. An unintended change sends the fix back to surgery.
  5. Ship. GO → auto-merge → deploy → production smoke test (automatic rollback on failure) → the reporter gets "Fixed ✓ — is it working for you now? 👍 / 👎". A 👎 reopens the case with their fresh replay. If she asked for it, an ElevenLabs agent calls her instead, and her spoken "yes" closes the case. After a rollback, the on-call developer is phoned: "send it back to the agents, or hold?" (rollback never waits for the call).
  6. Stay fixed. After every later deploy, every past case's regression spec runs on production; a bug that comes back reopens its case by itself and pages the on-call.

Blast Radius. Before anyone fixes anything, Triage posts which routes, components and (synthetic) patients the bug reaches, as a graph, from GitLab Orbit or a static import graph.

Ground truth. Control MRs with answers committed before they run go through the same pipeline: planted bad fixes must be blocked, harmless changes must pass. Mission Control's Numbers tab scores them.

Signed certificate. Each closed case gets a signed record: issue, MR, commit, Fix Confidence, human touches and the stopwatch. Every number on this page comes from one.

Risky paths (apps/carerx/src/dosing/**, auth/**, infra, SQL) drop to Supervised on their own and wait for a human. Every case adds to the case memory: hotspots, lessons from blocked fixes, and faster repeat fixes.

How we built it

  • GitLab Duo Agent Platform: custom flow (.gitlab/duo/flows/shakefix.yml), 4 agents, triggers, sessions, agent-config.yml sandbox with Chromium for Playwright.
  • GitLab CI/CD: lint, unit, regression spec, SAST, dependency scanning, secret detection, Cloud Build image, Cloud Run preview revision per MR, Second Opinion, gate, auto-merge, deploy, smoke, rollback, notify, ledger, Pages.
  • Google Cloud: Cloud Run (tagged revisions + traffic shifting), Cloud Build, Artifact Registry, Firestore, Secret Manager, Workload Identity Federation (no JSON keys), all in Terraform (infra/).
  • Voice: ElevenLabs Scribe (speech-to-text) for voice notes; ElevenLabs agents on a Twilio number for the reporter's confirmation call and the on-call rollback call; signed post-call webhook back into the intake.
  • Second Opinion: Playwright capture + deterministic diff + rule labeller. An optional Claude call can only make a label stricter; if it disagrees the other way, a human decides. Fix Confidence is a published formula, no model.

DevSecOps stages (9/9)

Plan · Create · Verify · Package · Secure · Release · Configure · Monitor · Govern (table in the README).

Challenges

  1. Triggers fire only on human actions, so the intake creates issues with the maintainer's token.
  2. Duo hides HTML comments from agents, so the report id travels as a visible line.
  3. The agent sandbox has small-icu Node and no corepack: formatting is locale-free, pnpm comes from npm.
  4. A harmless control MR was falsely blocked: the footer's version text differs between any two builds and shifted the centred line on every screenshot. Apps can now mark build-specific regions data-shakefix-mask.
  5. The intake's own 👍 note was once counted as a human touch (its webhook beat the stored note id); every intake note now carries a hidden marker.
  6. Second Opinion flaked when Cloud Run ran two instances of an in-memory demo app (reset and replay hit different instances). Pinning test revisions to one instance made diffs deterministic.

Accomplishments

Times from each case's signed certificate.

  1. #18 refill count (recorded live for the video) → MR !26, hands-off, 0 human touches: shake 05:52:49 UTC → issue 05:52:52 → MR !26 by the Surgeon agent 05:58:40 → Second Opinion 06:05:38 (4 intended, 0 unintended, Fix Confidence 100/100 GO) → bot merge 06:06:32 → live 06:11:23 (18m 34s). Her voice note is on the issue as "In their words" (ElevenLabs Scribe). Reporter 👍 later the same day.
  2. #14 double charge → MR !12, hands-off: shake 08:21:01 UTC → bot merge 08:34:03 → live 08:45:52 (24m 51s), Second Opinion 0 unintended, reporter 👍. No person touched code, review or merge; the certificate counts 1 touch, our intake's own 👍 note (a webhook race we fixed right after).
  3. #16 accessibility → MR !14, hands-off: Second Opinion blocked the first run on two of its own bugs; the agents re-operated by themselves, GO 100/100, bot merge, live 26m 03s after the shake.
  4. #17 dosing (mg vs mcg) → MR !15, supervised by policy: HOLD 80/100; a person approved and merged; live 37m 55s after the shake.
  5. Controls: 12 of 12. Twelve control MRs with the expected verdict committed before any ran (4 GO, 3 HOLD, 5 BLOCK) all matched, at the pinned SHAs: controls/RESULTS.md. Every planted side effect was blocked; no harmless change was.
  6. Rollback drill: forced smoke failure → traffic back on the previous revision in 40 s, on-call paged.
  7. Voice, tested live: ElevenLabs Scribe transcribed a voice note through the intake's real code path, and an ElevenLabs agent on a Twilio number placed a real 35-second "is it fixed for you?" call to an Indian mobile. The answer was unclear, so the case correctly stayed open.

What we learned

The fix is AI code too. Checking behaviour (what users see) instead of reading the diff catches the side effects reviewers miss, and the person who reported the bug is the right one to say it's fixed.

What's next

GitHub and other models through the provider switch (already config-only), native iOS/Android SDKs, and Second Opinion as a standalone check for any AI-written pull request.

Built With

  • artifact-registry
  • claude
  • cloud-build
  • elevenlabs
  • fastify
  • firestore
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • gitlab-pages
  • google-cloud-run
  • node.js
  • playwright
  • secret-manager
  • terraform
  • twilio
  • vite
  • workload-identity-federation
Share this project:

Updates

Submission history