CALLBACK

• 

  Production calls: CALLBACK rings the on-call engineer about slow checkout

• 

  One word, "Yes.", and CALLBACK starts investigating

• 

  "Ship it." The engineer approves the fix by voice

• 

  GitLab Duo triage: !21 is the suspect, with fix !22 and revert !23 ready

• 

  The fix pipeline runs tests, build, Golden Ears and security scans

• 

  Golden Ears passes: an AI listening test scores the agent's voice

• 

  Evidence gate: production deploys only with a recorded voice approval

• 

  !22 is approved and merged on the engineer's behalf

Inspiration

On small teams there is no SRE. When production degrades, someone gets paged, finds a laptop, fights the VPN and spends twenty minutes hunting for context before anyone looks at a fix. At the same time, AI agents are getting good enough to fix production themselves, which raises a harder question: who said yes?

CALLBACK started from one image: the phone rings, a calm voice says what broke and that a tested fix is ready, and you answer with one word.

What it does

When production slows down, CALLBACK phones the on-call engineer:

"Checkout is slow: p95 is 1.7 seconds since release 0.1.6. Want me to dig in?"

The engineer says "Yes." That reply is transcribed, matched to an intent and stored with the recording's SHA-256 hash. Only then does CALLBACK act as the human in GitLab, mentioning the callback-triage GitLab Duo flow on the incident. The flow finds the merge request that caused the slowdown, opens a fix with a regression test, opens a revert as a backup, and reports the root cause.

The fix pipeline runs tests, builds the image, runs SAST and secret detection, and runs Golden Ears, a perceptual listening test of the agent's own voice. When it's green, the phone rings again:

"Tests, security scans and Golden Ears all passed. Ship it, or roll back?"

"Ship it." CALLBACK approves and merges the fix as the human, with the transcript and audio hash in the approval comment. On main, an evidence-check job refuses to deploy any fix or revert without a stored voice approval. The callback-release flow writes release notes, the pipeline deploys and cuts a GitLab Release, and verify-prod measures production. CALLBACK closes the incident and calls one last time: "Fixed and live."

A second loop guards the voice itself: when a change degrades the agent's voice, Golden Ears fails the MR, and the golden-ears-guard flow pushes a fix commit and posts before/after scores, with no human beyond the push.

How GitLab automates the whole post-code lifecycle

Every loop touches all nine stages, and the incident page lights each one from a real event:

The governance idea

GitLab only starts a flow from a human action: never from a bot, a service account or another flow. So an alert alone can never start triage. I made that rule the feature: CALLBACK holds a bot token for routine writes and uses the engineer's own token only immediately after a recorded voice approval, posting the transcript and audio hash in the same comment. Every Duo session in the project traces back to a human voice.

How I built it

• Google Cloud Run (us-central1): shop (the storefront that breaks), hq (webhooks, call engine, approvals, the phone app) and voice (Kokoro TTS + faster-whisper on CPU)
• Firestore for incidents, calls and approvals; Cloud Storage for recordings; Cloud Monitoring for the alert; Secret Manager; Workload Identity Federation for keyless CI deploys. All in OpenTofu.
• Golden Ears on Modal GPUs: WhisperX for intelligibility, UTMOS for naturalness, ECAPA for speaker similarity, against a frozen baseline.
• Three custom GitLab Duo flows on Mention, Merge-request-approved and Pipeline-failed triggers.
• The phone is a home-screen PWA with push-to-talk audio captured through Web Audio.

Challenges

• The human-trigger rule forced a rethink of the whole design, and ended up being its spine.
• iOS audio: MediaRecorder silently dropped short replies like "yes", so push-to-talk now records raw WAV through Web Audio.
• Hearing a bad voice: an 8 kHz voice kept a 0% word error rate and barely lost naturalness. Speaker similarity (0.66 against a 0.80 floor) is what caught it.
• A perfect revert hides itself: when the guard's fix exactly undid the bad line, the MR had no diff left and CI skipped the re-score. The pipeline now always re-scores the guard's own commit.
• Real-world latency: GPU queues and shared runners made a full loop take about 25 minutes of wall time, so I added a GPU fallback and kept every step observable.

Accomplishments

Four full loops in the public pipeline history (two "ship it", two "roll back"), the Golden Ears guard beat live, and a demo where one spoken "yes" and one spoken "ship it" carry a real regression from alert to verified fix.

What I learned

Constraints make better products: the rule that flows need a human became the governance model. A release gate can listen, not just read. And voice is a surprisingly good approval channel: it's fast, hard to do by accident, and leaves an audit trail that a click doesn't.

What's next

Canary deploys between staging and production, a postmortem flow on the final call, and on-call rotations across a team.

Built With

• artifact-registry
• cloud-build
• cloud-monitoring
• fastapi
• faster-whisper
• firestore
• gitlab
• gitlab-ci
• gitlab-duo-agent-platform
• google-cloud
• google-cloud-run
• kokoro-tts
• modal
• opentofu
• pwa
• python
• react
• secret-manager
• speechbrain
• tailwindcss
• typescript
• utmos
• web-audio-api
• whisperx
• workload-identity-federation