Drill Gate

Inspiration

Twenty seconds without Redis, and LibreChat never came back. Every chat answered 503, then every login answered 500, long after Redis was healthy again. The code looked fine. Its tests passed. Nobody had ever broken Redis on purpose to find out.

We started from three findings that, put together, describe a problem nobody owns:

• More code is written by AI than anyone reviews. In JetBrains' 2026 survey of 15,000+ developers, about 47% of their code is written fully by agents, and over half write less than 20% by hand (JetBrains Research, Aug 2026). Across 500+ organisations, about 27% of merged code is AI-generated (DX, 2026).
• Teams already feel it in production. In a survey of 200 US technology leaders, 78% report more production incidents, 82% had a production failure tied to AI-generated code in six months, and 62% say teams often ship AI code without line-by-line review (New Relic 2026 AI Code Report; vendor-commissioned and self-reported).
• Outages come from failure handling, not from the happy path. In 198 real failures across Cassandra, HBase, HDFS, MapReduce and Redis, 92% of the catastrophic ones came from mishandling errors the software had already signalled, and a third of those were trivial mistakes in the error handler (Yuan et al., OSDI 2014).

So the riskiest part of a change, what it does when an outside service fails, is the part least likely to be read, and tests almost never exercise it. A silent catch that returns a default "so the user isn't blocked" reads as reasonable in review.

Buildings run fire drills because you don't find out whether the alarm works during the fire. We wanted a fire drill for every merge request: break every outside service the change depends on, on a live preview, and let the evidence decide whether it merges.

Hackathon fit: Path B on LibreChat (MIT, 43k stars). Autonomy: Supervised. GitLab Duo plans drills and commits fixes on its own; the pass/fail verdict is code; a person approves every merge and starts every production rollout.

What it does

Drill Gate follows a change through its life after the code is written: before the merge, at deploy, and in production.

Before the merge. Every merge request gets a private preview on Cloud Run with a fault-proxy sidecar. The proxy breaks each outside service the change depends on, three ways: the connection is refused, it hangs, or it is cut mid-reply. It also returns provider errors (429, 500). It works for HTTP services (AI providers, third-party APIs) and for plain TCP ones (databases, caches, queues). Then plain code, not an LLM, asks five questions per fault, from GitLab Observability and the response itself:

1. Marked failed: does the request's trace carry an error span?
2. Logged: is there an ERROR log with the same trace ID that names the service?
3. Alerted: does the repository's alert rule for that service fire?
4. User told: did the user get a clear error, not a silent 200 or a hang?
5. Recovers: once the service is back, does the same request work again within 90 s?

Any "no" blocks the merge. The GitLab Duo fixer reads the gate's report and commits the smallest fix (a timeout, an error span, an ERROR log, a 502, an alert rule); the gate runs again, and a person approves. The fixer gets two attempts, then hands the merge request to a person with a note.

Existing code too (regression mode). The repository lists its important request paths once, with the files that serve each. When a merge request touches one, CI deploys main's code beside the MR's and drills both with the same faults. The MR is blamed for a check only when main passes it and the MR fails it, and only when that is seen twice (under two faults, or again on a re-run). Problems main already has are reported, never blamed on the MR, so a team can adopt the gate on imperfect code.

At deploy. A person starts the rollout. The new version takes 10% of traffic, its own spans in GitLab Observability decide, and it is promoted to 100% or rolled back automatically.

In production. An hourly watch reads production from Observability. When it fails, the GitLab Duo incident responder opens an issue with the evidence, the trace IDs and the suspect commit.

For any project. Drill Gate ships as a GitLab CI/CD component: one include: gives a containerised app the merge-request gate, with a setup script for a team's own Google Cloud. We proved it on a second app in a second language (Python).

How we built it

• Fault proxy (Go): a sidecar next to the app, with ingress to the app, an egress proxy that refuses, hangs or resets connections per host, TCP forwards for databases and caches, and a token-protected control API. Arming a fault also cuts already-open connections, so pooled clients can't dodge it.
• The gate (Python, standard library only): a healthy control request first, then every fault; it reads traces, logs and alert rules from the GitLab Observability API, writes JUnit for the merge request's test widget and a compact report for Duo.
• GitLab Duo Agent Platform: three flows. The planner (on @mention) writes the drill plan from the diff, the fixer runs on a failed drill, and the incident responder runs on a failed production watch. Each has a narrow toolset and written rules on what it may never edit.
• Making it binding: "Pipelines must succeed", an approval rule, and a manual production deploy that can't be skipped.
• Google Cloud: Cloud Run (production, per-MR previews, mock services, the live results page), Cloud Build, Artifact Registry, Secret Manager, and keyless CI access through Workload Identity Federation with separate deployer, verifier and runtime identities.
• Measurement built in: the mutation eval's unit-test arm and coverage probe run as jobs in the same pipelines as the drill, so both arms are judged on the same commit.
• A live results page: CI rebuilds it every hour from the drill jobs' own artifacts, with every run, red ones too, each check for the MR and for main, each linked to its job.

The verdict is deliberately not an LLM. Duo plans and fixes; code decides, from evidence anyone can open.

Challenges we ran into

• Telemetry arrives late, and logs later than spans. Our first gate judged too early and failed correct code. It now polls until each check passes or a deadline expires, and records what Observability did see, so every red verdict explains itself.
• Some traces never arrive. A refused connection makes the spans of its trace disappear before Observability, in Node and in Python alike, while the ERROR log with the same trace ID still arrives; neither SDK reports an export error. In that one case check 1 passes as "unverified", and the report says so.
• A broken app poisons every later test. After a Redis fault LibreChat stayed down, so every later case measured only the outage. That is why check 5 exists: after each fault the gate waits for the app to work again, and stops the drill if it doesn't.
• Our own gate had bugs, and the drills exposed them. A login that expired mid-fault, a case main never ran counted as a regression, and a request path that was silently never drilled. Each made a run report something false. All are fixed, and the results page still lists the runs they affected, labelled.
• Our first eval sample hit code no failure reaches. We had to measure that rather than argue it: a probe build logged a line at each of 273 error-handling sites and was drilled like a mutant. Only 2 fired.
• Developer access only. The hackathon namespace gave us no GitLab Pages and no maintainer settings, so the results page runs on Cloud Run. Long branch names broke keyless sign-in (the identity subject went over Google's 127-byte limit), so we moved the mapping to project and job IDs.
• An agent that fixes needs a brake. Without a limit, the fixer kept committing to one merge request when the real cause was below the app. It now gets two attempts, then hands over.

Accomplishments that we're proud of

Two real bugs in LibreChat, found by the gate, root-caused and fixed. Nothing planted.

1. Redis never recovers. After a 20 s Redis outage, LibreChat stays down until restart. We traced it to redisClients.ts: the Redis clients stop reconnecting after 10 attempts (about 18 s of backoff) and nothing creates new ones. Our fix in !36 keeps reconnecting and logs an error while Redis is unreachable. Drilled side by side, main was still failing 90 s after Redis came back; the fix worked again in about 4 s. Two of three runs of the fix passed; the third failed partly on a gate bug we then fixed, and the results page shows all three.
2. Timeouts never reach the proxy tunnel. Behind a corporate proxy, an AI provider that never answers holds the chat for up to 5 minutes, whatever timeout is configured. We read undici's source: ProxyAgent builds the client that opens the CONNECT tunnel without the caller's timeouts. We reproduced it outside LibreChat (a 2 s timeout still waiting at 20 s; with the fix, 2.0 s) and fixed it in !7 with a test that fails before and passes after. Drilled side by side, main made 1 provider attempt in 150 s; the fix made 6, each cut off on time. LibreChat's own unit test mocks undici, so it could never have seen this.

A pre-registered evaluation, including where we lose. We committed the protocol before any mutant ran: every error-handling site in the seven files a chat request's failure passes through (273 sites), three mutation operators from the exception-handling mutation literature (swallow a catch, drop an error log, drop a re-throw), and a random sample of 24 drawn with a fixed seed anyone can redraw. Each mutant became a merge request judged by both arms in the same pipeline.

• On the random sample, LibreChat's unit tests caught 5 and Drill Gate caught 0.
• Our coverage probe explained why: only 2 of the 273 sites run when the provider fails, and none of the 24 sampled sites is one of them.
• On those 2 sites, both arms caught both mutants, and the drill also showed what the user loses: with one mutant the user is no longer told and no error is logged, under four faults.
• Every change to the plan is a dated amendment that says what we knew when we made it.

Regression mode on a real path. In !6, a merge request that quietly drops LibreChat's upstream-error log lost check 2 under four faults that main passes. The gate blocked it.

Red, Duo fix, green. In !4, a new translate route that silently returned the original text failed checks 1, 2 and 4 under all three faults, with no alert. The Duo fixer committed the timeout, error span, ERROR log, 502 and alert rule, and the gate passed. The same happened on a Python app through a single include:.

After the merge. The first production canary was judged from Observability and promoted. A rehearsed bad canary was rolled back in 56 s. A rehearsed outage made the incident responder open an issue within a minute, with trace IDs and the suspect commit.

Every run in the open. The live results page shows 20 drill runs across 5 merge requests, red ones too, each linked to its job.

What we learned

• Unit tests and drills catch different bugs, and we can now show it. Every unit-test kill in our eval was in code a request reaches only under conditions a drill doesn't create. Both bugs the drill found were invisible to unit tests: one because the test mocked the HTTP library, one because nothing tested Redis coming back. You want both.
• "It recovered" is a claim to check. The worst bug we found wasn't a missing error message; it was an app that never came back. Breaking a service and then restoring it taught us more than breaking it alone.
• Compare against main, or nobody adopts the gate. Real codebases already have gaps. A gate that blames every merge request for old problems gets switched off; regression mode blames each change only for what it made worse.
• Pre-register, or you'll grade yourself kindly. Writing the protocol before the first run is what made us publish the 0 vs 5, and then measure why instead of explaining it away.
• Keep the judge out of the model. Duo is good at reading a diff and writing a fix. The pass/fail decision stays in code that reads evidence, so it is reproducible and auditable.
• Showing failures makes the rest believable. Publishing the runs our own bugs broke, and the eval we lost, made every other number more credible, to us first.

What's next for Drill Gate

• Make a hanging Redis tell the user. Under that fault the chat still waits 150 s without an error, with our fix too.
• A total deadline for LibreChat's model request, so the proxy fix also reaches the user in time.
• More dependencies. Drill the database directly, and handle clients that discover servers on their own (MongoDB Atlas +srv, Redis Cluster), which bypass a single TCP forward today.
• More fault types: slow responses, wrong data and rate limits, not only dead services.
• Grow the eval. Drill more failure paths (database, job store), so more of the 273 sites are reachable, and re-run the same pre-registered comparison.
• Gate the production canary on the same five checks, and publish the component in the GitLab CI/CD Catalog with the fault proxy as a public image.

Built With

• docker
• gitlab
• gitlab-ci/cd
• gitlab-ci/cd-components
• gitlab-duo
• gitlab-duo-agent-platform
• gitlab-observability
• gitlab-package-registry
• gitlab-secret-detection
• go
• google-artifact-registry
• google-cloud
• google-cloud-build
• google-cloud-run
• google-secret-manager
• mongodb
• node.js
• opentelemetry
• python
• redis
• typescript
• workload-identity-federation