Proof of Fix

Inspiration

On every team I've worked with, "closed" means "a merge request merged". Nobody goes back to check that users stopped hitting the bug. Now agents write the fix, pass the tests and merge it themselves, so the gap between merged and fixed gets wider. Tests check the case someone thought of. Production runs every case.

What it does

Proof of Fix makes a bug issue close only when production shows the fix working.

1. A bug is filed. A Work item created trigger starts our triage flow, a custom flow in the AI Catalog. It reads production's recent errors, finds where the failure comes from in the code, and writes a proof contract into the issue: the log line that must stop, the event the fix must start logging, and how long to watch.
2. A fix merge request says "Proves #N" instead of "Closes #N". A CI gate refuses a fix that would close the bug on merge, that doesn't log the promised event and assert it in a test, or that deletes the code reporting the error.
3. After the merge, the pipeline deploys to production on Render and proof-watch judges the new build on its own logs, filtered by commit. It needs real traffic before it decides.
4. If the bug is gone, Proof of Fix posts a receipt with before and after numbers, closes the issue and publishes a release that carries the receipt. If not, the issue stays open with the production lines that show the bug is still there, and the build is rolled back only if it got worse. CI hands the receipt to our verdict flow through the Flows API. The verdict flow reads the merged diff, works out what the fix missed, and its diagnosis becomes the goal for the Duo Developer flow.

Our real run is issue #2 in the project. Splitting ₹100 three ways failed. Triage found the same failure on a second route the reporter never used (230 errors on one route, 158 on the other, out of 1180 requests). The first fix covered only the reported route and passed the tests, the contract gate and Duo Code Review. Production disagreed: failures fell from 32% to 9% of requests, all of them on the second route, so the issue stayed open. The verdict flow named the function the fix missed, the Developer flow wrote the second fix, and Duo Code Review caught an edge case in it before approving. After that deploy, production showed 0 failures on 875 requests and the new event 375 times. The issue closed itself with the receipt.

How we built it

• billsplit, the production service, is a small FastAPI expenses app with a real rounding bug on two routes. It sends OpenTelemetry logs to GitLab Observability. Synthetic users in CI give every code path traffic during deploys and proof windows.
• The checker is Python: contracts, the production judge, receipts, the merge request gate and registry retagging, with 38 tests.
• The pipeline is GitLab CI: unit tests with JUnit, the contract gate, SAST, secret detection, dependency and container scanning (in merge request pipelines too), a production environment with a resource group, the proof watch, rollback, the verdict and a release.
• On the Duo Agent Platform: two custom flows in the AI Catalog with Work item created and mention triggers, the Developer and Code Review flows, the Flows API from CI, AGENTS.md and .gitlab/duo/agent-config.yml.

Challenges we ran into

• For most of the build, our group's Observability ingestion endpoint answered 502, so nothing was stored. We kept Observability as the main source and added a fallback: billsplit serves its recent log lines, and the deploy job saves the old build's lines before it replaces them. Every receipt names its source. Midway through our real run GitLab fixed ingestion. The first receipt read billsplit's own lines, and the pipeline switched to Observability by itself for the second.
• Flows never see CI/CD variables, so the triage flow can't hold the Observability key. CI publishes recent production errors to the package registry, and the flow reads them with a standard-library script, because the flow image we got had Python without pip.
• The verdict flow's own start_flow call couldn't create the Developer flow's pipeline under its service account. It said so on the issue, and we started the Developer flow with its diagnosis through the Flows API.

Accomplishments that we're proud of

A merge request that passed every test, scan and review was caught by production minutes after it shipped, and the issue stayed open until the real fix was proven.

What we learned

Writing down what production will look like once a bug is fixed, before anyone writes the fix, is what lets a pipeline check it afterwards.

What's next

Contracts for latency and error-rate targets as well as log lines, and proofs that wait for real user traffic instead of synthetic users.

Built With

• docker
• fastapi
• gitlab
• gitlab-ci-cd
• gitlab-container-registry
• gitlab-duo-agent-platform
• gitlab-observability
• opentelemetry
• pytest
• python
• render