ChaosGarden

• 

  Repaired lost-ack experiment: one fulfillment and one durable receipt. Recorded synthetic CI evidence.

• 

  Evidence board linking native Duo, repair commit, independent CI and downloadable reports.

A webhook can commit an order, lose its acknowledgement and receive the same event again. ChaosGarden lets backend maintainers reproduce that failure against a real FastAPI/SQLite fixture. The unsafe handler fulfills twice; a receipt recorded in the same transaction keeps the side effect to one local fulfillment. GitLab Duo proposed the bounded handler repair, and Codex relayed the recorded gates under the owner's delegation. This demonstrates local idempotency, not exactly-once delivery across independent systems or an actual payment charge.

The project is a working FastAPI and SQLite reliability lab. Its lost-acknowledgement harness confirms that the database committed before replacing the response with a synthetic HTTP 504. It then retries the identical event and queries the real database. The unsafe handler produces two fulfillments; the receipt-based handler produces one. Concurrent delivery and conflicting-payload experiments challenge the same invariant from different directions. These are executed integration experiments with fictional orders, not simulated green dashboards or calls to payment providers.

GitLab Duo Agent Platform performs the repair workflow. A registered custom flow investigates the failure, proposes a bounded handler change, applies the approved proposal, verifies it and publishes it to the existing merge-request branch. Native checkpoints separate reproduction, repair, verification and publication. Model agents do not receive unrestricted tools: fixed programs enforce the approved scope, recheck the expected MR head and limit the repair to app/webhooks.py. The repair cannot make its own result green by changing the tests, fault harness or CI contract.

The strongest recorded outcome is native session 9758709. Unsafe commit b88b2445 failed in pipeline 2968053460; Duo published repair 50a465c2, and pipeline 2968364769 passed all nine jobs. Native verification recorded 73 passing tests, Ruff, Bandit and a lost-ack result of one fulfillment. These figures describe that retained run, not an invented benchmark or a claim about the size of today's test suite.

The owner set the permitted scope and delegated operation of the native approval controls to Codex. In session 9758709, Codex exercised four Approve decisions and one Modify under that delegation. The Modify path returned feedback to the proposal before the bounded repair proceeded. This was an Assisted run with agent-operated gates, rather than direct owner browser approvals. The public run-trail documentation maps the checkpoints to their actual executor jobs and pipelines; native session access can require GitLab membership.

The read-only console makes the mechanism inspectable. Reviewers can compare handlers, follow recorded deliveries, inspect fulfillment counts and response codes, and open the underlying incident JSON. The newest local workbench adds direct event selection, keyboard stepping and checkpoint-preserving comparison. Its controls replay existing measurements; they do not start a new experiment or authorize an agent. The public Pages mirror remains a separately labeled historical snapshot of the October 14 run, not the latest native repair or the locally refined interface.

The engineering lesson is that reliability evidence needs a precise boundary. The receipt and fulfillment share one SQLite transaction; identical retries become no-ops and changed payloads are rejected. This establishes at-most-once local fulfillment for an event in that database. It does not establish exactly-once delivery across external systems, reproduce a physical network outage, or replace a production incident response process.

ChaosGarden's demonstrated automation spans Create, Verify, Secure, Govern and Package. Release and Monitor are configured but unrun, and demo MR !1 remains unmerged. The useful outcome is a reviewable repair with a reproducible failure, controlled scope and independently inspectable CI evidence. A maintainer can understand why the retry failed and what the repair actually changed before deciding what happens next.

ChaosGarden makes a lost acknowledgement reproducible and the local fulfillment count observable. Duo proposed the handler repair, while Codex assisted engineering, evidence assembly and the explicitly delegated gates; Claude supplied read-only review. The narrated demonstration uses edited retained evidence and Deepgram synthetic speech. The result concerns one transactional fixture, not payment processing or distributed exactly-once delivery.

Lifecycle stages demonstrated: create, verify, secure, govern, package. Release and monitor are configured but not run.

Built With

• ajv
• bandit
• fastapi
• gitlab-ai-catalog
• gitlab-ci/cd
• gitlab-duo-agent-platform
• pydantic
• pytest
• python
• ruff
• sqlite
• typescript
• vite