FaultForge: Supervised Autonomous Resilience Agent

Closed-loop agent that injects chaos, diagnoses SLA breaches via AST & bisect, prompts GitLab Duo to heal code, and replays the exact fault to mathematically prove recovery under human supervision.

Comment

Inspiration

Most CI pipelines only verify that unit tests pass in isolation. But unit tests almost always mock external HTTP dependencies. In production, when a
downstream carrier or cloud API experiences transient network degradation, thread pools choke, request queues starve, and cascading 5xx timeouts
occur.

While modern AI coding tools can suggest patches, **AI cannot tell you if its patch actually survives real-world failure**. We wanted to eliminate
the dangerous gap between "AI says it fixed the bug" and "mathematical proof under identical network failure." We were inspired to build a
closed-loop platform engineering agent that connects **chaos engineering, GitLab Duo automated code remediation, and empirical before-and-after
proof**.

## What it does
FaultForge embodies the **"Break It, Heal It, Prove It"** lifecycle across all 9 GitLab DevSecOps stages under strict **Supervised Autonomy**:

1. **Injects Chaos:** Provisions Toxiproxy network chaos into an isolated downstream shipping dependency (`3,500ms` delay ± `250ms` jitter, profile
hash `644a89784316`).
2. **Detects & Proves Baseline Failure:** Records 5/5 SLA breaches (>1500ms SLA budget, baseline mean `3,637.98ms`), captures telemetry logs, and
provisions an automated Incident Work Item.
3. **Deterministic Diagnosis & Bisect:** Analyzes call-sites via AST inspection and executes automated `git bisect` to isolate the culprit commit
(`974035c6`) where timeout and circuit breaker policies were stripped.
4. **GitLab Duo Remediation:** Triggers a custom GitLab Duo Agent Platform flow (`flows/faultforge-flow.yaml`, Session `9909079`) whose Diagnostician
and Healer synthesize a minimal scoped fix: a bounded `0.8s` timeout, 3-failure circuit breaker, and deterministic fallback calculation in
`app/clients/shipping_client.py` (Merge Request `!3`).
5. **Exact-Fault Replay Proof:** Re-executes the **identical Toxiproxy fault profile** against the candidate release across 20 probes. All 20 probes
pass SLA budget (**0/20 breaches**, -86.6% mean latency reduction down to `487.92ms`, p95 `813.32ms`).
6. **Cryptographic Proof & Staging Deploy:** Generates a SHA-256 sealed Evidence Receipt and deploys candidate containers to Google Cloud Run staging
with a 10% canary traffic split.
7. **Supervised Production Promotion Gate:** Live production traffic remains safely pinned to the stable baseline. Automation intentionally halts at
a manual governance gate until a human operator reviews the cryptographic audit receipt and diff before signing off on full live promotion.

## How we built it
* **GitLab Duo Agent Platform:** Custom flow schema v1 (`flows/faultforge-flow.yaml`) featuring ambient execution, scoped tools (`read_file`,
`list_dir`, `run_command`), and automated Healer prompting.
* **Chaos Injection Engine:** Embedded Shopify Toxiproxy (`v2.9.0`) running inside CI job containers and Docker Compose environments to inject
reproducible network degradation without arbitrary sleeps.
* **Core Microservice:** High-performance Python 3.12 + FastAPI checkout microservice with an interactive telemetry workbench dashboard.
* **Testing & Integrity Suite:** 183 automated pytest test suites covering AST inspectors, autonomy routers, risk classifiers, and anti-fake truth
audit guards.
* **Google Cloud Run:** Multi-stage distroless container deployment with Knative traffic splitting and rollback automation in `us-central1`.
* **Tamper-Evident Ledger:** 9-stage DevSecOps cryptographic ledger (`stage_ledger.json`) sealing every pipeline artifact with SHA-256 hashes.

## Challenges we ran into
* **Avoiding Mocked Chaos in CI:** Many hackathon projects simulate chaos with `time.sleep()`. We wrote strict anti-fake truth audit guards
(`scripts/truth_audit.py`) that fail CI if artificial sleeps or mocked test fixtures are detected.
* **Duo Flow Schema Strictness:** GitLab Duo Agent Platform custom flow validation strictly rejects top-level `name`, `description`, and `model`
fields inside flow YAML. We aligned our flow specification directly with GitLab Duo's Flow Registry v1 specification.
* **Cryptographic Evidence Integrity:** Ensuring that the application tree evaluated by the chaos replay (`replay_tree_sha`) matched the deployed
Google Cloud Run container (`deployed_tree_sha`) byte-for-byte (`de7c0b1cc7d6b7edb90c6b3041b10885ea702492`).

## Accomplishments that we're proud of
* **Full 9-Stage DevSecOps Coverage:** Touched all 9 official GitLab DevSecOps stages with verifiable machine-readable artifacts on SaaS public CI
Pipeline [#2984589851](https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43424456/showcase/-/pipelines/2984589851).
* **Supervised Autonomy by Design:** You approve the outcome, not the individual steps. The agent autonomously handles bisecting, testing, MR
opening, and staging deployment, while placing the final production promotion into human hands.
* **100% Verifiable & Reproducible:** Any judge can run `./scripts/local_gate.sh` or `python scripts/truth_audit.py --final-audit` and verify all
tests and cryptographic claims locally in under 60 seconds.

## What we learned
* Autonomous software engineering is most trustworthy when paired with **exact before-and-after failure replays**.
* Rigorous DevSecOps governance requires machine-readable evidence receipts; an AI agent proposing code must be subjected to the same rigorous
compliance and security gates as a human developer.

## What's next for FaultForge
* Expanding the chaos library to inject distributed packet slicing, cross-region bandwidth throttling, and cascading upstream gRPC resets.
* Upgrading from static Toxiproxy profiles to dynamic, adaptive chaos scenarios driven by real-time production telemetry anomalies.

Built With

Share this project:

Updates

Submission history