Faultline

Inspiration

AI agents can fail without crashing. A tool may return stale or incomplete data, and the agent can confidently produce an incorrect result while normal monitoring remains green.

Faultline was inspired by this gap between infrastructure reliability and behavioral reliability. We wanted to build a tool that could deliberately break an agent, measure how it failed, and then help repair the underlying source code.

## What it does

Faultline is a chaos-engineering and hardening toolkit for AI agents.

It:

• Creates a ranked attack plan from the agent's codebase.
• Injects realistic faults below the agent framework.
• Grades failures from graceful recovery to silent-wrong behavior.
• Records transcripts, injected faults, end states, and failure reasoning.
• Sends failure dossiers to Codex for source-level repairs.
• Re-runs the same tests and accepts only patches that improve resilience.
• Provides a dashboard and static HTML report backed by SQLite.

Faults include stale data, empty results, timeouts, flapping tools, schema drift, and injected instructions.

## How we built it

Faultline is a Python CLI with a deterministic, seeded execution engine and SQLite ledger.

The architecture is:

  plan -> break -> grade -> score -> harden -> verify -> re-break

  It supports three injection surfaces:

  - Agent tool wrappers
  - An OpenAI-compatible LLM proxy
  - An MCP JSON-RPC interceptor

  GPT-5.6 is used for attack planning, structured judging, and anti-cheat review. Codex is used as the hardening engine through headless codex exec.

  The project also includes:

  - A FastAPI localhost dashboard
  - A self-contained offline HTML report
  - A fault-class heat map
  - A resilience survival curve
  - Patch provenance and ledger evidence
  - An offline demo that requires no API key

  ## Challenges we ran into

  The most important bug was a step-indexing mismatch. Human-readable plans used one-based step numbers, while the tool wrapper counted from zero. As a result, scheduled faults
  could silently fail to inject and produce an invalid perfect score.

  We fixed this by normalizing external plans at one boundary and keeping internal schedules zero-based. We also added guards that abort when scheduled faults produce no
  injection events and tests for unreachable schedules.

  Other challenges included preventing stale ledger data from contaminating later attempts, preserving deterministic behavior across seeds, validating Codex patches without
  duplicating engine logic, and keeping the entire offline demo reproducible.

  ## Accomplishments that we're proud of

  In the official run against the intentionally vulnerable support bot:

  - The baseline score was 20.6/100.
  - The score climbed to 41.2, 51.8, and finally 87.1.
  - The release gate was 85, and the final run passed.
  - All 12/12 scheduled faults produced injection events.
  - The baseline contained three real Grade-D stale-data failures.
  - Codex produced three accepted source-level repairs.
  - The final happy-path and regression suites passed.
  - The project includes 136 passing offline tests.
  - The dashboard and static report make the evidence readable and reproducible.

  Most importantly, the final score came from re-running the same fault scenarios rather than changing the scoring or hiding failures.

  ## What we learned

  We learned that agent reliability must measure behavior, not just whether a request completed.

  A loud failure can be safer than a confident incorrect answer. We also learned that reproducibility is essential for automated hardening: the same schedules, seeds, end-state
  assertions, and ledger records must be used before and after every patch.

  Finally, we learned that AI-generated repairs need strong gates. Codex patches must preserve the happy path, address the fault class generally, and improve the measured score
  before they are accepted.

  ## What's next for Faultline

  Next, we want to make Faultline easier to adopt in real agent projects by expanding framework adapters, improving installation and packaging, and adding stronger CI
  integrations.

  We also plan to add:

  - More production-oriented MCP and LLM proxy integrations
  - Richer fault taxonomies for multi-agent workflows
  - Team dashboards and run comparison
  - Long-running resilience trends
  - Public certification and reporting workflows
  - More examples from customer-support, research, and planning agents

  The long-term goal is to make behavioral resilience testing a normal release gate for AI agents.

Built With

• codex
• gpt
• openai
• playwright
• pytest
• python
• sqlite
• typer