BEDLAM

Fixed an outage? Bedlam tests whether the fix holds: it breaks your app, runs your patch, strikes again where the patch didn't reach, then hands you a trade-off to choose.

Comment

Inspiration

The internet is bedlam. Nothing out there is scripted, nobody waits for your patch to land and people break software in a million different ways at once. Any tool that breaks things on purpose, this one included, works in a sandbox. The real world never will.
So I didn't set out to build a shield. I wanted to build a habit: the reflex that a fix isn't finished just because the errors stopped. Something will always break or get compromised eventually, being thorough doesn't stop that, but it makes sure you didn't leave the door open yourself.
It's easy to 'tab + enter' your way through a project, accepting whatever the autocomplete or the assistant suggests. Bedlam is named after the chaos it rehearses for, and it's
built to make you mindful of your decisions and their consequences.

What it does

Bedlam is a chaos agent you play against in the terminal. It attacks a real three-tier app (FastAPI, Postgres and Redis) running in Docker on your own machine.

  1. Break. It injects one network fault on the API's link to the database or the cache, then probes all three layers to see what failed.
  2. Test your fix. You patch the app. Bedlam deploys the patch to the running container, puts the same fault back, and says whether the fix held, with the before and after timings on screen.
  3. Break it again. It reads your diff to work out what the patch defends and what it leaves open, searches the web for how those dependencies fail in the wild, and strikes where your fix doesn't reach.
    After two rounds it stops attacking and hands you a trade-off instead: two options built from that run's faults and patches, each with a gain and a cost. Bedlam doesn't recommend either one. Once you choose, it applies the cost you accepted, once, so you see it happen.
    Every model call and every search shows up on screen as it happens, with the model, token counts and latency.

How we built it

  • Target: benavlabs/FastAPI-boilerplate as a git submodule, run with Docker Compose next to Postgres, Redis and Toxiproxy. Toxiproxy sits on the API's connections to both stores, so faults are real network conditions (latency, bandwidth caps, cut connections, timeouts), not mocks.
  • Brain: two Nemotron 3 models on Nebius Token Factory, spread across seven call sites. Nano handles the narrow, structured jobs: classifying probe results, writing search queries and labelling search results by layer. Super handles the reasoning: reading the diff, turning search results into candidate faults, choosing the next fault and writing the trade-off. If Super fails, the call falls back to Nano, and the fallback is logged and shown. I left Ultra out because a sub-three-minute loop can't absorb its latency.
  • Search: Tavily, called live during every round. Queries name a dependency, its version and a failure mode, and results are restricted to GitHub issues, pull requests and changelogs. Nano labels each result as API, database or cache, the labels are scored, and the winning layer picks which of three specialists plans the next fault. The routing really does come from the search results, and the screen shows the results, the labels and the scores.
  • Interface: a Textual TUI, readable at 82 columns, with slash commands (/why, /diff, /search, /loop) that let you inspect each decision. Bedlam's voice (a small jester) is templated rather than generated, so it is deterministic, instant and free.
  • Stack: Python 3.12 and asyncio throughout, with read-only probes running in parallel and faults strictly one at a time. Every seed in the run derives from one root seed.

Challenges we ran into

  • Searching for the right thing. My first searches named the app itself and came back as noise. What Bedlam needs is stories of real failures, so queries now name the libraries underneath, with their versions and a failure mode, like "asyncpg 0.29 connection pool exhaustion". They are limited to GitHub issues, pull requests and changelogs, where developers describe what actually broke.
  • Making the patch actually run. In the first version, Bedlam read your diff but never deployed it, so the second break never tested the fix. Deploying it surfaced another problem: the boilerplate connects to Postgres at startup, so reloading it during a database fault left it down for good. The cycle now lifts the fault, restarts the API, warms it with one request, re-applies the fault and measures.
  • Small models mislabel things. Nano sometimes gave a search result a layer it never mentioned. Every label now has to name a component that appears in its result. A bad reply gets one retry, with the reason it was rejected.** If no valid label survives, routing falls back to the probe symptom, then to a seeded draw, and the screen says which basis it used.
  • Not deciding for the developer. The rules are enforced in code, not left to the prompt: exactly two options, two different consequences, short enough for the screen, and no recommending language.

Accomplishments that we're proud of

  • The second cycle (break). Watching Bedlam read a patch, confirm that it held and then go straight around it is the moment the whole project was built for.
  • Search that changes behavior. Tavily is not decorative, different live results route the round to a different specialist and a different fault.
  • Honest verdicts. When a patch doesn't hold, Bedlam points it out and shows the timings that prove it.

What we learned

  • Reproducibility comes from recording the inputs, not from tuning sampling parameters.
  • A small model that gets checked is worth more than a bigger model that gets trusted. Validation plus one retry fixed most of Nano's mistakes.
  • A search API only earns its place when its results change what the program does next.
  • Presenting a choice without steering it is harder than making the choice yourself, for a model and for the code around it.
  • A tool can find the gap, but only the developer can decide to stop and look.

What's next for BEDLAM

  • Security attacks. Today's faults are network failures. The next step is attacks like injection, broken authentication and abused endpoints, run against the same local
    sandbox with the same loop: attack, read your patch, test it, then attack where it doesn't reach.
  • Many agents in parallel. Several Bedlam agents, each with its own sandboxed copy of the app, working on different issues at the same time. Each copy still takes one fault
    at a time, so every patch stays tied to the fault it fixed.
  • More targets. Today each repo needs its own Compose stack, topology map and Toxiproxy wiring. Generating that automatically would let Bedlam run against any service.
  • More fault types: CPU, memory and disk pressure, and killed containers, alongside the network faults.
  • Stronger verdicts: checking a patch against every affected endpoint, not just one request.
  • In CI: run Bedlam against the patch in a pull request and post whether it held, and where it would break next.

Built With

Share this project:

Updates

Submission history