Drums

Software that heals itself

Comment

Inspiration

I have watched the same on-call loop play out too many times. Something breaks, you get paged, and you spend the next hour digging through logs, metrics, and code before you can even guess what happened. Then you ship a fix and stare at the graph, hoping. The uncomfortable part is that a green graph does not prove the failure is gone. Incident response and chaos engineering are usually separate practices. I wanted one system that turns the incident itself into a repeatable experiment and uses that experiment to prove the repair.

What it does

Drums is self-healing software that proves the heal. A demo checkout API calls a pricing dependency with no request timeout. One button injects 8 seconds of real latency into the pricing service. Nothing is faked: real traffic degrades, and Drums takes over. It detects the SLO breach from live measurements, collects evidence, has Gemini form a root cause hypothesis, turns that hypothesis into a chaos experiment, reproduces the failure in an isolated sandbox, has Gemini write the smallest safe repair, boots the candidate, runs tests, and then replays the exact same experiment against the repaired code. In my validation runs the same 8 second fault took p95 latency from 8.1 seconds (SLO fail) to 1.5 seconds (SLO pass). Only then does the incident become VERIFIED, and a human approves or rejects the change. The core rule: the agent never grades its own work. VERIFIED is computed from measurements, and the state machine refuses it otherwise.

How I built it

Python and FastAPI end to end. Google ADK orchestrates three Gemini 3.5 Flash agents (investigator, reproduction planner, repair engineer) through Vertex AI, each with versioned prompts and strict Pydantic output schemas. The verifier is deliberately not a model: it is deterministic code that replays the experiment and measures. The chaos engine validates every experiment against a target allowlist and hard blast radius limits. Everything runs on Google Cloud: three Cloud Run services, Pub/Sub carrying incident events so the browser never holds the workflow, Firestore for durable incident state, Secret Manager for tokens, Cloud Build and Artifact Registry for the image pipeline. 29 automated tests cover the verification invariant, including proof that a broken repair and a bare agent claim can never reach VERIFIED.

Challenges I ran into

Keeping the demo honest was the hard part. The trigger button does not create an incident record; it breaks a real dependency, and detection has to notice from real traffic. Real bugs surfaced along the way: Gemini once returned a repair that did not parse, so I added a compile check and one retry with the error fed back; the sandbox inherited the production chaos token and locked itself out; the workflow originally ran on the event loop the agents needed for themselves; and a trailing newline in a Secret Manager secret silently broke token auth. Each one became a fix plus a regression test.

Accomplishments that I'm proud of

The verification invariant. VERIFIED requires five things at once: the failure reproduced, the code actually changed, the identical experiment id replayed against the candidate, the SLO passed, and tests passed. There are tests proving it cannot be bypassed. I am also proud that the whole loop runs unattended on real infrastructure in about two minutes, and that a reviewer can trigger it themselves from one URL.

What I learned

Deterministic verification changes how much autonomy you can safely give an agent. Once the model cannot declare success, you can let it investigate, plan, and write code freely, because the only path to VERIFIED runs through real measurements. I also learned a lot about running sandboxed workloads inside Cloud Run and about the sharp edges of event-driven workflows (idempotency, redelivery, and detection dedupe are where the real bugs live).

What's next for Drums

More failure families: bad deploys, error storms, resource exhaustion. Past incident memory feeding investigation so Drums gets better with every incident. And a real rollout pipeline behind the approval boundary, so an approved repair becomes a pull request and a deploy. Also, better session replay understanding and understanding what software users want and their resentment, and improve fixes (RL).

Built With

  • artifact-registry
  • cloud-logging
  • docker
  • fastapi
  • gemini
  • github
  • github-actions
  • google-adk
  • google-cloud
  • google-cloud-firestore
  • google-cloud-pubsub
  • google-cloud-run
  • google-secret-manager
  • httpx
  • pydantic
  • pytest
  • python
  • uvicorn
  • vertex-ai
Share this project:

Updates