Agentic Chaos Monkey

Break it safely. Diagnose it intelligently. Recover with guardrails.

Inspiration

Production systems rarely fail in predictable ways. A deployment can introduce an incorrect configuration, a service can become unavailable, latency can spike, or a Kubernetes pod can enter a crash loop.

Traditional chaos-engineering tools are excellent at injecting controlled failures, but the work often continues after the experiment begins. Engineers still need to investigate dashboards, search logs, identify the root cause, choose a runbook, perform the recovery, and confirm that the service is healthy.

This inspired us to build Agentic Chaos Monkey: a platform that tests whether an AI-assisted reliability system can safely detect, diagnose, remediate, and verify an incident.

Our goal was to bring chaos engineering, observability, and agentic AI together in a closed-loop resilience-testing platform.

What It Does

Agentic Chaos Monkey introduces a controlled failure into an application and activates a reliability workflow powered by Google's Agent Development Kit (ADK).

The platform can:

Inject a controlled failure.
Detect abnormal system behavior.
Collect logs, metrics, Kubernetes events, and deployment history.
Correlate the evidence into a unified incident timeline.
Identify the most likely root cause and report its confidence level.
Select an approved remediation runbook.
Request human approval for higher-risk actions.
Execute the approved recovery action.
Verify that the application has returned to a healthy state.
Record an auditable incident and recovery timeline.

Unlike a monitoring assistant that stops after making a recommendation, Agentic Chaos Monkey closes the loop by checking whether the remediation actually restored service health.

How We Built It

We designed the project around a controlled Kubernetes environment that hosts a sample application and supports repeatable failure experiments.

The platform has four main layers:

Chaos injection
Observability and evidence collection
Specialized Google ADK agents
Policy-governed remediation and verification

Chaos Injection

A chaos controller can introduce controlled failures such as:

Terminating an application pod
Deploying an unhealthy application version
Adding artificial latency
Triggering readiness-probe failures
Applying an incorrect environment configuration
Reducing the number of available replicas

Each experiment is bounded by predefined safety policies so that its scope is controlled and its effects are reversible.

Observability and Evidence Collection

The platform gathers health and performance signals from the application and Kubernetes environment, including:

Application error rate
Request latency
Pod health and restart counts
Kubernetes events
Readiness- and liveness-probe failures
Recent deployment changes
Structured application logs

The agents organize this evidence on a shared timeline to determine what changed, when the incident started, and which events are likely related.

Google ADK Agents

Google ADK coordinates specialized agents for different stages of the incident lifecycle.

Investigation Agent

The Investigation Agent gathers telemetry and deployment information, correlates the signals, and produces a probable root cause supported by evidence. It also reports confidence and alternative explanations when the evidence is incomplete.

Policy Agent

The Policy Agent evaluates the proposed remediation against predefined rules. It determines whether the action can run automatically, requires human approval, or must be blocked.

Remediation Agent

The Remediation Agent executes an approved runbook, such as restarting an unhealthy pod, scaling a deployment within configured limits, or rolling back a faulty release.

Verification Agent

The Verification Agent checks whether error rate, latency, pod health, replica availability, and application responsiveness have returned to acceptable levels.

Separating these responsibilities prevents a single agent from having unrestricted control over the entire environment.

Policy-Governed Remediation

Every recovery action is assigned a risk level and an explicit execution policy.

Action	Risk level	Policy
Collect logs and metrics	Low	Run automatically
Restart one unhealthy pod	Low	Run automatically within limits
Scale a deployment	Medium	Require approval beyond the configured limit
Roll back a deployment	Medium	Require human approval
Change database configuration	High	Recommend only
Delete infrastructure	Prohibited	Always block

Agents select from approved tools and parameterized runbooks instead of generating and executing arbitrary shell commands.

Recovery Verification

A successful command does not necessarily mean that an incident has been resolved. Kubernetes may accept a rollback while the application remains unavailable.

After every remediation, Agentic Chaos Monkey independently checks that:

Pods are ready.
Error rates have decreased.
Latency has returned to the expected range.
The health endpoint is responding.
The expected number of replicas is available.
The service remains stable throughout the verification window.

If recovery fails, the platform records the outcome and recommends escalation or another approved action.

Demo Scenario

Our primary demonstration uses a faulty Kubernetes deployment:

A healthy application is running as version v1.
Agentic Chaos Monkey deploys a faulty version, v2.
The new version contains an incorrect configuration.
Readiness checks begin to fail, while errors and latency increase.
The Investigation Agent collects logs, metrics, Kubernetes events, and deployment history.
It determines that the incident began immediately after the v2 deployment and presents the supporting evidence.
It recommends rolling the application back to v1.
The Policy Agent classifies the rollback as a medium-risk action and requests human approval.
After approval, the Remediation Agent executes the rollback.
The Verification Agent confirms that the pods are ready, the application is responsive, and the error rate has returned to normal.

The resulting timeline shows the complete journey from failure injection to verified recovery.

Challenges We Faced

Preventing Unsafe Agent Actions

One of our biggest challenges was giving an AI agent enough capability to help resolve an incident without granting unrestricted access to the environment.

We addressed this with narrowly scoped tools, approved runbooks, risk classifications, execution limits, and human approval checkpoints. The agents can reason about an incident, but they can execute only the actions permitted by policy.

Correlating Multiple Observability Signals

A single log message or metric rarely explains an entire incident. The system must connect changes in error rate, Kubernetes events, application logs, and recent deployments.

We organized the evidence around a common incident timeline, making it easier to determine whether a deployment or infrastructure event occurred immediately before the application became unhealthy.

Verifying Real Recovery

It was tempting to mark an incident as resolved as soon as a remediation command completed successfully. However, a successful API response proves only that the command was acceptedâ€”not that users can access the service again.

We therefore created a separate verification stage that measures the application's actual health after remediation.

Reducing Unsupported Conclusions

Agentic systems can produce confident conclusions even when the available evidence is incomplete. To reduce this risk, every diagnosis includes:

Supporting evidence
A confidence level
Plausible alternative causes
A recommended next step

When confidence is too low, the platform recommends escalation instead of automatically changing the environment.

Keeping the Scope Manageable

Chaos engineering can cover hundreds of failure scenarios. Supporting too many experiments would have made it difficult to deliver a reliable hackathon demonstration.

We focused on a small set of understandable failuresâ€”especially an unhealthy Kubernetes deployment followed by a governed rollbackâ€”so we could demonstrate the complete detection-to-recovery lifecycle.

What We Learned

We learned that autonomous remediation is not only an AI problem. It is also a systems-design, observability, safety, and governance problem.

The quality of a diagnosis depends heavily on the available telemetry. Structured logs, reliable metrics, deployment metadata, and well-defined runbooks make the agents significantly more effective.

We also learned that self-healing must be treated as a complete feedback loop:

Detect Investigate Decide Approve Act Verify Learn

Skipping verification creates false confidence. A system should not claim that it healed an incident until service-level health has actually recovered.

Google ADK helped us divide the workflow among specialized agents instead of relying on one large prompt. This made the solution easier to understand, test, govern, and improve.

Most importantly, we learned that chaos testing can evaluate more than infrastructure. It can also test the reliability of AI agents, tools, policies, runbooks, and automated recovery logic.

Accomplishments We're Proud Of

Agentic Chaos Monkey connects chaos engineering and agentic remediation in one understandable, governed workflow. The project demonstrates:

Controlled failure injection
Evidence-based root-cause analysis
Multi-agent incident response
Policy-governed remediation
Human approval for higher-risk actions
Independent recovery verification
A complete, auditable incident timeline

Rather than simply claiming that AI can operate production systems, the project provides a controlled environment in which that claim can be tested safely.