RALLY: Agentic Reliability & Response Orchestration

An agentic AI reliability engineer that investigates incidents, reasons over evidence, proposes safe fixes, executes approved actions, verifies recovery, and generates an auditable incident report.

Comment

Inspiration

Software systems can fail within seconds, but identifying the root cause and safely recovering from an incident can take engineers much longer. We wanted to build something that could act like a reliability engineer rather than simply being another AI chatbot.

RALLY was inspired by the idea of an AI agent that can investigate incidents, reason over real evidence, recommend a solution, take controlled action with human approval, and verify whether the system actually recovered.

Our goal was to demonstrate a complete closed-loop incident response workflow:

Understand → Investigate → Reason → Plan → Approve → Execute → Verify → Report

What it does

RALLY: Agentic Reliability & Response Orchestration: is an AI-powered incident response system designed to act as a virtual reliability engineer.

When an incident occurs, RALLY investigates the affected system using controlled operational tools. It can analyze:

  • Service health
  • Error rates
  • Logs
  • Recent deployments
  • Dependencies
  • System telemetry

The agent correlates this evidence to identify the most likely cause and creates a structured remediation plan.

For risky actions, RALLY requires explicit human approval before execution.

For example, if a new deployment causes the Checkout Service error rate to increase from 2% to 38%, RALLY can investigate the incident, identify the deployment as the likely cause, recommend a rollback, request approval, execute the rollback, verify recovery, and generate an incident report.

The system operates inside a controlled incident simulator containing services such as Checkout, Payment, Inventory, Authentication, and Database. This allows us to safely reproduce realistic failures such as bad deployments, service crashes, latency spikes, dependency failures, and database issues.

Unlike a traditional chatbot:

Chatbot: Question → AI → Answer

RALLY: Incident → Investigation → Evidence → Reasoning → Plan → Approval → Execution → Verification → Resolution → Report

How we built it

We built RALLY as a modular agentic system with a clear separation between AI reasoning, operational tools, and the simulated infrastructure.

The AI agent is responsible for understanding incidents, investigating evidence, reasoning about possible causes, creating remediation plans, and determining when verification is required.

We created a deterministic incident simulator to reproduce production-like failures without connecting the AI to a real production environment.

RALLY interacts with the simulator through a controlled tool layer that provides capabilities such as:

  • get_service_health()
  • get_recent_logs()
  • get_error_rate()
  • get_recent_deployments()
  • get_dependency_status()
  • rollback_deployment()
  • restart_service()
  • verify_recovery()

The system uses a modern full-stack architecture built with Python, FastAPI, React, TypeScript, PostgreSQL, Docker, and an MCP-based controlled tool layer.

A major part of the architecture is human-in-the-loop execution. The AI can investigate and recommend an action, but potentially risky operations require explicit approval before execution.

After an action is performed, RALLY collects fresh telemetry and verifies whether the system has actually recovered instead of simply assuming that the action succeeded.

Challenges we ran into

One of our biggest challenges was making the AI behave like an operational system rather than a simple chatbot.

An AI can generate a convincing answer even when evidence is incomplete. For incident response, that is not enough. RALLY needs to base its decisions on actual system state and distinguish evidence from assumptions.

Another challenge was designing the boundary between AI reasoning and system execution. We did not want the agent to have unrestricted access to commands or infrastructure, so we designed explicit tools for specific operations and introduced human approval before risky actions.

We also needed the incident simulator to be deterministic and reproducible so that realistic failures could be demonstrated consistently.

Finally, verification was a major challenge. It is easy for an agent to say that a rollback worked, but a reliable system needs to actually check the system after the action and confirm that the incident symptoms have disappeared.

Accomplishments that we're proud of

We are proud that RALLY is designed as a complete closed-loop incident-response system rather than simply an AI assistant.

The system brings together:

Incident Detection → Investigation → Root-Cause Reasoning → Recovery Planning → Human Approval → Controlled Execution → Verification → Reporting

We are particularly proud of the controlled execution model. Instead of allowing the AI to execute arbitrary commands, RALLY operates through explicitly defined tools with clear boundaries.

We are also proud of the human-in-the-loop approval mechanism, which keeps humans in control of consequential actions.

Another important accomplishment is the deterministic incident simulator, which allows us to reproduce realistic distributed-system failures safely and demonstrate the complete workflow.

Most importantly, we built RALLY around a principle that we believe is essential for reliable agentic AI:

An AI agent should not only decide what to do — it should also verify whether what it did actually worked.

What we learned

Building RALLY taught us that creating an effective agentic system is much more than connecting an LLM to a few tools.

We learned that tool design is extremely important. Clearly defined tools, structured inputs and outputs, and strict execution boundaries make an agent more predictable and reliable.

We also learned the importance of separating:

  • Observation
  • Reasoning
  • Planning
  • Execution
  • Verification

Keeping these stages separate makes the system easier to test, debug, and audit.

Another important lesson was that human approval is not a weakness of agentic AI. In high-impact environments, it can be an important safety mechanism that allows AI to automate investigation and reasoning while keeping humans in control of consequential actions.

Finally, we learned that automation should be measured by outcomes rather than generated responses. RALLY therefore verifies system recovery instead of simply claiming that a remediation worked.

What's next for RALLY — Agentic Reliability & Response Orchestration

Our current implementation focuses on a controlled environment where we can safely demonstrate the complete incident-response lifecycle.

Our next goal is to move RALLY closer to real-world reliability engineering environments by integrating with observability and infrastructure platforms such as:

  • Prometheus
  • Grafana
  • Kubernetes
  • Docker
  • Cloud monitoring platforms
  • Incident management systems

We also plan to expand RALLY's reasoning capabilities so it can correlate larger amounts of telemetry and investigate more complex multi-service incidents.

Future capabilities could include:

  • Automatic incident severity classification
  • Multi-agent investigation
  • Dependency graph reasoning
  • Root-cause confidence scoring
  • Advanced rollback strategies
  • Canary deployment analysis
  • Continuous post-incident learning
  • Automated postmortem generation
  • Long-term incident memory
  • Policy-based approval workflows

Our long-term vision is to make RALLY a reliable AI operations engineer that can help engineering teams detect, investigate, remediate, and learn from production incidents while maintaining strong safety boundaries and human oversight.

Built With

Share this project:

Updates

Submission history