RALLY About the project Modern software systems can fail within seconds, but finding the cause and recovering safely can take engineers much longer. When an incident occurs, engineers usually have to manually check service health, logs, error rates, recent deployments, and dependencies before deciding what to do. Under pressure, this process can be slow, repetitive, and error-prone. RALLY is an agentic AI-powered reliability engineer designed to automate this incident-response workflow while keeping humans in control of critical actions. Instead of simply answering questions, RALLY follows a complete closed-loop workflow: Understand → Investigate → Reason → Plan → Approve → Execute → Verify → Report When an incident occurs, RALLY investigates the system using controlled operational tools. It collects evidence such as service health, logs, error rates, recent deployments, and dependency status. The AI then reasons over this evidence to identify the most likely root cause and proposes a structured remediation plan. For potentially risky actions, RALLY does not execute blindly. It requests explicit human approval before taking action. Once approved, the remediation is executed through a controlled tool layer. RALLY then verifies the system after the action to determine whether the incident has actually been resolved. For example, if a new deployment causes the Checkout Service error rate to increase from 2% to 38%, RALLY can:
- Detect the degraded service.
- Investigate health, logs, metrics, and recent deployments.
- Identify the new deployment as the most likely cause.
- Recommend rolling back the deployment.
- Request human approval.
- Execute the approved rollback.
- Verify that error rate and latency return to healthy levels.
- Generate an auditable incident report. For the hackathon, RALLY operates within a controlled incident simulator that models a distributed system with services such as Checkout, Payment, Inventory, Authentication, and Database. The simulator can reproduce realistic failures such as bad deployments, service crashes, latency spikes, dependency failures, and database issues. RALLY uses a controlled tool interface to separate AI reasoning from actual system operations. This prevents the agent from having unrestricted access to the environment and allows every operational action to be explicitly defined, tracked, and verified. The key difference between RALLY and a traditional AI chatbot is that RALLY does not stop at generating an answer. A chatbot follows: Question → AI → Answer RALLY follows: Incident → Investigation → Evidence → Reasoning → Plan → Human Approval → Execution → Verification → Resolution → Report Our goal is to demonstrate how agentic AI can move beyond conversation and become a reliable, auditable operational system that can reason about failures, take controlled actions, and verify the outcome. Inspiration Software systems can fail at any moment, but responding to those failures is still heavily dependent on humans manually investigating logs, metrics, deployments, and dependencies. During an incident, engineers need to answer several questions quickly:
- What is actually failing?
- Why is it failing?
- What changed recently?
- What should we do about it?
- Is the proposed action safe?
- Did the system actually recover after the fix? We were inspired by the idea of building an AI system that could act more like a reliability engineer rather than another chatbot. Instead of simply telling an engineer what might be wrong, we wanted the AI to investigate the incident, reason over real evidence, create a recovery plan, request human approval for risky actions, execute the approved action through controlled tools, and verify the result. This led us to build RALLY — Agentic Reliability & Response Orchestration. What it does RALLY is an agentic AI-powered incident response system designed to act like a virtual reliability engineer. It follows a complete incident-response loop: Understand → Investigate → Reason → Plan → Approve → Execute → Verify → Report When an incident occurs, RALLY investigates the affected system using controlled operational tools. It can inspect:
- Service health
- Error rates
- Logs
- Recent deployments
- Dependency status
- System telemetry RALLY then correlates the available evidence and determines the most likely cause of the incident. Instead of immediately taking a potentially risky action, it creates a structured remediation plan and requests human approval. After approval, the action is executed through a controlled tool interface. RALLY then independently verifies whether the system has actually recovered. For example, if a new deployment causes the Checkout Service error rate to increase from 2% to 38%, RALLY can investigate the incident, identify the deployment as the likely cause, recommend a rollback, request approval, execute the rollback, verify recovery, and generate an incident report. The system operates on a controlled incident simulator so that realistic production-like failures can be reproduced safely during the demonstration. The result is a closed-loop system that goes beyond AI-generated recommendations and demonstrates: Investigation → Decision → Controlled Action → Verification How I built it We designed RALLY as a modular agentic system with a clear separation between AI reasoning, operational tools, and the simulated infrastructure. AI Agent The agent is responsible for understanding the incident, investigating available evidence, reasoning about possible causes, creating a remediation plan, and deciding when verification is required. Incident Simulator Because giving an AI agent unrestricted access to a real production environment would be unsafe, we built a deterministic incident simulator. The simulator models a distributed system containing services such as:
- Checkout
- Payment
- Inventory
- Authentication
- Database It can reproduce operational failures such as bad deployments, service crashes, high latency, error spikes, dependency failures, and database issues. Controlled Tool Layer RALLY interacts with the simulated environment through explicitly defined operational tools. These tools provide capabilities such as:
- get_service_health()
- get_recent_logs()
- get_error_rate()
- get_recent_deployments()
- get_dependency_status()
- rollback_deployment()
- restart_service()
- verify_recovery() This creates a strict boundary between what the AI can reason about and what it is actually allowed to execute. Human-in-the-Loop Safety Risky remediation actions require explicit human approval before execution. This prevents the agent from blindly modifying the system and makes the workflow more suitable for real-world reliability engineering. Verification After an action is executed, RALLY does not assume that the problem is solved. It collects new telemetry and verifies whether the relevant health signals have returned to acceptable levels. Reporting Finally, RALLY produces an auditable incident report containing the incident, evidence collected, reasoning, recommended action, approval decision, executed action, verification results, and final status. The application is built using a modern full-stack architecture with Python, FastAPI, React, TypeScript, PostgreSQL, Docker, and an MCP-based controlled tool layer. Challenges I ran into The biggest challenge was making the AI behave like an operational system rather than a simple chatbot. A chatbot can generate a convincing answer even when the underlying evidence is incomplete. For incident response, that is not enough. RALLY needs to base its decisions on actual system state and clearly separate evidence from assumptions. Another challenge was designing the boundary between AI reasoning and system execution. We did not want the AI to have unrestricted access to commands or infrastructure. Instead, we designed controlled tools for specific operations and introduced human approval before risky actions. We also had to make the incident simulator deterministic and reproducible. A good hackathon demonstration should produce the same operational conditions every time rather than depending on random failures or unpredictable external systems. Finally, verification was an important challenge. It is easy for an agent to say that a rollback was successful. It is much more meaningful to actually check the system after the action and determine whether the incident symptoms have disappeared. These challenges pushed us to think about RALLY as a reliability system with safety boundaries, state transitions, evidence, and verification rather than simply an AI interface. Accomplishments that I'm proud of We are proud that RALLY is designed as a complete closed-loop incident-response workflow rather than just an AI assistant. The system can move through the entire operational lifecycle: Incident → Investigation → Root-Cause Reasoning → Recovery Plan → Human Approval → Execution → Verification → Report We are particularly proud of the controlled execution model. Instead of allowing the AI to execute arbitrary commands, we created a defined tool layer that limits what actions the agent can perform. We are also proud of the human-in-the-loop approval mechanism. The AI can investigate and recommend a remediation, but a human remains responsible for approving potentially risky actions. Another important accomplishment is the deterministic incident simulator. It allows us to reproduce realistic distributed-system failures and demonstrate the complete workflow safely without relying on a real production environment. Most importantly, we designed RALLY around a principle that we believe is critical for agentic AI: An AI agent should not only decide what to do — it should also verify whether what it did actually worked. What I learned Building RALLY taught us that creating an effective agentic system is much more than connecting an LLM to a few tools. The most important lesson was that tool design matters as much as model intelligence. Clearly defined tools, structured inputs and outputs, and strict execution boundaries make an agent more reliable and predictable. We also learned the importance of separating:
- Observation
- Reasoning
- Planning
- Execution
- Verification Keeping these stages distinct makes the system easier to debug, test, and audit. Another major lesson was that human approval is not necessarily a limitation of agentic AI. In high-impact environments, it can be an important safety mechanism that allows AI to automate investigation and decision-making while keeping humans in control of consequential actions. Finally, we learned that successful automation should be measured by outcomes rather than generated responses. RALLY is therefore designed to verify system recovery instead of simply claiming that a remediation worked. This project changed how we think about AI agents: not as systems that simply generate answers, but as systems that can reason, act within boundaries, and validate their own outcomes. What's next for RALLY Our current implementation focuses on a controlled environment where we can safely demonstrate the complete incident-response lifecycle. The next step is to move RALLY closer to real-world reliability engineering environments. We plan to add integrations with real observability and infrastructure platforms such as:
- Prometheus
- Grafana
- Kubernetes
- Docker
- Cloud monitoring platforms
- Incident management systems We also want to expand the agent's reasoning capabilities so it can correlate larger amounts of telemetry and investigate more complex multi-service incidents. Future versions could support:
- Automatic incident severity classification
- Multi-agent investigation
- Dependency graph reasoning
- Root-cause confidence scoring
- More sophisticated rollback strategies
- Canary deployment analysis
- Continuous post-incident learning
- Automated postmortem generation
- Long-term incident memory
- Policy-based approval workflows Another major direction is adaptive autonomy. Instead of treating every action the same way, RALLY could use risk policies to determine which actions can be automatically executed and which actions always require human approval. Our long-term vision is to make RALLY a reliable AI operations engineer that can help engineering teams detect, investigate, remediate, and learn from production incidents while maintaining strong safety and human oversight.
Built With
- agentic
- agents
- compose
- css
- docker
- fastapi
- llm
- mcp
- postgresql
- pydantic
- python
- react
- sqlalchemy
- tailwind
- typescript
- vite
