What it does
Production incidents are complex and time-critical. The traditional method of incident response requires on-call engineers to manually dig through logs, metrics, and code changes while under pressure.
The On-Call Assistant provides intelligent, automated incident analysis using a swarm of specialized AI agents that collaboratively investigate production issues, identify root causes, and provide actionable recommendations within minutes.
How we built it
We built a multi-agent system using the Strands framework that orchestrates specialized agents to analyze different aspects of an incident:
- Incident Search Agent finds and retrieves incident details from our production system
- Log & Metrics Agent analyzes telemetry data, logs, and performance metrics for anomalies
- Code Retrieval Agent examines recent deployments and code changes that might have caused issues
- Orchestrator Agent coordinates the swarm and synthesizes findings into comprehensive incident reports
We use MongoDB with Vector Search as a persistent shared memory system:
- Agents store and retrieve analysis findings using semantic search
- OpenAI embeddings enable intelligent information retrieval
- Session-based memory isolation keeps incident analyses separate
The system is powered by Amazon Bedrock using the Nova Pro model for agent intelligence.
We also built a FastAPI server to simulate a production incident management system with realistic scenarios for an AI voice platform.
Challenges we ran into
- Agent Coordination: Ensuring agents work efficiently in parallel without duplicating efforts or missing critical information
- Memory Management: Designing a shared memory system that allows agents to collaborate while maintaining context isolation between different incident analyses
- Real-time Analysis: Processing large volumes of logs and metrics quickly enough to provide timely incident response
- Context Window Limitations: Managing the amount of information each agent processes while maintaining comprehensive analysis
Accomplishments we're proud of
From a technical perspective:
- Built a truly collaborative multi-agent system where specialized agents work together seamlessly
- Implemented semantic search in MongoDB for intelligent information retrieval across agent memories
- Created realistic incident scenarios that mirror real production issues in AI/ML systems
From a product perspective, the On-Call Assistant:
- Reduces incident response time from hours to minutes
- Provides consistent, thorough analysis that doesn't degrade under pressure
- Identifies patterns and correlations that human operators might miss
- Generates actionable recommendations with clear next steps
What we learned
- Swarm Architecture is Powerful: Breaking down complex analysis into specialized agents makes the system more maintainable and accurate
- Shared Memory is Crucial: Agents need a persistent way to share findings and build on each other's work
- Simulation Helps Development: Building a realistic incident API helped us test edge cases and improve agent prompts
- Structured Output Matters: Well-formatted incident reports with clear sections make the analysis immediately actionable
What's next for On-Call Assistant
- Integration with Real Monitoring Systems: Connect to actual production monitoring tools like Datadog, CloudWatch, and PagerDuty
- Custom Agent Creation: Allow teams to create specialized agents for their specific tech stack
- Automated Remediation: Enable agents to execute safe remediation actions with approval workflows
- Learning from Past Incidents: Build a knowledge base from resolved incidents to improve future analysis
- Voice Interface: Add voice interaction for hands-free incident response during critical situations
Tech Stack
- Strands: Agent orchestration framework
- MongoDB: Persistent shared memory with vector search
- Amazon Bedrock: LLM capabilities via Nova Pro model
- FastAPI: REST API for incident data simulation
- OpenAI API: Text embeddings for semantic search
Log in or sign up for Devpost to join the conversation.