Clarity Agent

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Clarity Agent

Clarity Agent was inspired by watching real IT teams lose hours in “war rooms,” manually combing through Kubernetes logs, application traces, and cloud events just to answer a simple question: “What actually went wrong?” Downtime is expensive, but the real hidden cost is engineer time and the stress of firefighting instead of building. Clarity Agent was built to flip this pattern—from reactive scrambling to proactive, AI-augmented operations.

The project is a multi-agent AI system with three core roles: the Analyst, the Sentinel, and the Co‑Pilot. The Analyst Agent ingests heterogeneous logs, normalizes them into a unified timeline using Pandas, and sends a strict, schema-driven prompt to an LLM on AWS Bedrock (Amazon Titan) for structured root cause analysis. The Sentinel Agent continuously monitors logs asynchronously, detects anomalous error trends based on configurable thresholds, and generates professional proactive alerts via the same LLM layer. The Co‑Pilot Agent keeps both the parsed timeline and prior analysis in context so that operators can ask natural-language follow‑up questions like “Which services were impacted?” and receive precise, context-aware answers instead of raw log dumps.

The system was implemented with a clean separation of concerns: a Typer-based CLI orchestrator for commands like analyze, monitor, and ticket, an AI service layer that talks to Bedrock via boto3, and an MCP-compliant tooling layer exposed over FastAPI for safe remediation commands (e.g., generating kubectl rollout restart or kubectl rollout undo calls). All data exchanged between components is validated with Pydantic models such as LogEvent, RootCause, ProactiveAlert, and RemediationCommand, which made debugging significantly easier and ensured that LLM outputs were always parsed into well-defined structures. The architecture is fully asynchronous using asyncio, so long-running requests to external services never block the main event loop, which is critical for a responsive operator experience.

Building Clarity Agent involved several technical and product challenges. Designing robust prompts and JSON schemas for AWS Bedrock so that the LLM reliably returned parseable, structured output required multiple iterations, especially under noisy real-world logs. Handling failure modes was equally important: a resilient fallback path was implemented for cases where the AI output is invalid, ensuring the CLI still produces a meaningful report and never leaves the user stuck. On the monitoring side, tuning the trend detection threshold to balance real alerts with noise, while keeping the logic interpretable for judges and future users, required careful experimentation.

Another major learning was how to integrate AI safely with operational tooling. Instead of letting the model run arbitrary shell commands, the system routes all remediation through a constrained MCP server that exposes only specific endpoints like /tools/restart and /tools/rollback, each accepting a narrow JSON payload and returning a suggested command for human review. This enforced a human-in-the-loop model where AI proposes, but engineers approve and execute, reducing risk while still achieving a dramatic reduction in mean time to resolution (MTTR). Overall, the project taught valuable lessons in multi-agent design, prompt engineering, async Python, and how to turn raw logs into actionable, trustworthy automation for real incident response workflows.

Built With

• amazon
• amazon-web-services
• asyncio
• bedrock
• boto3
• cli)
• concepts
• context
• fastapi
• mcp
• mcp)
• model
• pandas
• protocol
• pydantic
• python
• rich
• server)
• terminal
• titan)
• typer
• ui)