CloudSentry

Ambient 3 AM Voice Incident Commander for AWS SREs. Triage Sev-1 outages, run 5-Whys audits, and execute zero-trust rollbacks in 45 seconds via Alexa+ and MCP.

Comment

Inspiration

Every on-call Site Reliability Engineer knows the 3:14 AM nightmare. A critical Sev-1 PagerDuty alarm screams on your nightstand. While an e-commerce checkout service or payment gateway is bleeding thousands of dollars per minute in 502 Bad Gateway errors, the engineer spends 15 to 20 minutes just getting into position: fumbling for glasses, booting a cold laptop, authenticating into corporate VPN, passing multi-factor challenges, and navigating through fragmented AWS Management Consoles.

We asked a simple question: What if your bedside Echo Show was an ambient, zero-trust Site Reliability Engineer?

Instead of opening a laptop at 3 AM, an on-call engineer can wake up, speak to Alexa, receive an autonomous root-cause diagnosis from AWS Bedrock, correlate deployment commits through a blameless 5-Whys audit, and safely authorize an Amazon ECS container rollback through an out-of-band two-factor security gate in under 45 seconds.

What it does

AWS CloudSentry transforms Amazon Alexa+ into an autonomous, ambient incident commander for AWS production environments using the official Model Context Protocol (MCP).

  1. Ambient Sev-1 Wake-Up and Triage: When a CloudWatch alarm triggers, CloudSentry surfaces an active incident summary. Asking "Alexa, what is failing in production?" triggers three MCP tools in sequence: querying CloudWatch alarm metrics via boto3, pulling trailing container stack traces from CloudWatch Logs, and running root-cause analysis through AWS Bedrock (Claude 3.5 Sonnet) to detect cgroup OOMKilled exits (exit code 137).
  2. Amazon Blameless Correction of Errors (COE) Audit: Rather than assigning personal blame, CloudSentry inspects local git log history and correlates the deployment timestamp with commit metadata. It generates an Amazon-standard 5-Whys root-cause audit identifying systemic gaps in CI/CD canary soak testing.
  3. Deterministic N-1 Pre-Flight Telemetry: Before touching production, CloudSentry computes the exact memory and CPU headroom recovery delta when reverting to task revision N-1 (projecting a 58.2 percent memory recovery). It queries AWS STS to synthesize a least-privilege IAM session policy document scoped strictly to ecs:UpdateService for that single ECS task definition.
  4. Zero-Trust Out-of-Band 2FA Rollback: Voice commands alone must never possess unrestricted destructive permissions. When an engineer requests a rollback, CloudSentry refuses immediate mutation and dispatches a fresh cryptographic four-digit one-time PIN out-of-band to the engineer's authenticated Slack channel (#sre-oncall-otp). The PIN is never spoken aloud over ambient audio. Once authorized ("Alexa, confirm rollback 4921"), boto3 updates the ECS service to the N-1 task definition and resets the CloudWatch alarm to OK.
  5. Automated Hotfix Diff and S3 Post-Mortem: Once production is healthy, CloudSentry synthesizes a unified Git diff replacing unbounded memory buffers with chunked streaming and automatically publishes a complete Markdown incident post-mortem directly to Amazon S3.

How we built it

CloudSentry was built as a cloud-native architecture connecting ambient voice interfaces to AWS infrastructure via the Model Context Protocol:

  • Voice Interface (Alexa Skills Kit & Alexa+): An authentic Alexa Skills Kit v1.0 custom skill package (alexa-skill/) exposing an HTTPS webhook endpoint (/api/alexa-webhook) that handles standard ASK request envelopes (LaunchRequest, IntentRequest) and maps conversational intent slots into MCP tool invocations.
  • Model Context Protocol Server (Python & FastAPI): Implements the official MCP 2025-11-25+ Streamable HTTP specification (/mcp) using the official Anthropic Python MCP SDK. Nine registered tools (get_active_alarms, fetch_error_logs, diagnose_incident, execute_rollback, post_slack_notification, propose_code_fix, generate_incident_postmortem, trace_deployment_culprit, simulate_blast_radius) are discoverable via tools/list and executable via tools/call over Streamable HTTP JSON-RPC 2.0 with session ID negotiation.
  • AWS SDK & SigV4 Wire Engine (boto3): All cloud interactions execute through real boto3 clients (cloudwatch, logs, ecs, s3, sts). To enable zero-cost local evaluation without requiring paid AWS credentials or Docker, we engineered an embedded SigV4 HTTP emulator that speaks AWS wire protocols (GraniteServiceVersion20100801 for CloudWatch, Logs_20140328 for CloudWatch Logs, AmazonEC2ContainerServiceV20141113 for ECS, REST-XML for S3, and Query-XML for STS), capturing every AWS4-HMAC-SHA256 signature in a live audit trail.
  • Frontend SRE Console (React & Vite): An interactive dual-display telemetry console featuring an ambient bedside nightstand simulation, real-time revenue bleed ticker, live interactive SVG workflow flowchart tracking every command in order, and a 3D software-rendered Amazon Echo 4th Generation sphere with interactive rotation and dynamic LED light ring states.

Challenges we ran into

  1. Ambient Voice Security and Preventing Accidental Mutations: A common failure mode in voice-activated infrastructure projects is allowing destructive actions via ambient voice. If an ambient speaker mishears room audio or an unauthorized person speaks, production cannot be permitted to mutate. We solved this by enforcing three mandatory guardrails: rollbacks are rejected if the service is not currently in an active CloudWatch ALARM state; rollbacks are strictly constrained to the N-1 task definition; and a cryptographic out-of-band PIN is required through authenticated Slack channels and never spoken aloud by the Echo speaker.
  2. Aligning with Amazon Culture: Early iterations used finger-pointing humor around identifying deployment culprits. We recognized this conflicted with Amazon's core engineering principle of blameless post-mortems and Correction of Errors (COE). We redesigned the commit analysis engine into an authentic 5-Whys systemic audit that analyzes testing gaps and canary deployment thresholds rather than individuals.
  3. MCP Streamable HTTP Mounting in ASGI Frameworks: Mounting low-level Starlette Streamable HTTP session managers inside FastAPI applications caused session lifespan lifecycle mismatches. We resolved this by explicitly bridging the low-level MCP session manager lifespan context into the FastAPI application lifespan, exposing mcp-session-id in CORS headers, and verifying end-to-end compliance against the official Anthropic MCP Inspector.

Accomplishments that we're proud of

  • End-to-End Real Hardware Interoperability: Integrating our custom Alexa Skill with physical Amazon Echo hardware on our desk, letting us speak naturally to an Echo device while our browser HUD and terminal execute real boto3 calls in real time.
  • 100 Percent Verifiable at Zero Cost: Any hackathon judge can clone our open-source repository and run our automated 16-test suite or external MCP client verification script (verify_mcp_client.py) without needing an AWS bill or Docker installation.
  • True Dual-Transport Architecture: The same Python server seamlessly serves both authentic Amazon Alexa Skills Kit v1.0 JSON payloads and the official Model Context Protocol Streamable HTTP standard (/mcp).
  • Measurable FinOps and MTTR Impact: Demonstrating a reduction in Mean Time to Remediate from an industry baseline of 18 minutes down to 42 seconds, saving over 140,000 dollars in downtime costs during high-traffic checkout outages.

What we learned

  • Building for voice requires a completely different design paradigm than graphical dashboards: responses must be concise, spoken numbers must be formatted naturally (saying "ninety-nine point four percent" instead of reading raw floats), and visual displays must present high-density supporting evidence without requiring eye contact.
  • Model Context Protocol (MCP) provides a clean, standardized abstraction boundary between conversational LLM agents and cloud infrastructure SDKs, making tool schemas discoverable and portable across external orchestrators like Amazon Q and Claude Desktop.
  • Designing for blameless engineering culture makes diagnostic tools significantly more valuable because they expose structural CI/CD pipeline improvements rather than superficial finger-pointing.

What's next for AWS CloudSentry

  • AWS Bedrock Guardrails Integration: Expanding prompt-injection protection and automated PII masking on container logs before passing stack traces to diagnostic models.
  • Automated Canary Rollouts with CloudWatch Synthetics: Extending the remediation lifecycle to automatically spin up blue-green canary tasks, verify synthetic browser transaction success for five minutes, and graduate to 100 percent traffic autonomously.
  • Multi-Account AWS Organizations Support: Implementing cross-account IAM role assumption via AWS STS so an enterprise platform team can manage multi-region ECS and EKS clusters across hundreds of AWS member accounts through a single bedside voice interface. ### 1. Amazon Alexa Skills Kit (ASK) and Alexa Developer Console
  • What we used it for: Building the custom interaction model (invocation name: "cloud sentry") and connecting our physical Amazon Echo speaker via an HTTPS webhook.
  • What worked well: The JSON Editor and Build Model pipeline are fast, and the live Test tab makes inspecting input and output JSON envelopes straightforward. Once enabled in Development mode, physical Echo devices linked to the same account picked up the skill seamlessly.
  • What needs work / pain points:
    • Sample utterances strictly forbid numeric digits (e.g., rejecting "s3" with an obscure error instead of automatically normalizing or suggesting "s. three").
    • Multi-locale synchronization: When creating a skill for English (US), a physical Echo device configured for English (India) silently ignores the skill with "I'm not sure how to help with that" rather than surfacing a clear locale mismatch diagnostic in the console.
  • How onboarding felt: Intuitive for custom intents and slot types, but debugging locale-specific hardware routing required searching developer forums.
  • Would you build with it again? Yes.

2. AWS Bedrock (Claude 3.5 Sonnet) and converse API

  • What we used it for: Automated root-cause analysis over raw CloudWatch container logs and unified Git diff patch synthesis.
  • What worked well: Exceptional accuracy in distinguishing infrastructure bottlenecks (cgroup OOMKill 137 vs. PostgreSQL connection pool exhaustion). The converse(toolConfig=...) format maps directly to JSON Schema tool declarations.
  • What needs work / pain points: Multi-call tool pipelines chaining CloudWatch alarms, log queries, and Bedrock inference can take 2.5 to 3.5 seconds on cold starts, requiring careful progress feedback for voice users.
  • How onboarding felt: Modern and developer-friendly.
  • Would you build with it again? Yes, by default for SRE automation.

3. AWS SDK for Python (boto3) and CloudWatch / ECS APIs

  • What we used it for: Executing SigV4-signed API calls for alarm state retrieval, log event filtering, task definition rollbacks, and S3 post-mortem uploads.
  • What worked well: Battle-tested and predictable. The transition to Smithy RPC v2 protocols in modern boto3 is seamless.
  • Would you build with it again? Absolutely.

Built With

Share this project:

Updates

Submission history