EchoOps - "The System That Speaks"

EchoOps is an ambient computing interface for DevOps that allows engineers to *hear* the root cause of an incident and *speak* the remediation.

Comment

Inspiration

DevOps "War Rooms" are chaotic. Engineers are drowning in dashboards, trying to correlate graphs while alerts scream. We asked: What if the monitoring system didn't just show you data, but talked to you? We wanted to move from passive screens to an ambient, active voice interface that acts as another engineer in the room.

What it does

EchoOps is an ambient computing interface for Datadog.

  • Listens to Datadog webhooks (alerts).
  • Analyzes the root cause using Gemini Flash Lite.
  • Speaks a concise "Situation Report" (SitRep) using ElevenLabs (sub-second latency).
  • Acts on voice commands to mitigate issues (e.g., "Echo, scale the payment service").

How we built it

  • Core: Python (FastAPI) on Google Cloud Run.
  • Brain: Vertex AI (Gemini 2.5 Flash Lite) for rapid root cause analysis and intent classification.
  • Voice: ElevenLabs Turbo v2 for ultra-low latency text-to-speech.
  • Observability: Datadog for everything. We used ddtrace to trace every AI thought process and Custom Widgets to embed the audio player directly into the dashboard.

Challenges we ran into

  • Latency: Voice interfaces must feel instant. We optimized by switching to Gemini Flash Lite and ElevenLabs Turbo, cutting response times from 3s to <1s.
  • Hallucinations: The AI initially tried to "invent" runbooks. We used strict prompt engineering and Datadog Log context to ground its responses.
  • Dashboard Integration: Making Datadog "speak" required building a custom iframe widget served by our backend, bridging static dashboards with dynamic media.

Accomplishments that we're proud of

  • Built a fully "talking dashboard" that integrates seamlessly into existing Datadog workflows.
  • Achieved real-time performance suitable for high-stress incidents.
  • Implemented Safety Monitors (Refusal & Sentiment detection) to observe the AI itself.

What we learned

  • Audio is a high-bandwidth channel: A 5-second spoken summary is often faster than parsing 10 complex graphs.
  • AI needs Observability: You can't trust an agent without monitoring its "thoughts" (tokens) and "feelings" (sentiment/refusals).

What's next for EchoOps - "The System That Speaks"

  • Two-Way Audio: Real-time conversation mode (WebSockets) instead of asynchronous commands.
  • Multi-Modal Analysis: Letting Gemini "see" the Datadog graphs (Screenshots) for better context.
  • Slack/PagerDuty Integration: Sending voice notes directly to incident channels.

Built With

Share this project:

Updates