AURA (Advanced Unified Response Assistant)

An AI-powered Crisis Management & Emotional Intelligence Coach that uses real-time voice synthesis and multimodal reasoning to de-escalate high-stakes conflicts and monitor agent wellbeing.

Comment

Inspiration

In high-pressure environments like crisis hotlines, emergency dispatch, and hostage negotiation, the difference between a positive outcome and a tragedy often comes down to tone and timing. Human operators face immense burnout, while traditional AI bots lack the emotional depth required for these moments. We were inspired to build AURA to act as a "Guardian Angel" for responder, providing an empathetic voice for the caller and a real-time "Emotional Dashboard" for the supervisor to ensure no one is left alone in the dark.

What it does

AURA is a dual-purpose system:

  1. The Voice: Using ElevenLabs Conversational AI, AURA provides a steady, human-like voice that adapts its pacing and stability based on the caller's distress levels.
  2. The Brain: Gemini 1.5 Pro sits in the middle of the stream, performing real-time sentiment analysis and "Crisis Logic." It suggests de-escalation tactics to the responder and alerts supervisors if a situation turns critical.
  3. The Shield: Datadog monitors the entire interaction. It tracks "Frustration Metrics," latency, and safety violations (e.g., self-harm detection), triggering immediate "Actionable Items" if the AI or the caller reaches a dangerous state.

How we built it

  • Vertex AI & Gemini: We leveraged Gemini 1.5 Pro’s multimodal capabilities to analyze live audio streams. This allowed us to calculate an "Agitation Score" $S_a$ based on volume, pitch, and keywords.
  • ElevenLabs: We configured a custom Conversational Agent with specialized "Voice Design." We used the API to dynamically adjust stability and clarity.Datadog: We instrumented the app using the Datadog SDK. Telemetry is streamed to a custom Crisis Command Center dashboard. We defined detection rules that trigger "Cases" when the sentiment score $S_s < -0.8$ or when latency exceeds $200ms$.
  • Google Cloud: The entire backend is hosted on Cloud Run, ensuring we can scale instantly during a mass-emergency event.

Challenges we ran into

The biggest hurdle was Latency vs. Empathy. In a crisis, a 2-second delay feels like an eternity. We had to optimize the bi-directional stream between ElevenLabs and Gemini to ensure the voice response felt "interruptible" and natural. We also struggled with "Hallucination Guardrails" ensuring the AI never gives incorrect medical or legal advice which we solved by using Datadog monitors to flag and kill any response that didn't meet our "Confidence Threshold."

Accomplishments that we're proud of

We successfully built a system where the AI's voice actually changes based on the caller's state. If the caller starts whispering, AURA lowers its volume and increases its "Stability" to provide a grounding effect. Seeing the Datadog Alert trigger an incident report in real-time when the AI detected a "High-Risk" keyword was a "Eureka" moment for the team.

What we learned

We learned that "Observability" isn't just for servers; it’s for emotions. By treating "User Frustration" as a metric in Datadog, we realized we could debug human interactions just as easily as we debug code.

What's next for AURA (Advanced Unified Response Assistant)

We plan to integrate with Smart Wearables (like Apple Watch or Fitbit) to pull the responder's actual heart rate into the Datadog dashboard. If a responder's heart rate spikes, AURA will automatically offer to take over the call or suggest a mandatory break.

Built With

Share this project:

Updates

Submission history