Nudge: Real-Time Social Assistance

Inspiration

We’ve all been in social situations where we struggle to communicate effectively—whether it’s rambling too much on a first date, getting overwhelmed in a heated family discussion, or feeling too anxious to approach someone at a networking event. While preparation helps, real-life interactions are unpredictable.
What if there were a way to receive real-time assistance to navigate these moments smoothly?

What It Does

Nudge is an opt-in AI-powered social assistant that listens to conversations, analyzes context, and provides real-time nudges through subtle, non-intrusive feedback. By analyzing speech patterns, pace, and tone, Nudge helps users adjust their communication in the moment—without disrupting the flow of conversation.

How We Built It

We leveraged Azure’s real-time speech-to-text API to transcribe speech, identify speakers (using Speaker Diarization), and analyze latencies. This enables us to detect when users are rambling, stuttering, or speaking too fast. To ensure nudges are delivered only when necessary, we implemented confidence scoring and thresholding mechanisms. For generating the actual nudges, we use OpenAI models.

For delivering feedback (via text-to-speech), we integrated ElevenLabs to personalize how users receive nudges. Users can customize how feedback is presented—some may prefer a gentler approach, while others might want direct feedback. Additionally, for a more comfortable and engaging experience, users can opt for lighthearted nudges from a familiar character like Michael Scott.

Challenges We Ran Into

• Most existing models still struggle with accurate speaker diarization for text-to-speech applications.
  
• Balancing precision and recall for nudges to ensure an optimal user experience required multiple iterations.
  

Accomplishments That We're Proud Of

• Achieving near real-time speech analysis with minimal latency.
  
• Developing nudges that are genuinely helpful—almost helping us avoid a fight at 3 AM! 😆
  

What We Learned

• Neither of us were deeply familiar with the complexities of audio processing, especially the challenges of making things work in near real-time.
  
• The trade-offs between accuracy, latency, and user experience are crucial when designing real-time feedback systems.
  

What’s Next for Nudge

• More Seamless Integrations: We’re particularly excited about exploring haptic feedback, similar to how the Apple Watch provides subtle vibrations for navigation.
  

Built With

• azure
• eleven-labs
• javascript
• lovable
• openai
• python