Sunoji

It’s like having that one friend who always gets you — in your pocket, on demand, no judgment. Just real talk when you need it. No spying or data storage.

Comment

Inspiration

Mental health support shouldn't require scheduling appointments or waiting for responses. We were inspired by the growing need for accessible, immediate emotional support—a companion that's always there when someone needs to talk, vent, or simply be heard. Traditional text-based chatbots lack the warmth and immediacy of human conversation, while real-time voice companions often miss the emotional nuances that make conversations meaningful.

We envisioned Sunoji as a bridge between technology and empathy—an AI that doesn't just hear words but understands emotions, adapts its responses dynamically, and provides genuine support through the power of voice. The name "Sunoji" combines the warmth of the sun with the Japanese spirit of companionship.

What it does

Sunoji is a low-latency speech-to-speech AI companion that creates natural, emotionally-aware voice conversations. Here's how it works:

Core Features:

  • Real-Time Voice Conversations: Speak naturally and receive immediate voice responses with minimal latency
  • Intent & Tone Analysis: Detects whether you're venting frustration, seeking advice, sharing excitement, or just chatting casually
  • Emotional Intelligence: Analyzes your tone (happy, sad, frustrated, anxious, excited, neutral) and responds with appropriate empathy
  • Dynamic Voice Modulation: Voice characteristics adapt to match your emotional state—calmer and more stable when you're anxious, energetic when you're excited
  • Content Safety Guardrails: Detects sensitive topics like self-harm, abuse, or crisis situations and provides supportive responses with crisis resources
  • Customizable Personas: Different AI personalities with unique speaking styles and voice characteristics
  • Quick Actions: On-demand jokes and inspirational quotes to lift your mood

Real-Time Connectors: The web app features seamless real-time audio streaming with:

  • Voice Activity Detection (VAD): Automatically detects when you start and stop speaking
  • WebRTC Audio Capture: High-quality browser-based audio recording
  • Low-Latency Pipeline: Optimized speech-to-speech processing under 2-3 seconds
  • Base64 Audio Streaming: Efficient audio transmission between frontend and backend
  • FastAPI WebSocket Support: Ready for future real-time bidirectional streaming

How we built it

Architecture:

  • Backend: Python FastAPI server handling the entire voice processing pipeline
  • Speech-to-Text: Google Cloud Speech-to-Text with automatic audio format conversion (WebM to WAV)
  • AI/ML Brain: Google Vertex AI (Gemini 2.0 Flash) for intent/tone analysis and intelligent response selection
  • Text-to-Speech: ElevenLabs API with dynamic voice settings for natural, emotionally-appropriate speech synthesis
  • Frontend: Vite + React with custom hooks for audio recording and playback

Key Technical Components:

  1. Audio Processing Pipeline (services.py):

    • FFmpeg integration for audio format conversion
    • Multi-fallback transcription strategy (LINEAR16, WEBM_OPUS, auto-detection)
    • Real-time audio buffering and streaming

  2. Emotional Intelligence (analyze_intent_and_tone):

    • Semantic NLU analysis beyond keyword matching
    • Context-aware classification into 4 intent categories and 6 tone categories
    • Confidence scoring for response quality

  3. Response Generation (custom_responses.py):

    • LLM-powered phrase selection from curated response database
    • Energy-level matching to user's implicit emotional state
    • Natural acknowledgment phrases that feel human

  4. Safety-First Design (content_safety.py):

    • Pattern-based harmful content detection
    • Crisis resource database with helplines and support contacts
    • Persona-independent safety layer that cannot be bypassed
    • Supportive, non-judgmental crisis responses

  5. Dynamic Voice Settings (compute_voice_settings):

    • Real-time voice parameter adjustment based on user tone
    • Stability and style modulation for emotional resonance
    • Persona-specific voice characteristics

Development Process:

  • Started with core STT → LLM → TTS pipeline
  • Added intent/tone analysis for context awareness
  • Built content safety guardrails with crisis support
  • Implemented dynamic voice modulation for emotional intelligence
  • Created custom response selection system for natural interactions
  • Optimized for low latency with Gemini 2.0 Flash and ElevenLabs Turbo

Challenges we ran into

  1. Audio Format Compatibility: WebM audio from browsers wasn't directly compatible with Google Speech-to-Text. Solved by implementing FFmpeg conversion pipeline with multiple fallback strategies.

  2. Latency Optimization: Initial pipeline took 5-7 seconds. Reduced to 2-3 seconds by:

    • Switching to Gemini 2.0 Flash for faster inference
    • Using ElevenLabs Turbo v2.5 model
    • Optimizing prompt engineering for structured JSON outputs
    • Parallel processing where possible

  3. Intent vs. Tone Ambiguity: Phrases like "I want the competition today" were being misclassified as anxious instead of excited. Resolved by:

    • Enhanced semantic analysis prompts
    • Context-aware classification beyond keywords
    • Examples-based few-shot learning in prompts

  4. Safety Without Sacrificing Warmth: Balancing supportive responses for crisis situations while maintaining the companion's warm personality. Achieved through:

    • Separate safety layer operating independently of personas
    • Carefully crafted supportive messages
    • Integration of crisis resources without being clinical

  5. Response Naturalness: Early LLM responses felt robotic. Fixed by:

    • Creating curated phrase database (customResponses.txt)
    • Using LLM as intelligent router, not generator
    • Energy-level matching for contextual appropriateness

  6. Real-Time Audio Streaming: Coordinating VAD, recording, transmission, and playback required careful state management in React with custom hooks.

Accomplishments that we're proud of

✨ Sub-3-Second Latency: Achieved near-real-time voice conversations with full emotional intelligence

🧠 Sophisticated Emotional Understanding: Goes beyond keyword detection to understand semantic meaning and context

🛡️ Robust Safety Guardrails: Comprehensive content safety system with crisis resources that operates independently of any persona configuration

🎭 Dynamic Voice Synthesis: Voice characteristics that adapt in real-time to match user's emotional state

🏗️ Production-Ready Architecture: Clean separation of concerns with modular services, proper error handling, and comprehensive logging

💬 Natural Conversations: Response selection system that feels genuinely human and contextually appropriate

🔄 Resilient Audio Processing: Multiple fallback strategies ensure transcription works across different audio formats and quality levels

📱 Modern Web Experience: Sleek Vite + React frontend with intuitive voice interaction

What we learned

Technical Insights:

  • Audio processing in web applications requires robust fallback mechanisms—browser implementations vary widely
  • Latency in conversational AI comes from multiple sources; optimizing each component is critical
  • Semantic NLU analysis is far superior to keyword matching for understanding user intent
  • Voice synthesis parameters have profound impact on perceived empathy and naturalness
  • Safety systems must operate independently of customization features to be truly effective

AI/ML Learnings:

  • Gemini 2.0 Flash provides excellent balance of speed and intelligence for real-time applications
  • Structured JSON outputs with strict validation prevent cascading errors in production
  • LLMs work better as routers/selectors than generators for maintaining consistent tone
  • Few-shot examples in prompts dramatically improve classification accuracy

Design Principles:

  • Voice interfaces require different UX patterns than text—immediate feedback and clear state indication are crucial
  • Emotional intelligence in AI requires modeling both detection AND response
  • Safety features should enhance, not interrupt, the user experience
  • Real-time systems need comprehensive error handling at every layer

Product Development:

  • Mental health features require careful consideration of edge cases and crisis scenarios
  • User tone and intent can contradict—semantic context is key
  • Low latency creates psychological "flow" that makes AI feel more human
  • Warmth and professionalism can coexist in supportive technology

What's next for Sunoji

Near-Term Enhancements

  • WebSocket Streaming: Implement true bidirectional streaming for even lower latency and interruption support
  • Voice Activity Detection Improvements: Better silence detection and natural conversation flow
  • Multilingual Support: Expand beyond English to support global accessibility
  • Conversation Memory: Context retention across sessions for deeper relationships
  • Mobile Apps: Native iOS and Android apps for on-the-go support

Advanced Features

  • Emotion Detection from Voice: Analyze acoustic features (pitch, tempo, energy) alongside words
  • Personalized Voice Matching: Learn user preferences and adapt voice characteristics over time
  • Contextual Awareness: Integration with calendar, weather, and location for more relevant responses
  • Group Conversations: Support for multi-user therapy or support group scenarios
  • Analytics Dashboard: Privacy-respecting insights into emotional patterns over time

Real-Time Connector Expansion

  • Live Sentiment Visualization: Real-time graphs showing emotional journey during conversations
  • Crisis Detection Streaming: Instant intervention when concerning patterns emerge
  • Therapist Integration: Optional secure connection to human professionals for escalation
  • Voice Biometrics: Optional authentication and personalization through voice recognition
  • Smart Home Integration: Hands-free access through Alexa, Google Home, etc.

Research & Innovation

  • Adaptive Learning: Continuously improve responses based on user feedback and conversation outcomes
  • Therapeutic Techniques: Integrate CBT, DBT, and mindfulness practices into conversations
  • Cultural Sensitivity: Context-aware responses that respect diverse cultural backgrounds
  • Accessibility Features: Support for users with speech impediments, hearing loss, or cognitive differences

Community & Scale

  • Open Response Database: Community-contributed phrases for more natural conversations
  • API for Developers: Enable integration with mental health apps and platforms
  • Research Partnerships: Collaborate with mental health organizations for validation and improvement
  • Privacy-First Infrastructure: End-to-end encryption and zero-knowledge architecture

Sunoji represents just the beginning of empathetic AI companions. Our vision is a world where everyone has access to immediate, judgment-free emotional support—wherever they are, whenever they need it.

Built With

Share this project:

Updates