Freddy Voice copilot

Freddy Voice Copilot is a real-time AI agent assist system that provides live guidance, knowledge retrieval, compliance monitoring automated quality assurance inside the Freshcaller workspace.

Comment

Inspiration

Freshworks' Freddy AI already assists agents across chat and email, but live phone calls remain largely unassisted. Agents still need to manually search for knowledge, customer history, and best practices while managers have limited visibility into call quality. We saw an opportunity to bring real-time AI assistance directly into the Freshcaller workflow.

What it does

Freddy Voice Copilot provides real-time AI assistance throughout a customer call:

  • Live speech transcription with speaker identification
  • Real-time customer intent detection
  • Automatic Freshdesk knowledge-base recommendations
  • Customer and cross-channel context
  • Real-time compliance alerts
  • Next-best-action suggestions and agent coaching
  • Live call quality scoring
  • Automated post-call QA scorecard
  • Automatic support ticket creation and routing

How we built it

We built Freddy Voice Copilot using a multi-agent orchestration architecture.

  • Frontend: React + Vite
  • Backend: Node.js + Express + WebSocket
  • Speech: Web Speech API
  • AI Agents: Intent Detection, Knowledge Retrieval, Context, Compliance, Coaching, and Quality Scoring
  • Integration: Freshdesk/Freshworks-style KB, CRM, and customer data WebSockets enable real-time communication between the agent interface and the AI orchestration layer.

Challenges we ran into

  • Making browser-based speech recognition reliable during continuous calls
  • Managing real-time WebSocket and React state without stale data
  • Coordinating multiple specialized AI agents
  • Demonstrating cross-channel customer context
  • Creating meaningful, transcript-backed QA scores
  • Building a convincing prototype without production Freshworks API access

Accomplishments that we're proud of

  • Built a functional end-to-end prototype rather than a static mockup
  • Implemented real-time transcription, AI guidance, compliance monitoring, and quality scoring
  • Demonstrated cross-channel Freshdesk context during live calls
  • Automated post-call QA and ticket creation
  • Designed the experience to feel like a native Freshworks capability

What we learned

  • Voice remains a major gap in AI-assisted customer support
  • Native access to customer and knowledge data makes AI assistance significantly more useful
  • Real-time intervention is more valuable than discovering issues after a call
  • Multi-agent architectures work well for complex, parallel support workflows
  • Customer research helps turn product assumptions into meaningful features

What's next for Freddy Voice Copilot

Our next steps are to move from prototype to production by:

  • Integrating ElevenLabs for production-grade transcription and speaker diarization
  • Connecting with Freshworks Agent Studio and MCP
  • Replacing mock data with live Freshdesk and CRM APIs
  • Deploying through the Freshworks Marketplace
  • Adding sentiment analysis, multilingual support, predictive churn detection, and advanced manager analytics

Our vision is to make every Freshcaller agent AI-assisted throughout the entire customer conversation.

Share this project:

Updates