LiveAssist

Real Time CS Agent Co-Pilot

Comment

Live Assist: Real-Time AI Assistant for Customer Service

Introduction

Customer service isn’t just about picking up calls anymore. It’s about context, speed, and having the right information right now. Enter Live Assist: a desktop-powered AI system that delivers real-time call insights, CRM integration, and document-aware responses—all packed into a sleek, serverless AWS stack.

Here’s how it works in action:

  • A customer reports a dashboard timeout.
  • The agent instantly logs a support ticket.
  • The customer updates their contact number—Live Assist updates it in HubSpot.
  • The customer asks about upgrade options—Live Assist fetches relevant deals.
  • Curious about known issues? The agent asks Kendra to search company manuals.
  • The customer requests an email follow-up—Live Assist prepares the email for agent confirmation.

This is not a chatbot. It’s an agent co-pilot that listens, learns, and acts alongside your team in real time.

Built for seamless integration with Electron-based desktop apps, Live Assist ensures agents stay focused while the AI does the heavy lifting.

🛠 How we built it

Frontend

  • Built with Electron + React + TypeScript
  • Captures mic and system audio
  • Streams audio to AWS Transcribe WebSocket
  • Sends 10s transcript windows with call metadata (UUID, window number) to the backend

Real-Time Backend

  • API Gateway + Lambda receive transcript windows
  • Lambda fetches previous windows from DynamoDB, assembles context
  • Passes input to LangChain Orchestrator using Claude via Bedrock

  • LangChain agent invokes tools:

    • Amazon Kendra: Searches company docs (vectorized via S3)
    • HubSpot CRM: Handles contact lookup, ticket creation, deal info
    • SES: Email requests routed through a confirmation Lambda

  • Outputs:

    • aiTips: Tagged insights (urgent, suggestion, info)
    • activityFeed: Executed tools

  • Stores everything in call-records DynamoDB table

Post-Call Backend

  • Aggregates window data by call_id
  • Triggers post-call Lambda for analytics

  • Outputs:

    • Sentiment score
    • Issue resolution
    • Agent performance
    • Memory box (deliverables + coaching)

  • Stored in call-analytics table

Core AWS Services Used

  • Transcribe – WebSocket real-time transcription with speaker labels
  • Bedrock – Claude LLM for all AI generation
  • Kendra – Intelligent document search (using S3 as data source)
  • DynamoDB – Stores transcripts, tips, and post-call insights
  • S3 – Company manuals for Kendra ingestion
  • API Gateway – Secure endpoints for real-time and post-call APIs
  • SES – Verified sender email service (triggered via frontend-confirmed Lambda)
  • ECR + Docker – Custom Lambda image runtime with extra packages
  • Lambda – Serverless compute that powers call analytics pipeline.

Agent Intelligence (LangChain-Powered)

The core of Live Assist’s reasoning lives in a LangChain-based agent that:

  • Receives transcript windows and call history
  • Uses a system prompt tailored for customer service support
  • Pulls relevant data from CRM and documentation tools
  • Outputs tips + tool calls in structured format

Tools:

  • get_contact_by_email
  • create_support_ticket
  • update_contact_property
  • get_contact_deals
  • search_contacts_by_company
  • send_email (requires confirmation)
  • search_company_manuals

Cross-Call Learning

The system's key innovation is intelligent memory retention across customer interactions. Each call generates a Memory Box containing:

  • Deliverables: Key outcomes and commitments from the call
  • Improvement Areas: Targeted coaching points for future interactions

This memory is stored with the customer's email as the primary key. When the same customer calls again, the system retrieves this context, enabling agents to pick up unresolved issues and personalize their response.

API Overview

POST /default/callAgent

  • Input: call_id, window_num, client_email, transcript turns
  • Output: AI Tips, Activity Feed

POST /getCallAnalytics

  • Input: call_id, client_email
  • Output: Sentiment, metrics, agent scores, deliverables, improvement areas

Security & Monitoring

  • IAM roles: least privilege
  • All keys via env vars
  • Encrypted API requests
  • CloudWatch: logs, metrics, alerts

Final Words

Live Assist is the future of live agent support. Real-time insights, context-aware automation, and hybrid action control. Less burnout, better service, happier customers.

Let your agents focus on understanding clients — we'll handle the data.

Share this project:

Updates