CallSage

Smart, calm system that listens and acts wisely

Comment

Inspiration

Contact centers handle thousands of customer calls daily, yet most organizations still rely on manual auditing and post-hoc sampling to detect compliance breaches or service issues. We wanted to build an AI-powered listener — one that not only transcribes calls but understands intent, emotion, and policy context in real time. With Elastic’s hybrid search and Google Cloud’s generative AI, we saw the opportunity to transform every conversation into actionable insight.

What it does

It allows teams to search, browse, and understand all internal policies and procedures related to customer interactions — from refunds to privacy, rapport, and data handling — all stored as vector embeddings using Gemini’s embedding model and indexed in Elastic’s hybrid search engine.

  1. When a call recording is uploaded:
  2. The audio is transcribed via Google Speech-to-Text.
  3. Gemini generates a summary capturing key actions, tone, and intent.
  4. The system runs a policy compliance check by semantically comparing the conversation’s embedding against every relevant policy section in Elastic.
  5. If potential violations or risks are detected, CallSage provides contextual, actionable feedback — such as citing the exact policy section breached or suggesting corrective steps.
  6. For confirmed complaints, it automatically creates a Jira ticket and routes it to the right department.
  7. Gemini runs across Jira tickets so that managers can ask using natural text. e.g (What did X work on this week) or (what are the most ranking complaints this week)

This gives managers a searchable intelligence layer across all customer communications, internal team working — enabling them to ask natural questions

How we built it

  1. Frontend: Next.js + Tailwind for a clean “AI workspace” with chat-style UI and upload support.
  2. Backend: NestJS orchestrating the pipeline (speech → LLM → classification → Elastic → Jira).
  3. Google Cloud: Speech-to-Text, Vertex AI / Gemini embeddings and reasoning.
  4. Elastic Search: Hybrid (text + vector) search for policy retrieval, semantic matching, and analytics dashboards.
  5. Jira API: Auto-ticket creation for complaints or violations.
  6. SQLite + Elastic: Local caching for metadata, vector indexing for search.

Challenges we ran into

  1. Maintaining context consistency when chunking and embedding long policy documents.
  2. Parsing Jira’s Atlassian Document Format (ADF) for rich text ticket creation.
  3. Balancing latency vs. accuracy when calling multiple Google services in sequence.
  4. Designing a reliable agent loop that can reason over transcripts, classify intent, and check policies without hallucination.

Accomplishments that we're proud of

  1. Completed a fully working pipeline from voice upload → transcript → LLM analysis → policy audit → Jira ticket → Elastic dashboard.
  2. Built a searchable compliance knowledge base that can cite the exact policy section violated.
  3. Created a beautiful OpenAI-style interface with modular tools for transcribing, auditing, and reporting.
  4. Achieved over 90 % accuracy in distinguishing complaints from general queries in test calls.

What we learned

  1. How to combine Elastic hybrid search with Google Cloud AI for truly contextual, explainable results.
  2. The importance of auditability and citation in generative AI systems.
  3. That small optimizations (like local caching and summarization before embedding) drastically improve both cost and speed.
  4. Realized that compliance and call-center analytics are perfect real-world domains for responsible AI agents. ## What's next for CallSage
  5. Real-time streaming mode: Live transcription and policy monitoring while the call happens.
  6. Agent assist overlay: Suggest next steps or compliance reminders during live conversations.
  7. Multi-language support using Google Cloud Translation + multilingual embeddings.
  8. Redaction pipeline for PII/PCI masking before indexing in Elastic.
  9. Deployment on GKE for horizontal scalability and enterprise security.
  10. Long term: evolve CallSage into a self-learning compliance copilot that continuously improves from feedback.

To test please use these credentials

username: asd@gmailc.cin password: asdasdsaddadasd

or create a new organization and setup your own project

Built With

Share this project:

Updates