AICRM

Production AI sales agent with real-time voice pipeline, deterministic lead scoring, and zero-cost hybrid AI stack on a single VPS.

Comment

AICRM — AI-Powered Voice Sales Agent for Motorcycle Dealerships

Inspiration

In India, motorcycle dealerships lose 40–60% of inbound sales calls — the phone rings during peak hours, the sales team is on the floor with walk-in customers, and the caller hangs up. That lost call was a ₹2 lakh sale.

We watched this happen at a Royal Enfield dealership in Kolkata. Callers asking "Hunter 350 ki price kya hai?" or "টেস্ট রাইড বুক করতে চাই" were met with hold music or voicemail. The sales reps weren't lazy — they were already helping someone. The problem was structural: one human can't be on the phone and on the showroom floor at the same time.

That sparked the idea: what if an AI could answer every call — in the caller's language, with real prices from the database, and hand off to a human only when needed?

What We Built

AICRM is a production AI telephone sales representative that plugs into Exotel (India's leading cloud telephony platform) and answers inbound dealership calls 24/7 in English, Hindi/Hinglish, and Bengali/Banglish.

It's not a chatbot reading a script. It's a full voice pipeline that:

  • Listens — VAD detects speech, silence-based turn detection knows when the caller has finished speaking
  • Understands — Groq-hosted Whisper transcribes, then language detection identifies English, Hindi, or Bengali (including Roman-script Hinglish and Banglish)
  • Thinks — A deterministic "conversation team" handles common questions (prices, variants, branches) instantly without touching an LLM; ambiguous questions route to a Groq → Gemini → OpenAI fallback chain
  • Speaks — Indian neural TTS voices (en-IN-NeerjaNeural, hi-IN-SwaraNeural, bn-IN-TanishaaNeural) respond naturally
  • Acts — 19 validated backend tools book test rides, schedule callbacks, send WhatsApp brochures, check stock, and transfer to humans
  • Learns — Every call is scored automatically across 5 quality dimensions, with full conversation replay for human review

The Numbers

Metric Value
Backend Python files 139
Frontend React/TypeScript files 47
Total lines of code ~24,000
Knowledge base documents 32
Agent tools 19
Supported languages 3 (EN, HI, BN)
Target response latency < 3.5 seconds
Server cost 1 shared VPS (6 vCPU, 15 GB RAM, no GPU)
AI inference cost ~₹0 (free tiers) to ~₹0.20/call

How We Built It

Architecture: The Conversation Team

The most important design decision was not sending every caller utterance to an LLM. Instead, we built a deterministic "conversation team" where specialized components handle different responsibilities:

Caller → VAD → Turn Detection → STT → Language Detection
                                          ↓
                              Deterministic Sales Stage Analysis
                                          ↓
                                    ┌─────┴─────┐
                              Known intent?    Ambiguous?
                                    ↓               ↓
                           Direct Answer      LLM Composer
                           (from knowledge     (Groq → Gemini
                            contracts +         → OpenAI chain)
                            database tools)         ↓
                                    ↓          Tool Calls
                                    ↓               ↓
                              Indian TTS ← Merge ← Validate
                                    ↓
                              Exotel PCM Stream → Caller hears reply

When someone asks "Hunter 350 ka price batao", the system:

  1. Detects Hindi/Hinglish
  2. Recognizes price intent via pattern matching (handles प्राइस, price, প্রাইস)
  3. Calls get_product_prices against live PostgreSQL
  4. Renders the Hindi direct-answer contract with real values
  5. Speaks through hi-IN-SwaraNeural

No LLM was involved. The response is instant, accurate, and costs zero tokens.

The LLM composer only activates for genuinely ambiguous turns — comparisons ("Hunter vs Classic?"), objection handling, or advisory questions. Even then, it has access to the same 19 validated tools with Pydantic argument validation and structured error handling.

Voice Pipeline: Built From Scratch

We deliberately did not use Pipecat (the popular voice AI framework). Inspecting its wheel metadata revealed dependencies on transformers, onnxruntime, and numba — a footprint that would crush our shared 15 GB VPS. Instead, we wrote a custom ~1,600-line voice pipeline (pipeline.py) implementing:

  • Energy-based Voice Activity Detection (VAD)
  • Silence-based turn boundary detection
  • Barge-in handling (caller interrupts → clear Exotel playback → retain interrupting audio)
  • Exotel AgentStream WebSocket protocol (PCM framing: minimum 3,200 bytes, multiples of 320)
  • Playback mark synchronization

Zero-Cost AI Stack

Running on a shared VPS with no GPU forced creative optimization:

We benchmarked qwen2.5:7b on the actual server — it generated at 9.5 tok/s idle but collapsed to 1.4 tok/s (a 10-word reply took 9.9 seconds) when co-hosted workloads spiked. A phone caller will not wait 10 seconds.

The solution: hosted free tiers for live calls, local models for background work.

$$\text{Cost per call} \approx \underbrace{0}{\text{Groq free}} + \underbrace{0}{\text{Edge TTS}} + \underbrace{0}_{\text{Groq Whisper}} = ₹0$$

When free tiers are exhausted, Gemini Flash-Lite costs approximately:

$$\text{Fallback cost} \approx \frac{\text{tokens per call} \times \text{price per M tokens}}{10^6} \approx ₹0.10\text{–}0.20$$

Multilingual Intelligence

Language detection uses a priority cascade — not just STT metadata:

  1. Script detection: Bengali (বাংলা) or Devanagari (हिन्दी) characters → definitive
  2. Roman-script cues: Strong Banglish or Hinglish markers
  3. STT metadata: Provider-reported language
  4. Conversation continuity: After a Hindi question, "Black" or "okay" stays Hindi
  5. Fallback: English

This prevents the common failure where a Hindi caller says "okay" and the system switches to English mid-conversation.

Quality Analytics

Every call is automatically scored across 5 weighted dimensions:

$$\text{QA Score} = 0.25 \cdot C_{\text{completion}} + 0.25 \cdot C_{\text{conversation}} + 0.20 \cdot C_{\text{latency}} + 0.20 \cdot C_{\text{reliability}} + 0.10 \cdot C_{\text{tools}}$$

Where each $C_i \in [0, 100]$. The system flags problematic calls (high latency, STT rejections, clarification loops) for human review, and the admin panel shows release-over-release quality trends, provider cost breakdowns, and conversation replay with per-turn timing.

Full CRM Backend

Beyond voice, the platform includes a complete dealership CRM:

  • Product catalog with categories, variants, colours, and historical pricing
  • Lead management with deterministic qualification scoring (hot/qualified/nurture/low_intent bands)
  • Customer deduplication with phone normalization
  • Appointment and callback scheduling
  • Branch management with availability checking
  • Offers engine with expiry enforcement
  • RBAC with permission-aware admin panel

Challenges

1. The Shared Server Problem

Our production VPS runs 6 other live websites. Any spike in MySQL/Apache on those sites could starve our AI pipeline. We solved this by moving all live inference off-server to hosted APIs, keeping only VAD (CPU-light) and background analysis local.

2. Exotel's Undocumented Quirks

The Exotel AgentStream WebSocket protocol has strict PCM framing requirements (minimum 3,200 bytes, multiples of 320) that aren't prominently documented. We discovered this through trial and error — audio that worked in local testing produced silence on real calls until we added proper padding.

3. STT Quality in Noisy Indian Environments

Callers on Indian roads, in auto-rickshaws, or at busy shops produce noisy audio. The hosted Whisper model sometimes transcribes environmental noise as words, or leaks its own prompt text. We built a rejection pipeline that filters transcripts below a confidence threshold, detects prompt leakage patterns, and prevents these ghost turns from entering call memory.

4. The "End Call" Problem

Early in testing, the LLM would decide to end calls prematurely — interpreting silence, short utterances, or failed searches as the caller wanting to hang up. We made end_call the only tool the AI cannot invoke autonomously: the application blocks it unless the latest caller turn contains explicit closing intent. Blocked attempts are logged as end_call_blocked for QA review.

5. Hindi/Bengali Continuity

When a Hindi-speaking caller says a single English word like "Black" or "Hunter 350", naive language detection switches the entire conversation to English. Our cascaded detection preserves the established language across short neutral fragments — the TTS continues in Hindi, maintaining a natural conversation flow.

What We Learned

  1. Deterministic beats generative for known facts. An LLM hallucinating a motorcycle price is worse than no answer at all. By routing known intents through validated database tools with direct-answer contracts, we eliminated price hallucination entirely.

  2. Latency is the product. On a phone call, 3.5 seconds feels conversational; 6 seconds feels slow; 10 seconds and the caller hangs up. Every architectural decision — the conversation team, the provider fallback chain, the cooldown on rate-limited providers — was driven by this latency budget.

  3. You can't test voice with unit tests. Our pytest suite verifies orchestration, tool validation, and state transitions. But audible naturalness, carrier timing, barge-in feel, and real-world STT quality can only be verified by making a real phone call and listening.

  4. Free tiers are a viable production strategy — for the right scale. At 20–50 calls/day with one concurrent call, Groq and Gemini free tiers handle the load. The architecture makes upgrading to paid tiers a .env change, not a rewrite.

  5. Build the evidence layer from day one. Every turn persists its answer source (direct_answer_selected vs conversation_team_routed), provider used, latency, tool calls, and STT confidence. This makes debugging a bad call trivial — you can see exactly which component handled each turn and why.

Tech Stack

Layer Technology
Backend Python 3.12, FastAPI, SQLAlchemy (async), Alembic, PostgreSQL, Redis
Frontend React, TypeScript, Vite, Zustand, Recharts
Voice Pipeline Custom (VAD, turn detection, transport — no framework dependency)
STT Groq-hosted Whisper (primary), Faster-Whisper (fallback)
LLM Groq → Gemini → OpenAI fallback chain (live); Ollama qwen2.5:7b (background)
TTS Edge TTS Indian voices (primary), Piper (fallback)
Telephony Exotel AgentStream/VoiceBot WebSocket
Deployment Ubuntu 22.04 VPS, Apache reverse proxy, systemd
CI/CD GitHub Actions (Ruff + MyPy + pytest + Alembic + frontend lint/build + gitleaks)

What's Next

  • Real-call tuning from production Exotel telemetry — optimizing first-turn STT quality, barge-in thresholds, and Hindi/Bengali voice naturalness
  • Multi-dealership support — the knowledge layer is already structured per-dealer; the routing needs to select the right knowledge set per inbound number
  • Outbound follow-up calls — using the lead qualification data to proactively call back high-intent leads
  • WhatsApp integration — the adapter exists (Meta Cloud API); needs production configuration and approval

Built With

Share this project:

Updates