callpilot

An AI that makes real outbound phone calls on your behalf — RAG-grounded, multi-provider (OpenAI + Gemini Live), with recordings, voicemail detection, and human-grade interruption handling.

Comment

Inspiration

Phone calls are still the worst part of personal infrastructure for most people. Booking a dentist, calling about an order, canceling a subscription, getting takeout — predictable, turn-based conversations that eat hours every week. Realtime voice LLMs hit a usability threshold in 2026, and I wanted to see whether a single engineer could ship a production-grade outbound calling agent that actually works on real PSTN calls, not just a demo on stage. The answer was yes — in a recent test, CallPilot autonomously called a local Indian restaurant, ordered chicken biryani for pickup, confirmed the price, and ended the call. The transcript and MP3 are saved on my laptop.

What it does

CallPilot is a FastAPI server that places outbound phone calls and runs the entire conversation autonomously.

You give it three things:

  • a phone number to call
  • plain-English instructions ("order one chicken biryani for pickup at 7pm under Raju")
  • a folder of personal context documents (insurance card, profile, addresses)

It then:

  • indexes those documents into a per-client ChromaDB vector store at startup
  • runs a RAG query against your instructions to surface relevant facts
  • places the call via Twilio and bridges audio bidirectionally to a realtime voice LLM
  • speaks naturally, handles interruptions, refuses to invent details it doesn't have
  • detects voicemail and hangs up automatically
  • saves a full transcript (.txt) and a full call recording (.mp3) when the call ends

All of this is local-first. There is no SaaS layer. Your documents never leave your machine. The provider abstraction lets you switch the underlying voice LLM by changing one environment variable.

How it works (architecture)

Five components:

  1. Web UI (browser) — minimal dark-themed UI to enter a number + instructions and watch the live transcript scroll.
  2. FastAPI server — serves the UI, exposes /call to start a call, /twiml/{id} for Twilio's webhook, and /media-stream/{id} as the WebSocket bridge.
  3. Twilio outbound call — places the actual PSTN call and streams the caller's audio to the server as base64 g711_ulaw 8kHz over a WebSocket.
  4. Provider abstraction — pluggable layer in app/providers/. Today supports OpenAI Realtime and Google Gemini Live. Each provider yields the same six normalized events: audio, transcript_ai, transcript_caller, speech_started, response_done, error. Switch with AI_PROVIDER=openai or AI_PROVIDER=gemini.
  5. RAG layer — ChromaDB per-client collections. PDF/TXT/DOCX/MD are parsed and chunked at ~500 chars, embedded with OpenAI embeddings, stored, and queried at call time. Top 5 chunks are injected into the system prompt with explicit "if it's not in here, say you'll check with the client and get back to them" guardrails.

The bridge is two asyncio coroutines: one forwards Twilio audio to the provider, one forwards provider audio back to Twilio. Interruption handling is the hardest piece — when the caller starts speaking mid-AI-response, the bridge cancels the in-flight LLM response and sends a "clear" event to Twilio to flush already-buffered audio. Both steps are required; missing either feels broken.

The Gemini provider also handles audio format conversion (g711_ulaw 8kHz to PCM16 16kHz inbound, PCM16 24kHz to g711_ulaw 8kHz outbound) using audioop / audioop-lts.

What's new since the last Devpost update

  • Multi-provider abstraction: OpenAI Realtime and Google Gemini Live both supported via a clean provider interface. Cost per call drops ~40% on Gemini.
  • Call recordings: every call saves a full .mp3 to the recordings/ folder via Twilio recording status callbacks.
  • Voicemail detection (AMD): Twilio AnsweredBy=machine_* triggers an automatic hangup. No more 90-second monologues to answering machines.
  • Production-grade interruption handling: in-flight LLM cancellation + Twilio media buffer flush in lockstep.
  • Per-client document namespacing: clients// folders give each user their own ChromaDB collection and system prompt template.
  • Hallucination guardrails baked into the system prompt: model defers to the human when a fact isn't in the indexed documents.
  • Demo proof: autonomously placed a real food order to a local restaurant. Recording saved.

How I built it

Stack:

  • Python 3.11+, FastAPI, asyncio, websockets
  • Twilio Programmable Voice (Media Streams + AMD + recording callbacks)
  • OpenAI Realtime API (gpt-4o-realtime-preview)
  • Google Gemini Live (gemini-2.0-flash-live)
  • ChromaDB (vector store, per-client collections)
  • OpenAI text-embedding-3-small (RAG embeddings)
  • pypdf, python-docx (document parsing)
  • audioop-lts (Python 3.13 audio resampling shim)
  • ngrok (localhost-to-public tunneling for Twilio webhooks)

Process:

  1. Started with a single-provider OpenAI Realtime bridge (~300 lines).
  2. Added the RAG layer once the basic call worked end-to-end.
  3. Hardened with interruption handling and AMD after the first batch of real calls.
  4. Refactored to a provider interface to add Gemini Live without forking.
  5. Added recordings, per-client document namespacing, and the live transcript UI.

Challenges I ran into

  • audioop got removed from Python stdlib in 3.13. Spent an hour debugging "module not found" before discovering audioop-lts.
  • Interruption handling looks like it works until you realize Twilio buffers ~500ms of audio on its side. Cancelling the LLM response alone makes the AI sound like it's drunkenly mumbling for half a second after you cut in. The fix is sending a Twilio "clear" event simultaneously.
  • Voicemail detection sounds easy ("just hang up if it's a machine") but Twilio's async AMD callback can fire 5-10 seconds into the call. Had to design the bridge to gracefully tear down mid-conversation.
  • Gemini Live wants different audio formats than Twilio. The double transcoding (8kHz mulaw to 16kHz PCM, 24kHz PCM back to 8kHz mulaw) added latency I had to budget for.
  • Hallucinations on personal data. Without explicit "if it's not in your docs, defer" instructions, the model will confidently invent insurance policy numbers. Prompt engineering on the system prompt is the cheapest fix.

Accomplishments I'm proud of

  • An AI agent on my laptop autonomously ordered dinner. The recording is real. The receipt is real. The biryani was good.
  • Provider abstraction is clean enough that swapping voice LLMs is a one-env-var change.
  • Per-call cost is $0.07-0.23 — below a postage stamp — for a 2-minute end-to-end RAG-grounded conversation.
  • The codebase stayed small (~600 lines of Python excluding the UI). The system prompt does most of the heavy lifting; the architecture stays out of the way.

What I learned

  • The realtime voice APIs in 2026 are good enough for production task-oriented conversations. Latency, prosody, and turn-taking have all crossed the bar.
  • Most "AI voice agent" complexity is not in the LLM — it's in the audio pipeline (interruption, AMD, format conversion, buffer management).
  • A clean provider interface pays off the first time you swap backends. I now switch between OpenAI and Gemini for cost reasons without changing application code.
  • RAG works for voice exactly the way it works for chat. The LLM doesn't care that the channel is audio.
  • Guardrails need to be in the system prompt, not in retrieval. The instruction "if you don't know, say you'll check with my client" is the difference between an agent I trust and one I don't.

What's next for CallPilot

  • Mid-call RAG re-querying via function calls (prototyped, tuning).
  • Speaker diarization for multi-party calls (when a receptionist transfers you to scheduling).
  • Local-only inference path (Whisper + local LLM + Piper TTS) for fully air-gapped use.
  • Subscription cancellation runbook: cancel a Sirius XM / gym / streaming subscription, decline retention offers, save the confirmation number.
  • Web UI document upload (currently the clients/ folder is the source of truth).

Built with

python, fastapi, websockets, asyncio, twilio, openai, google-gemini, chromadb, pypdf, python-docx, ngrok

Try it out

GitHub: https://github.com/kennedyraju55/callpilot License: MIT

Built With

Share this project:

Updates