Inspiration
Live calls are hard. You have to listen, translate, think and reply, all at the same time. It gets even harder when the other person speaks Tamil and you need to answer clearly in English. Beginners often rush a price, promise a deadline too early, or miss what the client really needs. I wanted an assistant that sits beside you during the call and helps you find the right words. I named it Mozhi, which means "language" in Tamil.
What it does
Mozhi Copilot is a real-time desktop assistant for live calls.
- It captures the other person's voice and your own microphone as two separate sources, so it always knows who said what.
- It waits for a natural pause before processing, so suggestions never arrive in the middle of a sentence.
- It understands Tamil and English. The original text is shown next to a clear English version.
- The "Say This" box suggests one or two polite sentences that answer the other person's latest words and follow topic changes.
- The "My Words" box shows what you said.
How we built it
- Python and Tkinter for the desktop window.
- PyAudioWPatch for Windows loopback capture of the call audio and microphone capture.
- A voice-activity detector with a pause timer that finds the end of each speaker's turn.
- Groq-hosted Whisper large-v3 for speech recognition.
- A Groq language model (gpt-oss-120b) to translate Tamil into English and to write the suggested reply from the recent conversation.
- Threads and queues so recording, transcription, AI calls and the window all run at the same time without freezing.
Challenges we ran into
- Half-finished sentences. My first version cut audio every few seconds, so suggestions arrived before the speaker finished. I replaced it with pause-based turn detection.
- Invented words. In silence or room noise, Whisper made up phrases like "Thank you". I added mic calibration, a threshold slider, and filters based on loudness and Whisper's own no-speech confidence.
- Tamil accuracy. Whisper's built-in translation mode mistranslated Tamil. Transcribing in Tamil first and then translating with a language model fixed most of it.
- Speed. Running Whisper on a laptop CPU was too slow for live calls, so I moved speech recognition to Groq.
- Context. The AI kept answering an old topic, so I changed the prompt to focus on the client's latest words.
Accomplishments that we're proud of
I built a complete working pipeline, from raw call audio to a suggested reply, and tested it on real WhatsApp calls in Tamil and English. It keeps both speakers separate and deals with real-world problems like noise and pauses.
What we learned
Real-time AI is mostly about the details around the model: audio handling, timing, noise, and choosing the right moment to respond. I also learned that real-world testing finds problems that clean demos never show, and that measuring and tuning matters as much as writing code.
What's next for Mozhi Copilot.
- An end-of-call summary with key points and follow-up actions
- Separate labels for each remote speaker in conference calls
- A neater floating window design
- More Indian languages
- Clear consent and privacy controls, since call audio is sent to a cloud service for processing
Current limits: it runs on Windows only, needs an internet connection, and Tamil recognition is good but not perfect.
