Codemix Skill — code-mixed support calls, English tickets

Inspiration

Listen to how people actually call support in India.

"En account la காசு வரல" — my money hasn't come into my account. Tamil, then English, then Tamil again, all in one breath. Nobody switches at a sentence boundary. They switch mid-thought, wherever the word comes first.

Every voice agent on the market says it supports 70+ languages. But "multilingual" means you pick one language and stay in it. The moment a caller switches inside a sentence, the agent loses the intent and asks them to repeat. That's where calls die, and where people give up and just stop calling.

This hits Indian small businesses hardest. Their customers are the least likely to speak one clean language, and they are the least able to staff a big support team.

Before we wrote any code, we checked whether someone had already solved this. We looked at warm-transfer briefing agents, AI roleplay trainers for support staff, stale-knowledge detectors — all of them already exist as shipped products. What we could not find was an agent that handles a language switch inside a sentence and still keeps a clean English record. So we built that.

What it does

Codemix Skill sits between speech and intent:

• Listens to the caller in whatever mix they use (Tanglish, Hinglish, Benglish, etc.)
• Tags each word by language and finds the switch points inside the sentence
• Pulls one unified intent out of the mixed speech
• Looks up the order and decides an action
• Replies in the caller's own mix
• Writes the ticket in clean English

The idea in one line: The customer speaks how they speak, and the company's records stay in one language. Search, routing, reporting, and QA all read English, while the caller never has to switch for the machine's benefit.

It ships as a reusable skill, not a single bot. Any agent on the platform — support, sales, HR — can call it and stop breaking when the customer switches language.

How we built it

The whole thing runs as a single page. Record a caller, and the audio goes to Scribe, the transcript goes to Gemini for token-level tagging and intent, the reply goes back out through ElevenLabs, and the English ticket renders alongside.

Challenges we ran into

• Tamil script broke our tokenizer: Our first word-splitting pattern used the standard \w character class, which only covers A–Z and 0–9. It shredded Tamil into single characters and reported zero language switches on a sentence that had four. We rewrote it to be Unicode-aware and detect the actual script block of each character. It now handles Tamil, Devanagari, Bengali, Telugu, Kannada, Malayalam, Gujarati, Punjabi, and Odia.
• We had the language detection backwards: We listed Indic words and defaulted everything else to English. So any Tamil word we hadn't listed came out tagged as English. The fix was to invert it: support English is a small, predictable vocabulary, and Indian languages are not. English became the closed list, and anything outside it defaults to Indic. Tanglish, Hinglish, and Bengali all started working immediately.
• Gemini went down mid-build: A 503 during testing, and the screen went blank. We were one bad wifi connection away from a dead demo on stage.

What we learned

• You cannot validate a multilingual system in English: The two bugs above are the kind a judge would hit in ten seconds by typing one Tamil sentence. We found them first, and that changed how we think about testing this class of product.
• The 503 changed the architecture: Every step now degrades instead of failing: the model retries twice, then falls back to a deterministic engine running entirely in the browser; ElevenLabs falls back to the browser voice; an incomplete model response fills its gaps from the built-in result. A demo that dies on venue wifi is not a demo.
• Check before building: Three of our first four ideas were already shipped products. Searching first cost us an hour and saved us a day.

What's next

1. Measure it: Test accuracy against a labelled code-mixed dataset. That is the honest gap and the first thing we will address.
2. Move keys to a backend: They currently live in the browser, which is fine for a demo, but not for production.
3. Connect a real order system: Integrate live order/CRM APIs instead of our current fixture.
4. Widen coverage: We have tested Hindi, Tamil, and their romanized forms thoroughly. Next is expanding validated benchmarks across all other Indic languages. ```

Built With

• api
• eleven-labs
• freshworks-agent-studio
• html5
• javascript