Twin Tongue

• 

  Live transcription

• 

  Control panel

Inspiration

Language barriers are a daily productivity problem, especially in customer service and technical support. They lead to repeated explanations, unnecessary call transfers, longer resolution times, and situations where the right specialist cannot help simply because they do not share a language with the customer.

This is something I see regularly in my professional work. I built Twin Tongue to explore a practical tool for companies and individuals who want to communicate more effectively without requiring both participants to speak the same language.

What it does

Twin Tongue is a Windows application that provides bidirectional, near-real-time speech translation during voice calls.

It creates two independent audio pipelines:

• The remote participant’s speech is captured, transcribed, translated, synthesized, and played through the agent’s headphones.
• The agent’s speech follows the same process in the opposite direction, sending the translated voice back into the calling application.

Each person speaks and listens in their preferred language. Twin Tongue currently supports English, Spanish, French, and Catalan, with configurable synthesized voices.

A local control panel shows both interlocutors clearly, including the original transcription and translated text for each direction. It also provides language selection, audio-device configuration, pipeline state, voice selection, diagnostic recording, and visibility into connected call applications.

How we built it

Twin Tongue is implemented in Python as two isolated, asynchronous audio pipelines connected to Windows audio devices through VB-CABLE.

Audio is captured as 48 kHz PCM in small real-time blocks. It is converted to mono and resampled to 16 kHz for Silero VAD and ElevenLabs Realtime Speech-to-Text. STT audio is grouped into configurable chunks—100 ms by default—to balance responsiveness and stability.

Final transcripts are translated using Google Cloud Translation and synthesized with ElevenLabs streaming Text-to-Speech. The resulting audio is resampled and routed either to the agent’s headphones or back into the calling application.

The system includes:

• Independent provider sessions for each conversation direction.
• Configurable voice activity detection and end-of-speech segmentation.
• Bounded queues and stale-segment protection.
• Streaming audio playback and latency monitoring.
• Asynchronous, configurable logging and metrics.
• Diagnostic WAV capture for end-to-end analysis.
• A loopback-only web control panel.
• 169 automated tests covering the main audio, provider, pipeline, configuration, and UI components.

Codex and GPT-5.6 were used as engineering collaborators to trace interactions across the real-time system, explore edge cases, keep implementation and tests aligned, and accelerate repository-wide iterations. The product direction, architecture, testing decisions, and reliability trade-offs remained human-led.

Challenges we ran into

The hardest problems were not isolated API integrations, but the behavior of the complete real-time audio loop.

Speech segmentation required careful tuning. Short silence intervals could split a long sentence into multiple segments, producing duplicated or incomplete translations. At the same time, excessive silence made conversations feel slow. Background audio and low-confidence final words could also generate text that nobody had spoken.

Audio feedback was another major challenge. Synthesized speech played locally could leak into the microphone, be detected as new agent speech, translated again, and sent back to the remote participant. We introduced a configurable, agent-side barge-in guard to prevent this loop while preserving the remote-to-agent pipeline.

We also found that antivirus scanning, frequent log writes, diagnostic recording, audio resampling, network variability, and multiple concurrent provider sessions can all affect a low-latency pipeline. This led us to buffer recording writes, make diagnostic features configurable, move logging away from time-critical paths, and add detailed operational metrics.

Finally, Windows audio routing required explicit device discovery and validation. Physical microphones, speakers, communications defaults, virtual cables, and application-level audio settings all need to agree for the complete experience to work.

Accomplishments that we're proud of

We completed successful end-to-end functional calls in which both participants spoke in their own language and received translated synthesized speech through an existing calling application.

We are especially proud that Twin Tongue translates the complete conversation in both directions. It does not only help an agent understand a customer; it also allows the customer to understand the agent.

Other important accomplishments include:

• Two independent, configurable real-time translation pipelines.
• Clear visualization of both interlocutors and both translation directions.
• Integration with existing Windows calling applications through virtual audio devices.
• Configurable STT chunking, VAD, segmentation, and confidence diagnostics.
• Evidence-based debugging through recordings, metrics, transcripts, and word-level log probabilities.
• Successful real-world end-to-end validation beyond isolated component tests.

What we learned

We learned that conversational translation quality cannot be measured by transcription accuracy alone. A technically correct transcript can still produce a poor experience if segmentation is late, sentences are duplicated, synthesized audio is queued for too long, or translated playback re-enters the microphone.

We also learned that bidirectional translation is fundamentally an audio-routing and concurrency problem. STT, translation, and TTS are essential components, but reliable turn-taking, queue management, echo control, device selection, and recovery behavior determine whether the system feels like a real conversation.

Most importantly, observability must be designed into a real-time system. Timestamped events, word-level confidence data, diagnostic audio, queue metrics, and visibility into both interlocutors made it possible to replace assumptions with evidence.

What's next for Twin Tongue

The next stage is turning the validated prototype into a more accessible and consistently reliable product for businesses and individuals.

The main priorities are:

• Reducing end-to-end latency across segmentation, provider startup, synthesis, buffering, and playback.
• Adding reference-based acoustic echo cancellation so translated audio cannot re-enter the microphone while preserving natural barge-in and full-duplex conversation.
• Adding noise reduction before voice detection and transcription.
• Continuing to improve STT segmentation and false-positive rejection.
• Simplifying first-time audio setup for non-technical users.
• Validating reliability across more headsets, microphones, accents, languages, network conditions, and calling applications.
• Improving packaging and deployment for controlled business pilots.

The goal is a dependable communication tool that helps support teams, specialists, colleagues, and individuals collaborate effectively, regardless of the language each person speaks.

Built With

• elevenlabs
• python
• silero
• speech-to-text
• text-to-speech
• v-cable
• vad