PolyglotVision – Project Story

🌟 Inspiration

During a family video call last year, my grandmother (who only speaks Spanish) sat silently while everyone else chatted in English. No live interpreter, no instant subtitles – just a wall between languages. I couldn’t stop thinking: “Why can’t we just talk and be understood instantly, like in sci‑fi movies?” That moment became PolyglotVision – an experiment to make language invisible in real‑time video calls.

💬 What it does

PolyglotVision is a peer‑to‑peer video calling app that translates your spoken words into another language and makes your lips move accordingly. Speak English, and the other person sees your mouth shape Spanish words in real time – complete with live captions and a spoken translation. No accounts, no downloads, no cost. Just open the link and talk.

🛠️ How we built it

Frontend: Next.js (React) with Tailwind CSS for a sleek, minimal interface.
Video & Audio: WebRTC via PeerJS for direct browser‑to‑browser calls, removing the need for a media server.
Speech‑to‑Text: The Web Speech API (SpeechRecognition) captures your voice continuously and converts it to text.
Translation: A serverless Next.js API route proxies requests to the free MyMemory Translation API – no API key needed.
Text‑to‑Speech: The Web Speech Synthesis API speaks the translated text aloud in the target language.
Lip‑Sync: An HTML5 Canvas overlay on the local video feed uses the Web Audio API to analyse microphone volume in real time. The mouth shape is driven by the RMS energy of the audio signal:

$$V_{RMS} = \sqrt{ \frac{1}{N} \sum_{i=1}^{N} x_i^2 }$$

This volume value is smoothed and mapped to a mouth‑openness parameter, then drawn as two curved lip arcs that separate proportionally.

Deployment: The entire app lives on GitHub and is deployed to Vercel with zero configuration. All development was done directly in the browser using GitHub’s web editor – no local tools required.

🚧 Challenges we faced

Real‑time lip‑sync without heavy ML: We originally wanted to use Wav2Lip, but running it in the browser wasn’t feasible. We pivoted to audio‑driven canvas animation – making it look natural required careful smoothing of the audio signal and tuning the mouth shape geometry.
Browser API timing: SpeechRecognition, translation fetch, and SpeechSynthesis all run asynchronously. We had to design the flow so subtitles update immediately while translation and speech happen in the background without blocking the UI.
PeerJS signalling: Ensuring stable connections across different networks with PeerJS’s free cloud server took trial and error – we handled edge cases like early media and ICE restarts gracefully.
Free translation reliability: The initial open‑source API was unstable, so we switched to MyMemory, which offers a generous free tier and low latency.

🏆 Accomplishments we’re proud of

Built a fully working real‑time translation + lip‑sync call using only browser APIs – no GPU, no expensive cloud services.
Created the entire codebase and deployed it from a browser, proving that powerful apps can be built with minimal resources.
Achieved a convincing lip‑sync effect that reacts dynamically to voice volume – users reported they “forgot” the mouth was drawn.

📚 What we learned

How to combine WebRTC, Web Audio, Canvas, and multiple speech APIs into one cohesive pipeline.
The mathematics of audio analysis (RMS, frequency bins) and how to map physical phenomena to visual animations.
That constraints breed creativity: not having a GPU forced us to invent a lightweight lip‑sync solution that turned out to be more robust and fun.

🔮 What’s next for PolyglotVision

Integrate a real neural lip‑sync model (e.g., Wav2Lip) via WebAssembly for photorealistic mouth movements.
Add automatic language detection.
Build a browser extension for Zoom, Google Meet, and other platforms.
Support more languages and regional dialects.