LingoLens

LingoLens: Conversational language learning, grounded in your real-world context.

Comment

Inspiration

I've always found traditional language apps to be a bit too textbook for my liking. You spend hours matching words to static icons but the moment you step outside and see a real-world object, your brain freezes. I wanted to build something that felt like having a native-speaking friend standing right next to you. Someone who can just point at a coffee mug or a stray cat and tell you how to say it in Spanish. LingoLens was born from that desire to bridge the gap between digital learning and actual reality.

What it does

LingoLens is a real-time language coach that lives in your browser. It uses your camera to see the world around you and starts conversations based on what it finds. If you hold up a book, it doesn't just translate the word. It encourages you to pronounce it, corrects you if you're off, and then uses it in a sentence. It’s multimodal and low-latency, so the conversation happens naturally through audio and video without you having to click any "submit" buttons.

How we built it

The backbone of this project is the Gemini Live API. To get this off the ground, I leaned heavily on the Gemini CLI and its recently launched Plan Mode. It was a massive help for architecting the streaming logic. Having an agent that could map out the dependencies between the frontend WebWorkers and the backend token service saved me hours of head-scratching.

I also used the gemini-live-api-dev skill, which provided the specialized knowledge needed to handle PCM audio processing and the v1alpha authentication flow for ephemeral tokens. The stack is rounded out with a Node.js/Express backend for secure API key management and a Vite-powered frontend for the UI.

Challenges we ran into

I spent a good chunk of time wrestling with browser audio APIs, especially handling 16kHz mono PCM data and ensuring the VAD (Voice Activity Detection) didn't cut off the tutor mid-sentence.

Accomplishments that we're proud of

I’m genuinely proud of how seamless the "Live" experience feels. There’s something a bit magical about moving your camera around and having the AI accurately identify a houseplant or a laptop in near real-time. I also managed to get a robust production setup running on Google Cloud Run.

What we learned

Building this was a deep dive into real-time WebSockets and multimodal AI. I learned that latency isn't just a technical metric. It’s the difference between a tool feeling like a machine and a tool feeling like a companion. I also realized how much more efficient development becomes when you use a coding agent in Plan Mode to think through the implementation before writing any code.

What's next for LingoLens

I'm looking at adding "Visual Quests." The tutor might ask you to go find something "yellow and sweet" to test your vocabulary in the wild.

Built With

Share this project:

Updates

Submission history