LensLingo — Real-Time Multimodal Conversation Assistant

An AI assistant for Meta glasses that translates conversations live using vision and audio context.

Comment 1

Inspiration

Language barriers still make everyday interactions difficult — especially when traveling, meeting new people, or working in international environments. While translation apps exist, they often fail in real conversations because they lack context. Short spoken phrases can be ambiguous, and without understanding the environment around the speaker, translations can be inaccurate or unnatural.

We were inspired by the idea that AI assistants should not only hear what we say, but also see what we see. By combining real-time audio translation with visual context from smart glasses, we can create a much more natural and helpful communication experience.

LensLingo was inspired by this vision: a multimodal AI assistant that helps people communicate across languages in real-world situations.

What it does

LensLingo is a real-time multimodal conversation assistant for iPhone and Meta AI glasses.

The system listens to live conversations, understands the surrounding visual context, and translates dialogs instantly using Gemini on Google Cloud.

Key capabilities

Real-time conversation translation

  • Detects spoken language automatically
  • Translates conversations in both directions
  • Displays subtitles on the iPhone in real time
  • Optionally generates spoken translations

Visual context awareness

Using the camera feed from Meta AI glasses, the assistant can:

  • Recognize menus, signs, and products
  • Understand objects in the environment
  • Improve translation accuracy using scene context

Conversation assistant

LensLingo can also:

  • Summarize what another person said
  • Extract key information such as prices, addresses, or times
  • Suggest helpful replies during a conversation

Session memory

The assistant keeps track of conversation context by:

  • Remembering language preferences
  • Tracking previously discussed topics
  • Improving translation continuity during the session

How we built it

LensLingo uses a streaming multimodal architecture powered by Google Cloud and Gemini.

The iOS app, built with Swift and SwiftUI, captures live audio and camera input from the user (and optionally from Meta AI glasses). Audio is processed using AVFoundation, while real-time communication with the backend is handled via WebSockets.

The backend is deployed on Google Cloud Run and orchestrates the AI pipeline.

When a conversation begins:

  1. Audio and visual context are streamed from the iPhone to the backend.
  2. The backend sends multimodal prompts to Gemini via Vertex AI.
  3. Gemini processes the speech transcript and visual input together.
  4. The model generates translations, summaries, and suggested replies.
  5. Results are streamed back to the iOS app and displayed as live subtitles.

Session state and conversation context are stored in Firestore, while temporary media artifacts are stored in Cloud Storage. Cloud Logging provides observability for debugging and monitoring.

Challenges we ran into

One of the biggest challenges was latency. Real-time conversations require extremely fast responses, and delays can break the natural flow of dialogue. To solve this, we experimented with streaming responses and partial translations rather than waiting for full outputs.

Another challenge was multimodal context integration. Combining visual signals with conversational text required careful prompt design so the AI could use useful visual information without overwhelming the model.

We also had to design the system to handle noisy environments, variable lighting conditions, and incomplete speech inputs.

Accomplishments that we're proud of

We are proud of building a working multimodal AI assistant that combines audio, vision, and language understanding in real time.

One of our biggest accomplishments was demonstrating how visual grounding improves translation quality. When the system can see menus, products, or signs in the environment, it can disambiguate short spoken phrases and produce much more accurate translations.

We also successfully deployed the backend on Google Cloud, integrating Gemini via Vertex AI and building a scalable architecture capable of supporting real-time multimodal interactions.

What we learned

Building LensLingo reinforced how powerful multimodal AI systems can be when applied to real-world problems.

We learned that translation quality improves dramatically when the AI understands environmental context. Visual cues can help resolve ambiguity that would otherwise lead to incorrect translations.

Another key lesson was that in conversational systems, speed and responsiveness matter as much as accuracy. Streaming partial responses can make interactions feel much more natural.

What's next for LensLingo — Real-Time Multimodal Conversation Assistant

Our next goal is to evolve LensLingo into a fully immersive AI communication companion for wearable devices.

Future improvements could include:

  • Direct integration with smart glasses for hands-free interaction
  • AR subtitles displayed directly in the user’s field of view
  • Persistent conversation memory across sessions
  • Offline fallback support for travel scenarios
  • Expanded language coverage and domain-specific vocabulary models

Ultimately, we envision LensLingo as a step toward seamless global communication, where AI assistants help people connect and collaborate regardless of language barriers.

Share this project:

Updates