Inspiration

Sometimes the hardest part of a conversation is finding your next sentence. You have experiences to share and questions worth asking, but the moment moves faster than the words.

We built Voca around that gap: a conversation companion that helps you express yourself while staying present. A short cue through your glasses can offer a starting point without making you reach for your phone.

What it does

Voca combines environmental context from Ray-Ban Meta glasses, recent dialogue captured through a Vexo bracelet, and personal memories you explicitly approve to suggest something relevant to say.

Tap the bracelet to request a short spoken cue through your glasses. It might help you ask a follow-up question, connect the topic to an experience, or find an opening.

The app brings this into three spaces:

  • Session: Get contextual cues and rate whether they helped.
  • Review: Revisit cues and their supporting context, review a conversation excerpt, and practice with a spoken partner.
  • Memory: Approve or dismiss suggested details and manage what Voca can draw on.

Inference and speech processing run fully on-device and offline, ensuring your conversations and environment remain private and safe from the world at all times.

How we built it

We built a native SwiftUI app with Gemma 4 running locally through LiteRT-LM. E4B-it is the primary model, with E2B-it as a fallback; only one model is loaded at a time.

The Meta Wearables SDK provides visual context, while a direct BLE integration handles Vexo gestures and audio. Gemma (as a fallback, Native Swift code) decodes audio and uses Apple’s on-device speech recognition.

Embedded Pipecat coordinates the voice pipeline inside the app, with Swift handling audio capture and playback. A shared Swift package, VocaCore, manages contracts, evidence validation, persistence, and coaching policy.

Explicit helpful/unhelpful feedback updates intervention preferences.

Challenges we ran into

The hardest part was coordinating the devices and runtimes into one dependable interaction. A tap has to become fresh context, a grounded model response, and completed speech through the correct audio route.

We worked through gesture calibration, Bluetooth connections, model lifecycle management, and interruptions. We also had to distinguish a response being generated from the wearer actually hearing it.

Grounding introduced another challenge: captured dialogue does not reliably identify the speaker. We preserve that uncertainty and keep captured excerpts separate from approved personal memories.

Accomplishments that we're proud of

We verified a complete physical-device flow: a bracelet tap triggered an E4B cue grounded in fresh dialogue, and the wearer confirmed hearing the full, relevant cue through the glasses.

We also brought local inference, wearable input, spoken delivery, persistent feedback, and explicit memory approval into one native app.

That verified run reached native playback start in about 6.9 seconds. It gives us a concrete baseline for improving responsiveness.

What we learned

  • A useful conversational cue needs to arrive at the right moment, stay grounded, and leave room for the wearer’s own voice.

  • We learned that privacy decisions shape the architecture: local processing, temporary dialogue buffers, explicit memory approval, and careful source tracking all affect how the system works.

  • We also learned to separate technical success from user benefit. Delivering a valid cue proves the pipeline works; understanding whether it helps takes real conversations and explicit feedback.

What's next for Voca

Our next priorities are reducing latency, testing longer sessions, and validating spoken practice with real microphone turn-taking.

We also want to evaluate whether cues feel relevant, natural, and personal across different situations, and whether practice helps people feel more prepared for their next conversation.

Built With

Share this project:

Updates

Submission history