Inspiration

I live in Spain. At the town hall, the doctor or a dinner table full of people, the language barrier makes you a spectator in your own life. A phone app helps, but you keep looking down at a screen and handing it back and forth. On Quest the interpreter can disappear into the conversation: you look at the person, you speak, they hear you in their language, and your hands stay free.

What it does

SpeaKore VR is a live speech-to-speech interpreter and a hands-first assistant for Meta Quest and Meta VR glasses. It is the headset side of SpeaKore, which already runs on iOS and Android. Quest VR meets iOS and Android.

  • At the table: everyone on their own device, anywhere. Open a table and you get a six-letter code. Others join from their own Quest, Meta glasses or browser, like a meeting code, in the same room or across the world. You always sit at the head; a person appears at the table only when they really join. One person speaks at a time; everyone reads and hears it in their own language. Subtitles float right in front of you at reading height, in your language, and above the speaker's seat. While someone talks you see exactly what was heard (raw transcript, never "cleaned up" by an AI); then the translation replaces it.
  • In the street: headset meets phone. Meta glasses are made for the street. Your Quest or glasses connect to someone's SpeaKore phone app (iOS or Android) with a code. Each of you just talks in your own language and hears the other in theirs, on your own device.
  • Kore, the assistant you talk to. "Open translate", "show me Notion", "go back", "speak Spanish", "use a male voice". Kore knows which Quest screen you are on and navigates for you. It speaks and listens in your language.
  • Hands-first, no pointing needed. Every movement means something. Quest microgestures: tap your thumb on your index finger to talk (tap again when done), swipe left for back, right for home, forward/back for bigger or smaller captions. Palm up opens a menu on your hand; a fist mutes. Pinch, poke and controllers (A/X talk, B/Y back) work everywhere too.
  • Accessible by design. Live subtitles for people who are Deaf or hard of hearing and for anyone in a noisy room; three caption sizes; every action works by hand, by voice or by touch; contrast and target sizes follow WCAG 2.2.

How we built it

  • Quest app: Meta's Immersive Web SDK (IWSDK), WebXR and Three.js, in TypeScript. Mixed reality with passthrough; the app finds your real table, or places a virtual one. Hand tracking for microgestures, palm-up and fist detection from the hand joints, pinch and poke.
  • One server for every device: a FastAPI backend on a Nebius VM. Every device (Quest, glasses, phone, browser) connects over the internet with WebSockets, like a video call. The server is the meeting room: it runs speech recognition per speaker, translates once into every language at the table, and sends each person text and voice in their own language.
  • Interpreting: streaming speech recognition (Deepgram), translation with NVIDIA Nemotron on Nebius Token Factory, natural voices (Deepgram Aura and Cartesia). Measured: about one second from the end of your sentence to the spoken translation, 16 languages.
  • Kore: NVIDIA Nemotron with tool calling, with a separate Quest tool set (screens, connectors, table) so it never acts on screens that are not there.
  • No sign-in for the jury: the headset gets a guest account automatically, with hard limits on the server.
  • Connectors without logos: following the competition rules, connectors appear in six neutral categories with simple icons, no third-party logos. Money is read-only: SpeaKore never moves money.

Challenges we ran into

  • One device or many? We first built the table like our phone app: one device passed around. On a headset that makes no sense. We rebuilt it as a room where everyone joins from their own device, which is what makes Quest meet iOS and Android.
  • Pointing is not hands-first. A ray pointer is just a joystick. We moved the core actions to microgestures and hand poses so you never have to aim.
  • Reading in VR. Our first translations were cards lying flat on the table: unreadable. Now they are subtitles at eye level and upright cards above each seat that keep the same readable size at any distance.
  • No echo, no wasted minutes. Half-duplex audio (the microphone pauses while a translation plays) and a speech gate (silence is never sent).

Accomplishments that we're proud of

Three devices, three languages, one sentence: an English speaker, a Spanish listener and a Dutch listener on Quest, from spoken words to translated voice and subtitles in about a second. And an assistant you can steer with a tap of your thumb.

What we learned

Hands-first is a design discipline, not a feature: every action needs one clear movement, clear feedback and a way back.

What's next

The phone app joining the headset table (the server already supports it), card games at the table, sign language support built together with Deaf users, and the Horizon Store.

Built With

Share this project:

Updates

Submission history