Speak French at seven Paris shop counters, with your hands, to AI shopkeepers who serve you, count your change and test you. Free in the Quest Browser, nothing to install: https://lingo-pal-vr.vercel.app

Inspiration

Language apps teach you the word for croissant. Then you land in Paris, a shopkeeper looks at you, says a price fast, counts change out loud, and you freeze. The scary part of a trip is not vocabulary, it is the counter: a real person waiting, money in your hand, no time to think.

I wanted a place to rehearse exactly that moment, with your body, before the trip. VR is the only medium where you can stand (or sit) at a Paris counter, point at the pastry case, hold up two fingers, hand over a note and hear the change counted back. That physical loop of look, gesture, speak and pay is what makes it stick, and it is why Lingo Pal VR is hands-first from the ground up.

What it does

The core loop takes about a minute. You sit at Madame Lune's café counter. Look at the croissants in her glass case, hold up two fingers and say « Deux croissants, s'il vous plaît ». She answers in French, serves two croissants on the shop's tray and tells you the price. Pinch a 5 € note onto the payment dish and she counts your change back aloud. Every line has English subtitles, and every French word you hear goes into your Carnet for review.

Seven places in Paris, seven AI shopkeepers. Café Lune (Madame Lune), La Boulangerie (Monsieur Paul), Le Marché (Madame Rose), Le Bistrot (Monsieur Marcel), Le Métro ticket window (Monsieur Karim), La Pharmacie (Madame Inès) and L'Hôtel (Madame Claire). Each has its own menu, prices, words, voice, 360° backdrop, a display that fits the shop (glass pastry case, produce crates, pharmacy rack, ticket rack) and a real-life curveball: « Sur place ou à emporter ? », a free carafe d'eau at the bistrot, directions to the Eiffel Tower at the métro, a symptom to explain at the pharmacy, a bank card at the hotel.

AI shopkeepers the game keeps honest. Each shopkeeper is an AI agent (Claude) that acts in the world only through tools: serve, quote a price, give change, run out of stock and offer a swap, or slip you wrong change on purpose to see if you catch it (« Attendez… ça fait combien ? » / « Bien vu ! »). The game validates every tool call before it happens in 3D: it owns the menu, prices, stock and arithmetic, and it holds the shopkeeper to the number of items you asked for. Shopkeepers remember you between visits and plan tomorrow's practice from your weak words. Without the AI, the game still runs on built-in scripted shopkeepers, so the experience never depends on a third-party service.

Pal, your English helper. Pal is a little sparrow on the counter. Pinch Pal (or look at it) and ask in English, "How do I ask for the bill?", and Pal gives you the French line (« L'addition, s'il vous plaît »), how it sounds, and a sounds-like spelling. Shopkeepers only speak French; Pal only speaks English; they never talk over each other.

Plan de Paris. Travel on a little 3D Paris that unfolds on your counter: pinch or touch a pin to go there. Open it from the folded paper map on every counter, the "Paris" button, the "Plan du métro" poster at the métro, or by saying « au marché ».

Vraie vie (Real Life) mode. No English, no hints, no reply chips, plus curveballs. Finish a six-item order at the boulangerie from memory and Monsieur Paul tells you « Bravo, vous parlez comme à Paris ! ».

Prêt pour Paris: a reason to come back. Readiness for a real trip is measured in 28 situations across the seven places (order, say it right, pay the exact amount, do it with no help), each ticked only by something the game verified, never by the AI's opinion. Set your trip date and the home panel counts down ("PRÊT POUR PARIS 34 % · J-12") with today's 10-minute plan. Around it: XP and an A1 → A2 level ladder, streaks with rest days, Mon Carnet (spaced repetition of the words you met), a daily shopping list across three shops, the Passeport de Paris with 16 stamps, Ma semaine (your week: XP per day, minutes, new words, orders, pronunciation, trend), and a « Bon retour ! » greeting from the shopkeeper who remembers your streak.

The first five minutes. A start page that loads in the background and explains the app in one line, a choice of hands, eyes or voice, then a ghost hand and Madame Lune's voice teach the first gestures with no wall of text. The first order is paid in about a minute.

Accessible by default. Every action has a voice alternative and an eyes-only alternative (look at a button for one second). High contrast, larger text, slow speech, always-on subtitles, left-handed layout, reduced motion, and the whole experience is seated: everything is within reach of the "airplane seat".

Mixed reality on your real table. Choose « Dans ma pièce » and the café counter lands on your real table using plane detection, with a spatial anchor so it is back in the same place tomorrow and depth occlusion so your hands pass in front of it. The shopkeeper stands across your own table.

Café à deux (social). Invite a friend with a link. They join from any browser (another headset, a laptop or a phone), peer-to-peer, appear at the counter as a person, and you take turns ordering from the same shopkeeper: « À vous, Sam ! ».

Built for Meta Quest 3 / 3S and Meta VR Glasses. On narrow-view devices the side panels swing in and shrink so everything stays in view, and gaze + pinch works everywhere.

How I built it

  • Meta Immersive Web SDK (IWSDK), three.js and an ECS architecture: 27 systems for hands, gaze, dwell, panels, coins, the diorama, mixed reality, social and performance. UI panels are UIKitML.
  • Hands: WebXR hand tracking with finger counting from all 25 joints, robust to hand angle (index-first and the French thumb-first count), pinch, poke, palm-up to talk, wave and thumbs-up. A test harness replays captured hand-pose recordings to keep finger counting reliable.
  • Gaze: gaze picks what you are ordering; eyes-only mode turns a one-second look into a press.
  • Mixed reality: plane detection to find your table, spatial anchors to restore the counter, depth sensing for occlusion.
  • A deterministic game core (menu, prices, stock, French numbers, change, readiness) is pure TypeScript shared with the server through one data file, so the AI and the game cannot drift apart. The agent proposes, the game validates, the world shows the result.
  • Server routes on Vercel: Claude (strict tool use) for the shopkeepers and Pal, Inworld TTS-2 natural French voices (CDN-cached, so repeated lines are free), and speech recognition, all behind rate limits so no key ever reaches the headset. On-device Whisper is the fallback for French speech recognition.
  • Pronunciation scoring on every spoken line; Vraie vie removes the help.
  • Café à deux: WebRTC peer-to-peer sessions with keyed invite links; the friend's browser needs no install either.
  • Performance: static scenery merged per material, a render-scale profile per headset, fixed foveation, and a frame-rate guard that sheds extras step by step if frames drop and restores them later. A ?perf mode shows frame times and sends anonymous reports from real headsets.
  • Art: Microsoft Rocketbox avatars (MIT) with procedural animation, Poly Haven and ambientCG scans (CC0), a few CC-BY models (credited), and procedural props; sound effects and the café ambience are synthesized live with WebAudio. No real brands anywhere.
  • Quality: 430 automated tests cover the game core, tool validation, finger counting, served-goods layout, readiness and the map.

Challenges I ran into

  • Keeping an AI honest in a world with money. A language model will happily serve three croissants when you asked for two, or round the change. The fix was architectural: the model never does arithmetic or changes state directly; it calls tools, and the game checks every call against the order and the till.
  • Counting fingers at any angle. Raw joint distances fail when the hand turns. I built a harness from captured hand poses and tuned the counter until it held across poses and both counting styles.
  • No headset. I do not own a Quest, so the whole app was built and filmed in the IWSDK Meta Quest 3 emulator. I designed defensively for real hardware: a per-device render profile, the frame-rate guard and the ?perf reporting mode.
  • Seven shops on one counter. Served orders, the tray, the money dish, Pal, the map and the shop display all share a small counter; a packing layout with measured keep-outs makes every order fit without overlap.

Accomplishments that I'm proud of

  • An AI agent that genuinely acts in VR (it serves, quotes, gives change, runs out of stock, tests you) while the game guarantees it never cheats you.
  • A first order paid in about a minute, with no controller and no wall of text.
  • Every interaction works with hands, with eyes only, or with voice.
  • Seven complete places, a real progression loop, mixed reality on your own table, and multiplayer with a friend on any device, built solo.

What I learned

Let the game own the truth and let the AI own the conversation. And hands-first design is not a control scheme, it is the lesson: holding up two fingers while saying « deux » is how the number sticks.

What's next

  • Testing on Quest 3 and 3S hardware, and tuning from real ?perf reports.
  • More cities with the same engine (Madrid, Rome), and more situations per place.
  • A teacher mode: assign situations to a class and see who is prêt pour Paris.
  • Target launch: public web launch in December 2026 (it is already live and free at https://lingo-pal-vr.vercel.app), then the Meta Horizon Store as a PWA.

Try it (judges)

Testing status, honestly: built and tested in the IWSDK emulator with the Meta Quest 3 profile; not yet run on a physical Quest. The demo video is real gameplay from that emulator; the narration is AI text-to-speech in Pal's voice.

Built With

Share this project:

Updates

Submission history