Inspiration

Most of us meet Van Gogh's paintings behind glass, a few steps away. We wanted the opposite: to stand inside the picture, at the spot where he set up his easel, and look around. The paintings are in the public domain, and Vincent left hundreds of letters describing what he saw and why he painted it. That gave us a world to walk through and a voice for the host.

What it does

You open a link in the Quest browser and step into a village rebuilt from The Starry Night. The swirling sky is above you and the houses and cypress are around you. One door leads to Vincent's studio, a room rebuilt from The Bedroom in Arles. Vincent is there, a character inspired by the historical painter, voiced from his public-domain letters, and you can ask him things out loud.

Everything works with your hands alone:

  • Move: pinch with your right hand, aim the dotted arc, let go to land there. It only lands where walking could reach, so you never end up inside a wall.
  • Turn: pinch with your left hand and swipe sideways.
  • Act: look down and a small panel shows what you can do right here, like opening a door, sitting on the chair, lying down, or a gesture. Point and pinch.
  • Talk: point at Vincent, pinch and hold while you speak, let go to send. He answers in his own voice, and his words appear in a bubble over his head.
  • Point and ask: pinch on the bed, the window or the chair instead, and your question carries what you are pointing at. While he answers, a line runs from his hand to that object.
  • Be together: other visitors in the same village show up with their heads and hands moving as they really do. A raised, waving hand becomes a wave everyone sees.

First-time visitors see three small hand pictures (Pinch / Point / Let go to move) instead of a wall of text. They go away after the first move. If you are seated, your eye height is lifted to standing height, so the whole experience works from a chair.

How we built it

  • Rendering: three.js on WebXR (immersive-vr, local-floor, hand-tracking). The village and studio are 3D scenes built from the original paintings: objects are modelled and placed to match each composition from the painter's own viewpoint, and the sky and far background stay as his brushwork.
  • Hands and locomotion: Meta's Immersive Web SDK. @iwsdk/xr-input (XRInputManager) tracks and draws hands and controllers, and its parabolic sampling from @iwsdk/locomotor draws the teleport arc. On top of that we wrote pinch ownership, so a pinch that starts on a button or on Vincent belongs to that target and never also teleports. We also wrote swipe-to-turn and wave detection. Vision Pro's gaze-and-pinch arrives through the same code path.
  • Collision-aware teleport: every landing is tested by walking a copy of the player with the same collision the desktop walk uses.
  • Vincent: a server-side character grounded in a knowledge base of his public-domain letters, with speech-to-text in and streamed text-to-speech out. The headset shows his answer sentence by sentence as his voice reads it. Pointing questions add the object's name to the prompt, and answers that name an object in the room point back at it.
  • Multiplayer: a WebSocket room server. Head and hand poses are packed into a small binary frame and sent only to clients that support them.
  • Comfort and performance: a VR-only quality ladder (fixed foveation, crowd level of detail, texture filtering) steps down if the frame rate falls below the headset's target. The painting itself is never cut. A seated-height calibration runs at the start of each session.
  • Languages: the whole interface, including the VR panel, hints and Vincent's replies, follows the visitor's language. English is fully supported.

Challenges we ran into

  • Pinches that mean different things. The same pinch can be "move", "press this button" or "talk to Vincent". We settled on one rule: whatever the pinch starts on owns it until release.
  • A teleport arc that ran backwards. The parabola we sampled first went up and behind the hand, because the solver picked the other root when gravity points down. We found it with the WebXR emulator and fixed the velocity sign.
  • Menus that do not exist in VR. The flat page's buttons are invisible inside a headset. Rather than port every sheet, we put only the actions that make sense where you are standing on one panel that appears when you look down, and left out the ones that need a flat screen.
  • Being honest to the paintings. Our rule is to show what is in the painting and not add effects that are not. That ruled out many easy VR tricks like glowing outlines or hover highlights.

Accomplishments that we're proud of

Asking Vincent about his own bed while standing next to it, and watching him point at it as he answers. That moment only works in VR.

What we learned

In VR, pointing becomes part of language. Once the hand can say "this", questions get shorter and the answers feel closer.

What's next

  • More paintings as connected places, each entered through Vincent's studio.
  • Eyes as an input where the headset supports it, for example looking at a star to paint it in, as in our desktop version.
  • An accessibility pass: voice alternatives for the panel actions and a one-handed mode.
  • Target launch date: Available now on the web (WebXR); a Meta Horizon Store build will follow

Built With

  • claude-(anthropic)-for-the-character
  • cloudflare
  • javascript-(es-modules)
  • meta-immersive-web-sdk
  • node.js
  • speech-to-text
  • text-to-speech
  • three.js
  • web-audio
  • websocket
  • webxr-device-api
Share this project:

Updates

Submission history