Lucid Dreams brings someone from your phone's videos into your room. One short clip or a few photos becomes the person, photoreal and standing life size in passthrough on a Meta Quest.

Ian hasn't been home to Vietnam in four years, and his grandmother lived in his camera roll, six inches tall. One video of her became her, at her real height on his floor. When he talks to her in Vietnamese she answers in her own voice, cloned with her and her family's consent and labelled as AI.

It works the same for anyone you have a video of. Walk all the way around them, talk with them, learn a move from them or sit across your own table from them, with your hands only, from one link in the Quest browser. When you open the build, our teammate Vishal's phone video plays life size in front of you, and then he steps out of it and looks at you.

Inspiration

Most of the people we love live in our phones now: a parent laughing at dinner, a friend's jump shot, a teacher showing a guitar strum, a grandparent telling the story they always tell. You can't sit across a table from a video. We wanted the people in those videos to step out of the phone and stand in the room with you, at their real height, close enough to walk around.

Ian's grandmother was the first. Then we kept going: a guitar teacher who sits beside you, Bobby Fischer and Magnus Carlsen across a chessboard on your own table, Roger Federer on the far baseline.

What it does

It's interactive video. Open a video and the person in it steps out: the clip plays life size in your room, wipes away from the feet up, and they're standing there at their real height, looking at you. Give Lucid Dreams a short video or a few photos plus one sentence ("teach me this strum", "play chess with me", "just hang out"), and it rebuilds that person from hundreds of thousands of tiny 3D Gaussians, standing on your real floor in your Quest's passthrough. Reality check paints blue whatever the camera never saw.

There are two kinds of people. A replay is the clip itself rebuilt in 3D, frame by frame: you can walk around them, freeze them mid-air and see them from angles the camera never had. Open the dance routine and its 5-second source clip plays life size first; then the dancer steps out of it and dances the same routine in 3D, where you can watch it from every side. An avatar is a rigged body built from photos or a few frames, so it can do things the footage never showed: sit at your table, look you in the eye, shake your hand.

Your first five minutes. Open the link on a Quest 3 or 3S and step into your room. Vishal's phone video plays a couple of meters away, life size: seven seconds of him turning around in a hallway, the clip he was made from. As he turns to face you, the video wipes away from his feet up and he's standing there instead, on your floor at his real height, breathing. He says hi first, one line of his own about how a minute ago he was a video on a phone. Then one card shows the only gesture you need: turn your left palm up, press with your right index finger. From your chair, lean in and look at his face; his head turns to follow you. (You can stand and walk around him. You never have to.) Press Talk and ask him anything; he answers in his own voice. Then sit at your table: the board lands on it, he takes the chair across from you, holds out his hand, and the game begins with a handshake.

What happens next depends on your sentence.

  • Learn. The guitar teacher sits beside you and scores every strum against the clip's own beat ("Early 102 ms", "On beat", a streak); the golfer tees up a ball at your feet and your swing sends it flying (Library, Guitar and Golf). Lesson mode splits the move into steps, slows each one down, repeats it, then waits for your turn. Follow along puts glowing targets where your head and hands should be right now, taken from the clip's own 22 body joints and scaled to your height. Your attempt replays as a ghost beside them, with the biggest gap called out in plain words ("left hand 18 cm low at the drop"), and a card gives every step its stars.
  • Talk. A person who agreed to it answers out loud in their own cloned voice, captioned and labelled AI voice, a median 3.1 s after you stop speaking.
  • Play. Chess lands on your real table with them in the chair across from you. They hold out a hand before the first move, think, reach over, move their own pieces and hit the clock, and they remember how your last game went. Pick Fischer's or Carlsen's level and their avatar sits down (labelled "Stockfish at Fischer's peak rating. Not affiliated."), or sit down at a position from one of their real games and find the move they're remembered for. Tennis puts Roger Federer's avatar on the far baseline of a full-size grass court: serves, faults, lets, deuce, tiebreaks, and a ball that dips with topspin and skids on the grass. In your room, LeBron plays his own jump shot until your pass reaches him, then catches it and shoots.
  • Court Side. A real NBA moment, Luka Dončić's step-back three over Rudy Gobert, plays as courtside video around you. Step into the court and that one frame becomes 3D: every player frozen mid-air at real size, and you walk between them before the ball leaves his hand.
  • Reality check. One tap turns everything the camera never saw blue and tells you how much of the person was real.

Why you'd come back: next time Vishal is waiting where you left him in your room (a persistent anchor) and greets you like he remembers, because Talk is told how many times you've visited and when; the people you add stay in your library, the board and the chair stay where you put them in your room, an unfinished game is still on the table tomorrow, he plays one level stronger every time you beat him, and he remembers how the last game went because that memory is a real record of it.

The build you'll judge opens on our teammate Vishal, rebuilt from a short phone video of him turning around; he agreed to his likeness and his cloned voice, and 75% of his Gaussians were seen in his own frames (Reality check paints the rest blue). Everyone else is one press away, under Library on the palm menu: Ian's grandmother, six replays you can learn from (guitar, golf, tai chi, a dance routine, two jump shots), the chess legends, Federer and Court Side. Every public figure is labelled not affiliated, and none of them ever speaks. In the judged build you can't upload your own clip yet: Make a dream runs on our university's GPU cluster, and everyone in the library came from that pipeline. Tennis and Court Side are played standing (Court Side switches to full immersion); everything else works from a chair.

How we built it

Any video in: the replay pipeline

A camera only ever sees one side of a person, so rebuilding a moving person in 3D used to take a ring of synchronized cameras. 4DAnyone (August 2026) gets there from one casual clip: it generates 48 consistent views of the person in three rings of 16 cameras, 121 frames at 25 fps. On the University of Florida's HiPerGator, 12 NVIDIA B200s fit about 290,000 3D Gaussians per frame with gsplat, 7,000 Adam steps for the first frame and 1,500 for each later one, warm-started from the one before. Each frame minimizes

$$\mathcal{L} = 0.8\,\lVert \hat I - I \rVert_1 + 0.2\,\big(1-\mathrm{SSIM}(\hat I, I)\big) + 0.1\,\lVert \hat A - A \rVert_1 + 0.3\,\mathcal{L}_{\text{VGG}} + \mathcal{L}_{\text{reg}}$$

where \(\hat I\) and \(I\) are our render and the target view, \(\hat A\) and \(A\) the rendered and true silhouettes, \(\mathcal{L}_{\text{VGG}}\) a perceptual loss, and \(\mathcal{L}_{\text{reg}}\) small penalties on opacity, scale and stretched Gaussians. Every frame ships as SPZ (about a tenth of a PLY), with GVHMR's 22 body joints for the lessons. A Gaussian counts as seen when a camera facing the real clip finds it in front of everything else, \(z_i \le D(u_i) + 4\,\text{cm}\) with accumulated opacity above 0.5; the rest is what Reality check paints blue. Against 4DAnyone's views the fit trained on (every 8th of the 48 cameras, median over 121 frames; none held out, so read it as how well the fit converged, not accuracy from unseen angles), our replays reach 26.0 to 31.9 dB PSNR and 0.963 to 0.982 SSIM. Reconstruction runs at about a minute per frame per GPU once warm, so a 121-frame clip split across 12 GPUs takes roughly a quarter of an hour after the views are generated.

A few photos in: the avatar pipeline

LHM++ (photos in, a rigged 3D body out) built Vishal's 190,815-Gaussian body on the SMPL-X skeleton from 14 frames of his video. UIKA, which rebuilds a head from a set of photos, made his face; we rendered that head from 91 viewpoints and fit fresh, opaque Gaussians to the renders directly on the body's own head, so nothing is pasted on. On viewpoints held out of the fit, the distilled head matches UIKA's own renders at 29.0 dB PSNR, up from 18.7 dB for the head LHM++ guessed (a measure of how faithfully the head was carried over, not a likeness score). Blinks are fitted to his own eyelids through FLAME; a 20-second idle adds 2-3 mm of breathing at 0.25 Hz and small glances; eye contact turns his head 70% and neck 30%, up to 35 degrees to either side and 20 up or down, settling in about 300 ms. The browser poses every Gaussian each frame with dual-quaternion skinning.

Your room

The app is WebXR in the Quest browser, three.js with the Spark splat renderer, hands only, from one link with no install. People stand on your real floor through a local-floor reference space. WebXR plane detection reads the planes from the Quest's Space Setup: the chessboard goes on the plane labelled table (or any horizontal plane 60 to 95 cm above the floor; a lower coffee table gets our own table instead) with the chair on its far side, and with no table in five seconds we bring our own and say so. WebXR anchors pin people and the board through recenters and drift, and a persistent anchor brings the board back to the same spot next session. Your tracked hands and the detected table are drawn depth-only, so your fingers pass in front of him and his legs stay hidden under your real table; the session also asks for WebXR depth sensing, so where the headset provides it, real furniture can hide people too. A replay streams its 121 frames as SPZ, about 4 MB each, and starts playing once a quarter of them are in; Vishal is 18 MB. If you've been turned away from him or the board for 2 s, a small chevron near the middle of your view points the way back.

Games

Chess runs Stockfish 19 (single-threaded WASM) in a web worker at seven levels: 1400, 2000, 2400, 2700, Fischer's peak rating (2785), Carlsen's (2882) and full strength. The ratings are Stockfish's own UCI_Elo scale, calibrated at desktop speed, so they're a level, not a claim about the man. Tennis is our own physics at a fixed 240 Hz step, shared by live play, the opponent's predictions and the shot solver, so what the AI predicts is what happens. The 57 g, 6.7 cm ball follows

$$\dot{\mathbf v} = \mathbf g - k_D\,\lvert\mathbf v\rvert\,\mathbf v + k_L\,C_L\,\lvert\mathbf v\rvert^2\,\frac{\boldsymbol\omega\times\mathbf v}{\lvert\boldsymbol\omega\rvert\,\lvert\mathbf v\rvert},\qquad k_D=\frac{\rho\,C_D A}{2m},\quad k_L=\frac{\rho A}{2m},\quad C_L=\min!\left(0.32,\ \frac{1}{2+\lvert\mathbf v\rvert/(r\lvert\boldsymbol\omega\rvert)}\right)$$

with \(C_D = 0.55\) and \(\rho = 1.21\,\text{kg/m}^3\), so topspin dips and slice floats. Spin decays by a seventh each second in the air, and the grass bounce trades pace for spin through friction at the contact patch (\(\mu = 0.42\), low and skiddy). Federer's avatar moves with real tennis motion we captured from clips with GVHMR (ready, split step, shuffles, sprints, forehand, backhand, three serves), foot-locked so his shoes don't slide.

Speed

Splats cost screen area, not triangles. Per eye, per frame,

$$N_{\text{px}} = \sum_i 4k^2\,\sigma_{1i}\,\sigma_{2i}\left(\frac{f}{z_i}\right)^2$$

for Gaussian \(i\) with on-screen standard deviations \(\sigma_{1i}, \sigma_{2i}\) in meters, drawn out to \(k\) standard deviations, at depth \(z_i\), with focal length \(f\) in pixels. A full replay at 1.8 m is 116 million shaded pixels per eye. The headset keeps the 160,000 Gaussians that add most to the image (82 million), the session asks the Quest for 72 Hz, and a governor watches frame times the way a game's dynamic resolution does. After 0.4 s below 85% of the display rate it steps down through five levels of splat extent (\(k\) from 2.83 to 1.9), faint-Gaussian culling and foveation; after 5 s above 95% it tries one level sharper. A try that doesn't hold is undone within a second and doubles the wait before the next one, up to a minute, so a scene that only just fits never stutters every few seconds probing. Simulated on a scene whose sharpest level draws at 55 fps (scripts/check_governor.ts), it tries again at 7, 18, 38, 79 and 140 s, and 1.9% of frames run late over three minutes. On our Quest 3S, the ?fps=1 readout on the judged link shows a locked 72 fps at the 72 Hz the session asks for, in every scene: the opening, Vishal at the chess board, the replays, tennis and Court Side. At the headset's default 90 Hz (the main link) the same scenes run a steady 75 fps. Vishal loads as 18 MB. Add ?fps=1 to the link to see the frame rate, the p95 frame time and the governor's level live in the headset.

Voice, hands and sound

Talk: Whisper transcribes you, gpt-4o-mini writes a short reply from notes the person wrote about themselves (at the chess table, plus the position, the last moves, the evaluation and your last game), and ElevenLabs Flash speaks it in their cloned voice, 1.5 s to hear, 1.0 s to write and 0.5 s to speak. Its instructions forbid inventing memories, and replies are capped at 320 characters. If the voice service is ever down, the reply still arrives as captions beside them, never in a stand-in voice. Take the services away and the person still moves, blinks, looks at you and plays.

Hands: no controller anywhere. The menu appears on your left palm and you press it with your right index finger. At the board, a blue ring shows which piece your pinch will take; captures go into your hand until you hit the clock. Every reach and handshake is arm IK on the avatar's own skeleton, and a checker replays about 75,000 frames of reaches to all 64 squares, clock presses and whole games: no finger goes more than 5 mm into the board, a piece, the clock or the body.

Sound: no audio files at the table. Each piece set down gets a synthesized wooden clack in one of three pitches, the clock a soft click, and every sound plays from where it happened. Voices come from the speaker's mouth.

Challenges we ran into

Phantom limbs. 4DAnyone's generated person and the real one in the clip sit a few centimeters apart, and training on both gave people a second pair of shoes. Now each frame aligns the real camera to the generated person and counts real pixels only where the two agree: spill outside the person fell from 2.3% to 0.4% and view PSNR rose 0.5 to 2.1 dB.

Overdraw. The first full replay drew at 4 fps on a Quest 3S, because every Gaussian is a quad and a person a meter away covers the screen a hundred layers deep. The budget, the 72 Hz target and the governor are the fix.

Hands only, for real. In our first headset session the board moved whenever someone held a pinch in mid-air. We moved every control to the palm menu, anchored everything, and made sure no gesture does anything on its own.

Accomplishments that we're proud of

Any phone video becomes a person you can walk around, learn from, talk to and play, at their real height in your own room, from one link with no install.

What we learned

Realism comes from data that agrees more than from a better optimizer. Speed in a headset is about pixels covered, not polygons. And showing people what was generated makes them trust what was real.

What's next

Faster turnaround from upload to your room, sharper faces up close, more than one person at a time, and a Meta Horizon Store release in spring 2027. Before that release we'll swap the research-only parts (SMPL-X, GVHMR and LHM++'s weights are licensed for non-commercial use) for commercially licensed ones.

Built With

  • 4danyone
  • elevenlabs
  • flame
  • gaussian-splatting
  • gsplat
  • gvhmr
  • hand-tracking
  • hipergator
  • lhm++
  • meta-quest
  • openai
  • passthrough
  • plane-detection
  • python
  • pytorch
  • smpl-x
  • sparkjs
  • stockfish
  • three.js
  • typescript
  • uika
  • vite
  • webxr
  • webxr-anchors
  • whisper
Share this project:

Updates

Submission history