Inspiration

Most of our best memories are stuck behind a rectangle. We replay old party videos and concert clips on a phone screen, always from the outside.

Putting moving people into 3D from one old phone video is still research, so we asked a smaller question: can we keep the video exactly as it was and rebuild the place around it?

What it does

Rewind puts you inside a moment from a video, using just your phone and a Google Cardboard viewer.

  • Party mode: you stand where the original camera stood. The original video plays in front of you, unchanged, and AI builds the room around it so you can look around.
  • Concert mode: we take fans' clips of the same song, work out where each person stood, and let you jump between their seats with the music staying in sync.

The room is generated, and the moment is real. You can look around, but you can't walk around, and the people are in the video, not rebuilt in 3D.

What it does

Rewind puts you inside a moment from a video, using just your phone and a Google Cardboard viewer.

  • Party mode: you stand where the original camera stood. The original video plays in front of you, unchanged, and AI builds the room around it so you can look around.
  • Concert mode: we take fans' clips of the same song, work out where each person stood, and let you jump between their seats with the music staying in sync.

The room is generated, and the moment is real. You can look around, but you can't walk around, and the people are in the video, not rebuilt in 3D.

How we built it

Party mode. The World Labs Marble API turns the video into a 3D room, which we load as a Gaussian splat in three.js with Spark. We project the original video back onto the room from the original camera's position, with faded edges so it blends in. Head tracking uses WebXR on Android and the gyroscope with a stereo fallback on iPhone.

Concert mode. We pull frames from several fans' clips of one song, then use COLMAP and nerfstudio (splatfacto) to rebuild the venue and recover where each camera was. All five clips registered together, so every seat has a real camera position. Looking at a marker for 1.5 seconds jumps you to that seat.

Syncing the clips. Every clip starts at a different moment, so we line them up by audio. For a reference clip r(t) and another clip c(t), we find the offset:

$$\tau^* = \arg\max_{\tau} \sum_{t} r(t)\, c(t - \tau)$$

and play every clip from one master clock:

$$\text{clipTime} = \text{masterTime} - \tau^*$$

This keeps the music continuous when you jump between seats.

The demo. We pre-rendered the walkthrough as a video with voiceover, so it plays the same way every time.

How we used AI Agents Collaboratively: We worked with AI agents across the whole team, and Jira and Confluence kept them and us lined up. We ran the project out of Jira, with one ticket per piece of work (KAN-4 to KAN-17), like picking the video, building the Marble room, the Cardboard view, the projection, the upload page, and the concert viewer. Each ticket had one owner, so everyone knew what they were building and which agent was helping with it. Our background notes and team split lived in Confluence, so every person and every agent worked from the same page. Handoffs went through the tickets. When someone finished their part, they moved the ticket and passed the files on. For example, the venue builder handed off the venue splat plus a cameras.json with each clip's position, rotation, and field of view, and the clip person added the audio offsets. On the AI side, we used Claude Code to write and wire up most of the viewer, and Cursor with Grok 4.7 to build the Grok features that find other fans' angles on X. Marble turns the video into a 3D room. We checked every AI output by running it and said plainly what was tested and what wasn't.

Grok Bot: We use Grok for image matching in our "Find more angles" page. The page takes three frames and a short stretch of audio from an uploaded clip and sends them to Grok. Grok searches X for other fans' videos of the same moment and compares the stage, lighting and crowd in our frames against what people posted. It sends back links to likely matches, which we can pin into the 3D venue as extra camera angles. Since these links are only suggestions, a person still checks that each one is the same moment.

Cursor: We wrote all of the code for the Grok features in Cursor, using Grok 4.7 as the model. In Cursor, we built the "Find more angles" page and the server endpoints behind it. The page sends three frames and a short stretch of audio from the uploaded clip to Grok. Grok then scrapes X for posts and videos from fans at the same concert. It cross-verifies each candidate against our clip by checking the stage, lighting, setlist moment, crowd, and audio, so it keeps only clips that match the same moment of the same show. The endpoints also check the search while it runs and return page links.

Challenges

  • Found footage is hard to use. To pin one camera pose per clip, a clip has to show the same moment, stay steady, not zoom, and overlap with the others. Most fan clips pan, zoom, shake, or are edited montages. We went through dozens of clips and kept five.
  • Syncing phones in a noisy crowd. The best matches agreed to about 0.02 seconds and the weakest to about 0.15 seconds. One clip had a 50 second cut, so it needed two offsets. A sign error once flipped all the offsets until we listened to mixes of the clips and heard the echo.
  • One pose per video only works if the phone stayed still. Candidate clips that moved had to be dropped.
  • Being honest about what's real. The AI-filled surroundings are generated, not recorded, and we say so in the demo.
  • Different phones behave differently. iPhone Safari has no WebXR VR, so it needs its own fallback.
  • Time. We had 36 hours, four people, two Macs and two NVIDIA laptops.

What we learned

  • Choose the footage before building the pipeline. The footage decides what's possible.
  • Check audio sync by listening, not just by trusting the numbers.
  • In VR, never move the camera for the viewer. Cuts and dissolves are comfortable. Motion with no head movement is not.
  • Say plainly what's generated. It builds trust and keeps you from overpromising.

What's next

Any moment that several people filmed, such as a game or a wedding, could work the same way. Next we want better rooms, more seats, and eventually moving people in 3D.

Team

David Densmore, Soham Deshmukh, Sri Srikanth, Neven Zurcher

Built with

three.js, Spark, World Labs Marble, COLMAP, nerfstudio (splatfacto, gsplat), WebXR, Python (NumPy, SciPy), ffmpeg, Node.js, Vite, Google Cardboard, Cursor, Grok Bot

Built With

Share this project:

Updates

Submission history