Memos

Client Rendering
Website Page
Website Memos

Inspiration

Imagine you are talking to your friends about your experiences in life: imagery and words limit how we express these lived moments. Memos elevates those experiences with a shared 3D space expanded from 2D videos taken with Meta glasses. That way, people can come together to share their cultures and experiences beyond words and encourage appreciation for one another.

What it does

Memos turns a 2D video from Ray-Ban Meta glasses (or camera roll) into a 3D memory you can walk through or experience on the web, phone, or in a Quest 3S VR headset.

You can capture the initial video easily through your Meta Glasses with our iPhone app, which uploads the clip automatically. Our GPU pipeline hosted on Runpod reconstructs the place at real-world scale. The scene is a 3D Gaussian splatting with a point cloud temporal overlay. These 3D models are stored in the database and can be fetched by the website, phone, or VR headset for viewing.

You can share these experiences with your friends by inviting them into your memories. Everyone in the group would join the same 3D scene in real-time, and you, as the leader of this group, can control the timestamp to be the curator of your own moments!

How we built it

At the core is the video-to-Gaussian Splatting pipeline hosted on Runpod. This pipeline works by first using NVIDIA ViPE to estimate the camera's path, lens, and depth for every pixel and predict a mask for every person and object. Then, we train this as a Gaussian splat of up to 500000 Gaussians. On top of this scene, we then overlay the per-frame 3D point clouds. The entire pipeline is hosted on Runpod H200 instances.

The viewer uses TypeScript, three.js, Spark, Rapier, and WebXR, where one web address serves the Quest browser, phones, and laptops. The multiplayer component is enabled by SpacetimeDB, where the shared world is one SpacetimeDB module, allowing multiple people to interact, sync, and simulate the same world in real-time. The capture and upload iOS app uses Swift and the Meta Wearables SDK.

Challenges we ran into

The pipeline of converting the 2D video feed into a 3D view is the most challenging part. 3D reconstruction and semantic segmentation are still open problems in computer science. There is no "best" way to do it. We used state-of-the-art methods such as pose estimation with ViPE to estimate the camera intrinsics and extrinsics, implemented semantic segmentation, and trained the Gaussian splats.

Serving the rendering and allowing real-time interactions was also a big problem. Thankfully, SpacetimeDB allowed this process to be done efficiently and easily.

Accomplishments that we're proud of

We are especially proud to have converted the 2D video into a 3D Gaussian splat that incorporates the moving elements in the 2D video. This 3D Gaussian splat is not just a static scene. It includes moving people and objects, making the experience much more immersive.

We are also very proud of the product that we have because of its versatility in being able to serve a wide range of experiences from VR to the internet to phones. The process of getting from the idea to execution required a lot of moving parts, and learning to navigate around them is a good learning experience for all of us, whether that is working on the backend, training the model, or considering customer-first principles when designing the UI.

What we learned

Across all our experiences, we learned how to combine many parts of a functional product into a working one. We also learned about the importance of planning out API endpoints and making sure the data moving across devices and applications is consistent. For most of us, we learned about how front end and back end come hand in hand with each other and the importance of having a working backend through unit testing.

What's next for Memos

We really liked this idea, and it provides a very good research point for developing algorithms that can produce better 3D models based on 2D videos. We believe that an idea like this can transform the way people can share experiences with each other and therefore build unity across many cultures.

We would like to thank the sponsors of Calhacks for providing the backend services and GPU compute time that allowed this project to come alive.