MemoraVision

Building the headset
Our first hologram trial run
Testing out industrial applications

Inspiration 💡

Imagine a world where the confines of 2D imagery dissolve, where static pictures spring to life in captivating three-dimensional forms. Our inspiration for MemoraVision emerged from the desire to bridge the gap between the visual allure of two-dimensional snapshots and the immersive depth of holographic reality. In every photograph, there's a story waiting to be told. A family portrait holds cherished memories, a historical snapshot encapsulates a moment in time, and a scenic view captures the beauty of the world. What if we could elevate these moments, not just by viewing them but by stepping into them, by making them tangible and dynamic? This vision spurred our journey to create a tool that unlocks the hidden dimensions within every picture.

What it does 🔬

MemoraVision taking a single 2D image and, through an innovative approach, transforming it into a mesmerizing 3D masterpiece. This advanced method is designed to deconstruct a two-dimensional image, anticipate the placement of 3D vertices, create a colored 3D mesh, and apply textures, elevating ordinary images into immersive 3D environments. We then take this 3D creation and project it as a hologram on our headset and XR setup, providing an unparalleled, immersive visual experience.

How we built it 🔨

Our method weaves together three key components: multi-view synthesis, camera pose estimation, and 3D reconstruction. First, it utilizes a clever tool called Zero123 to generate multiple views from that one image, giving us a comprehensive understanding of the object. Then, we refine the camera's position and angle using these new views, like fine-tuning a camera's focus. Finally, a specialized neural network sculpts these estimated poses into a 3D mesh. We use refraction properties to build the hologram headset.

Challenges we ran into 😥

Firstly, the multi-view predictions generated by the 2D diffusion model exhibited inherent inconsistencies, which could lead to significant failures in the 3D reconstruction process. Secondly, traditional optimization-based methods were time-consuming, involving thousands of iterations for full-image volume rendering and prior model inferences. To tackle these challenges, we devised a solution that combines a 2D diffusion model with a cost-volume-based 3D reconstruction technique. This innovative approach allows us to create a high-quality 360-degree textured mesh from a single image in a single, efficient pass. To fix the multi-view problem, we pivoted to using relative spherical coordinates instead of preset angles.

Accomplishments that we're proud of 🥇

The machine was originally designed for linux but we coded it locally and ran the software on our computers.
The accuracy and clarity in the 3D models and their projections onto the headsets and as holograms.
Forming the headset by cutting the plexi glass and figuring out the exact angles to perfect the projections.
Crafting the entire model on HuggingFace without prior experience.

What we learned 📚

Throughout our journey, we've learned the invaluable art of adaptability and quick thinking. Early on, we encountered a significant roadblock when Linux failed to run as expected. Instead of letting it derail our progress, we swiftly pivoted, to Hugging Face to deploy our model. Another challenge we faced was the "Janus problem," where our renders seemed to have two heads, but we tackled it head-on by switching to spherical coordinates. The underlying theme was, we realized the importance of thinking on our feet and efficiently applying different tech stacks to address the ever-evolving needs of our project. And let's not forget the unexpected MVP of our project: Red Bull, the secret sauce that transformed our tired team into coding machines.

What's next for MemoraVision 🔮

Our goal for Memoravision is to further develop and refine our hologram headset technology for industry applications. This innovative device promises to revolutionize various sectors, including healthcare and construction. By transcending the limitations of physical distance and fostering global collaboration, our technology is set to catalyze a brighter, more connected future for these industries. We plan on using a system processor instead of running the program from our local computers towards the mid term. Our long term goal is to allow users to interact with holograms in real-world environments, expanding the application to fields like education, architecture, and remote collaboration.