Inspiration
Coming out of high school, some of us went to the same universities, while others ended up hundreds or even thousands of miles apart. For years, every day was spent together, suddenly stopped by the advent of college.
It's a natural part of life to experience separations from people you love, but how do you make the most of your new experiences, new college, new life, while staying true and close to your old friends who held you through your past life. And for most people, the cycle is also the same. You promise to stay in touch, you text and call almost every day for the first week, but then slowly, and unintentionally from both parties, communications other than one or two people naturally stop.
This isn’t just limited to friends unfortunately, it can also happen to families too. We move away from our parents, grow apart from our siblings, and find ourselves more connected to our devices than any of the people who helped us become who we are.
But that same technology is what doesn’t make sense. We've made incredible advances in how we connect with the world, yet for decades, the way we stay close to the people we care about has remained largely the same: texting, calling, and occasionally video chatting. While these tools let us catch up on each other's lives, they don't necessarily let us experience life together. There's only so much you can capture in a "How have you been?" or a quick FaceTime call. As our lives move in different directions, conversations become less frequent, shared experiences become memories, and people who once felt like an everyday part of our lives slowly become someone we just catch up with.
We have thousands of photos and videos scattered across our camera rolls and social media, preserving memories of the people and places that brought us together in the first place. Yet most of these moments are left forgotten, only occasionally revisited when we stumble across an old photo and wish we could go back.
Maybe it's a random afternoon with your closest friends, a childhood memory with your family, or one last moment with someone you've lost. We've all experienced that feeling of wanting to return to the past, even if just for a few minutes.
And that's what got us thinking. What if these memories could become more than just pictures on a screen? What if we could actually step inside them, explore the places and moments that shaped us, and bring the people we care about along with us?
That's what inspired us to build a way to transform ordinary photos and videos into immersive, explorable 3D worlds, allowing us to reconnect through the memories that brought us together.
What it does
Our product brings together photos and videos from platforms like Google Photos, Instagram, and other sources users choose to connect, transforming their memories into immersive, explorable 3D environments that they can revisit and share with friends and family.
Instead of having thousands of memories scattered across camera rolls and social media platforms, our product helps bring them together in one place. Using Gemini, we analyze and organize these memories into meaningful events, whether that's your high school graduation, a family vacation, or a random weekend with friends.
From there, our 3D reconstruction technology takes ordinary 2D photos and transforms them into interactive environments you can actually step inside and explore. Rather than just looking at a picture of your childhood home or favorite vacation, you can walk through a reconstructed version of that moment, experiencing it from an entirely new perspective.
But what makes our product special is that you don't have to experience these memories alone. Through real-time multiplayer and immersive VR capabilities, we aim to let friends and family enter the same memories, explore familiar places, and interact with one another, regardless of how far apart they live.
Rather than simply preserving memories, our product turns them into new experiences we can share with the people who made them special.
How we built it
We built our product using React, Vite, and TypeScript, with a Node.js 22+ backend coordinating our AI processing, data ingestion, and 3D reconstruction pipelines.
Google Gemini played a central role in making our memories intelligent. We leveraged its multimodal capabilities to analyze images and videos, understand their contents, generate descriptions, and organize memories into meaningful experiences. This enables users to rediscover moments based on what happened in them rather than manually searching through thousands of photos.
For data integration, we combined the Google Photos API with Apify-powered social media ingestion. Google Photos provides access to media explicitly selected by users, while Apify gives us a flexible way to extract content and metadata from additional platforms. Together, they form the foundation for bringing memories scattered across different services into one unified experience.
Our 3D reconstruction pipeline combines computer vision with AI-powered scene understanding. We use Depth Anything V2 to estimate spatial depth from individual images, then process that information into explorable 3D geometry. Gemini's visual understanding provides an additional avenue for interpreting objects, spatial relationships, and scene context. We render the reconstructed environments using Three.js, enabling users to move beyond the original photograph's perspective and explore their memories interactively.
SpacetimeDB powers the real-time multiplayer experience. Its real-time synchronization capabilities allow us to manage shared rooms, participant presence, and player movement, creating a foundation where multiple people can explore the same reconstructed memory together. We also incorporated WebXR to support immersive VR experiences.
By combining Gemini's multimodal intelligence, Apify's data integration ecosystem, computer vision-based reconstruction, and SpacetimeDB's multiplayer infrastructure, we brought together technologies that turn static memories into immersive, shared experiences.
Challenges we ran into
By far, our biggest challenge was reconstructing 3D environments from videos. Unlike individual photos, videos introduce constant camera movement, changing perspectives, and variations between frames, making it incredibly difficult to generate a consistent, explorable environment. Preserving spatial accuracy while keeping the reconstruction process computationally efficient was especially challenging, and integrating these reconstructions into a smooth, interactive experience added another layer of complexity.
Even transforming a single image into an explorable 3D environment presented significant challenges. Accurate depth estimation was essential, since even small inaccuracies could lead to distorted objects, incorrect proportions, and unrealistic environments. Beyond estimating depth, we had to reconstruct the scene's geometry, account for perspective, and generate a usable 3D mesh, all while minimizing computational costs and maintaining visual quality. Balancing accuracy, efficiency, and immersion throughout this process was one of the most technically demanding parts of our project.
Accomplishments that we're proud of
Bring people together, finding a way technology can improve the way we connect
We're especially proud of successfully turning ordinary 2D images into immersive 3D environments that users can actually walk through and explore. What started as an ambitious idea became a working experience where memories are no longer limited to photos on a screen. Getting our reconstruction pipeline to work alongside real-time 3D rendering and interactive navigation was a huge milestone, especially within the limited time of a hackathon.
Beyond the technical side, we're proud of bringing together so many different components into one cohesive experience, from processing personal memories to making them explorable. Most importantly, we're proud that we were able to take a problem we personally connected with and build something tangible around it, showing how emerging technology can help people reconnect with their past and the people who made it meaningful.
What we learned
We gained a much deeper understanding of depth estimation, mesh generation, and how to handle tradeoffs between computation and reconstruction accuracy. We also learned how to stack and integrate multiple technologies onto each other and use all of their capabilities to support our final product.
What's next for The Palace of Fine Memories
Our biggest technical priority is moving our reconstruction pipeline to cloud-based GPU infrastructure, allowing us to process larger videos, generate higher-quality 3D environments, and dramatically reduce the computational burden on users' devices. We also want to expand integrations with more social media platforms, improve reconstruction accuracy, and make multiplayer VR experiences seamless across different devices.
But beyond the technology, we want to make revisiting memories a social experience people genuinely look forward to. We envision families exploring old vacations together during game nights, friends reconnecting by walking through their high school memories, and people sharing reconstructed moments with loved ones across the world. We want to build excitement around the experience of rediscovering your past, turning something as simple as opening an old photo album into an immersive activity everyone can participate in.
Ultimately, our goal is to make exploring memories together as natural and exciting as sharing photos is today.
Log in or sign up for Devpost to join the conversation.