Memories Restored

Inspiration

Memories Restored was inspired by the idea of parallel universes, worlds that share the same physical structure, yet tell completely different stories. We wanted to explore how actions in one reality could affect another, and how ordinary objects can carry memories powerful enough to restore places that have been forgotten.

The experience plays with the contrast between abandoned, lifeless spaces and the vibrant memories they once held, allowing players to bring those memories back through meaningful physical objects.

What it does

Memories Restored is a mixed reality experience that overlays AI-generated worlds directly onto the physical hackathon venue.

Guided by a narrator, players use a virtual flashlight to uncover fragments of a hidden parallel world aligned with their real surroundings. Discovering key points reveals larger sections of a desolate version of the space.

Players then interact with real-world objects to restore these lost environments. Placing and scanning the correct physical object causes it to transform within the virtual world, triggering the environment around it to come back to life. An abandoned bar becomes vibrant and occupied again, while another forgotten space transforms into a warm bookshelf lounge with someone relaxing beside it.

Each restored scene reveals a glimpse of the memories that once inhabited that space, blurring the boundary between the physical venue and its alternate reality.

How we built it

We created a rough photogrammetry mesh of the physical venue and used Marble to generate initial panoramas and alternate versions of the environment.
We fed these panoramas into Seedance to generate continuous environmental transitions, sampled frames from those videos, and brought them back into Marble to generate spatial splats that maintained a consistent physical layout.
We imported the generated splats into Unity and developed a scripted system that uses them as sequential frames, creating dynamic transitions between different states of the world.
We used Multiset's VPS and object detection systems to recognize physical objects and trigger events within the experience. Once the correct object is detected in the correct environment, virtual content is aligned with it and the corresponding world restoration sequence begins.
Characters and props throughout the experience were generated using Tripo AI.

Challenges we ran into

One of our biggest challenges was creating convincing transitions between multiple generated splats while keeping them spatially consistent with the physical environment. Integrating several complex systems at once also introduced significant performance challenges, requiring a large amount of optimization to keep the experience running smoothly.

Developing a short but meaningful narrative within the limited timeframe was another challenge. Using ChatGPT to rapidly iterate on the story and narration, alongside ElevenLabs for voice generation, allowed us to move from ideas to finished narrative audio much faster.

Accomplishments that we're proud of

We're especially proud of how many experimental technologies we were able to integrate into a cohesive experience in roughly one day of development.

We created dynamic AI-generated splat worlds that maintain visual and spatial continuity with a real physical venue, built transitions between multiple states of those worlds, and integrated VPS and object detection so that interactions with real physical objects could directly trigger events and transformations within the mixed reality experience.

Most importantly, these systems work together as parts of a single narrative rather than as isolated technical demos.

What we learned

Scope matters a lot, especially at a hackathon. We pushed several experimental technologies simultaneously, and a smaller scope would have given us more time to polish the final experience.
Splats in Unity are still challenging to work with. Performance and stability became major considerations, and we believe a WebXR-based approach may be worth exploring for future iterations.
AI generated environments become much more compelling when grounded in physical space. Maintaining spatial continuity between generated worlds and the real environment made the transformations feel much more meaningful than simply switching between unrelated virtual scenes.

What's next for Memories Restored

We want to take the technical foundation we developed for Memories Restored and explore how world models, generative AI, spatial computing, and physical locations can come together to create new kinds of location based entertainment.

Future versions could transform real venues into dynamic, AI-generated worlds while preserving their underlying physical architecture, allowing museums, historical sites, entertainment venues, and public spaces to become gateways into alternate time periods, memories, and parallel realities.

Rather than building virtual worlds that replace physical spaces, we want to explore experiences where the physical location itself becomes the foundation for an entirely new world.