Inspiration

We were inspired by the sponsors SATIE, who were showing us their new powerful spacial audio tool for immersive environments. We decided we wanted to try and find a way to implement that into a project that uses real media. We settled on this idea as one of the members of the team has been doing research on neurodegenerative diseases and music therapy, and and believed that this oculd be a powerful tool for someone trying to relive memories.

What it does

MemoryScapes creates gaussian splats from an input video, accepting either normal and spacial video which is available with newer iphones. It uses each frame from this video to create a gaussian splat and displays it on the web. The audio goes through a source separation and localisation model to track where the sources are, allowing us to script their positioning and potential movement using Satie.

How we built it

We used FastAPI to create workers that we used for our pipelines. The website uses Next.js. We split up every essential process into separate tasks and worked in parallel, testing while going through. We then started testing integration part by part until we were able to put everything together.

Challenges we ran into

Two of us being on Macs was a bit of a problem, the gaussian splats getting generated required having a NVIDIA GPU, which only one of us had at his house. Because of this we couldn't properly start testing until the second day, meaning we had to have a lot of faith in our ability to properly compartmentalise our tasks so that we can work in parallel and make progress everywhere we could before having to start testing. We were able to later deploy to website for that teammates PC, and remotely access it to pull new changes.

Accomplishments that we're proud of

We're proud of getting it running on a deployed server, this was essential to use being able to keep working on it and end up working out really well!

What we learned

  • Spatial cameras help. An iPhone spatial video's second lens gives the scene's true scale, twice the training views and per-frame depth, all without extra filming effort.
  • Phones shift exposure as you film. Learning a small colour correction per frame stops the splat from baking those shifts in as wrong colours and floaters.
  • Render it the way it was trained. Matching the viewer's blur to the trainer's made the same splat look sharper, and streaming levels of detail keeps it smooth on phones.
  • Design is iterative. The globe went from a glowing night Earth to a cute low-poly world through many quick mockups.

What's next for MemoryScape

  • Users and sharing: real login, so the app recognises each user, with private and public memories. Faces and number plates get blurred before anything goes public.
  • Production deployment: Postgres for the database, S3 for storage, and hosting on servers with GPU workers so many people can upload at once.

Built With

Share this project:

Updates

Submission history