reach out and i will be there

Inspiration

Having both worked extensively with HMD-based XR, we were drawn to working with the Looking Glass as a headset-free 3D device. We liked the idea that the Looking Glass could exist as a "charmed object" that lives in your home and invites low-commitment XR interaction without complicated setup.

Reflecting on the overwhelming loss and separation experienced by people across the world during the past 2 years of the COVID-19 pandemic, as well as our personal experiences of the loss of a parent and long-distance separation from family, we envisioned an asynchronous way to experience connection with an inaccessible loved one that evoked a stronger sense of presence than a video or photograph.

What it does

When the user reaches their hand towards a field of dynamic particles floating inside the Looking Glass, the particles converge towards the user’s hand, then rediverge into a fluid point cloud representation of the user’s Loved One. The user can subtly interact with the particles that make up the ethereal representation of their Loved One, causing visual shifts that create a sense of embodied contact not possible with a traditional photograph, video, or even synchronous video chat.

How we built it

In our current prototype, the Loved One is represented with a short video clip that includes depth information captured with an approachable mobile volumetric capture app called Record3D. Using Keijiro Takahashi’s RSVFX project, the developer of Record3D, Marek Šimonik, created a Unity project that powers VFX graphs with volumetric capture footage from his app. We used this as the starting place for our project, and then brought in the Looking Glass and Leap Motion SDKs. We then redesigned the VFX graphs to optimize for the Looking Glass display and built hand tracking interactions with the particles.

Challenges we ran into

Getting all of the separate systems to play nicely with each other was, as always, a challenge. In order to use VFX graphs in a Standard Render Pipeline project (required for the current Looking Glass SDK), we added a package provided to us by the Looking Glass team. One of us developed on a new Silicon chip Mac, which caused some difficulty with importing a required dynamic library.

Accomplishments that we're proud of

We were excited for the challenge of working with numerous tools that were new to us, and it was very satisfying to fuse all of the moving pieces into one refined, holistic experience. We set out to the push the boundaries of what has been done before with the Looking Glass, and we’re proud of a result that is deeply custom-built for its particular medium.

What we learned

We learned how to work with the Looking Glass and the Leap Motion, as well as with the Record3D app - it was our first time working with all of these!

What's next for reach out and i will be there

We chose to focus on asynchronous volumetric video for this prototype, but we also set up video streaming from the Record3D app into the point cloud on the Looking Glass, and we’d be interested in exploring a synchronous streaming version of the project, as well as pairing interaction between two Looking Glass devices.

Thank yous

Thank you to all the folks from Looking Glass who helped out at the Hack, especially Oliver and Bryan, to Marek Šimonik for excellent and speedy Record3D support, and to Peter Szolovits for being a Loved One.

Built With

• c#
• leap-motion
• looking-glass
• record3d
• unity