With the advent of the pandemic, event coordinators have had to say hello twice, once to those in person and then to those in zoom. For those stuck in the virtual environment seeing events take place in person can be isolating. What if there were a way to be there without being there? This was the inspiration for Leap Here, they want to be somewhere.

What It Does

Leap Here allows a user to navigate through a 3D world created using LiDAR through a Leap Motion. One is able to “leap” to anywhere they would like as the LiDAR file can be changed to any scanned location.

How we Built It

Leap Here was built using Unity, Leap Motion, LiDAR, C#, and various APIs from the Leap Motion C# documentation. Unity was used as the main platform to integrate our Leap Motion sensor with the LiDAR scan of 3D renderings that were captured. Through Unity, we built the interactive platform for Leap Motion to use hand gestures to navigate through the LiDAR 3D renderings and smoothly flow through the rooms. First with Unity, Leap Here was set up the 3D space to navigate through importing point cloud files of the LiDAR images. Then, the Leap Motion aspect was incorporated through addition of Leap Motion Assets and modules. With calibration of the Leap Motion in the 3D LiDAR environment, Leap Here set up the hands in real time on the scene. The next aspect focused on was controlling the LiDAR environment with hand gestures through the Leap Motion sensor. VSCode was used to code the functions of each hand gesture in C#. The right hand was coded to control the front, back, and side-to-side movement and the left hand was coded to control rotation. Our logo was created using LogoMakr. Through a combination of all these tools on Unity, Leap Here provides an interactive model through any 3D world with just the touch of your hands!

Challenges we ran into

Many challenges were faced as the project as the team was new to Unity, Leap Motion, and C#. The biggest challenge we faced was with the initial setup of Leap Motion. Our programmers and engineers were most technically adept in Java and Python, so we were trying to incorporate the Java or Python version of Leap Motion in order to code the hand gestures and control it. However, through experimentation on various computers, IDEs, and manipulation of code, we soon realized that the versions we were trying to use were mostly out of date or lacking the proper documentation. Once realizing this and getting guidance from a Pinnacle mentor, our team navigated to Unity for cross-platform integration. On Unity, we set up the LiDAR rendering and incorporated Leap Motion, but soon realized we had to code in C#. Using our knowledge in Java, we worked to learn how to implement our previously written code from Java to C# through tutorials and documentation. We also faced many issues with coding the hand gestures and getting the Leap Motion to properly move the screen based on where both hands are. There were many configuration issues since the local reference point and the global reference points were different which caused confusion in the code. However, through debugging and aid from a mentor, we fixed the reference issues.

What's next for Leap Here

The applications of Leap Here do not end at the Pinnacle Conference. Since any space can be scanned and mapped using LiDAR, any location humans want to go can be replicated. Specifically for the travel industry, hotels and companies like AirBnB can give their customers a 3D walk through of their rooms. For construction sites, abandoned buildings, or other dangerous places, a LiDAR camera can be attached to a drone, and the site can be scanned without a human needing to be present. This scan can then be used within Leap Here to have people inspect or travel to these places without fear. For inspection purposes, hard to reach nooks and crannies can be scanned then inserted into Leap Here to get a better view of what is going on. Truly the possibilities for Leap Here are only limited to ones imagination.

Built With

Share this project: