Headset-free Pseudo 3-D Experience

A perfect hit in example game
Example game screenshot
Some code from tracking
Ad example

Inspiration

A VR headset can be an expensive, messy, hard to maintain function of ones work or personal interests. We were inspired to find a solution.

What it does

This is an intuitive 3-D interactive environment, without the VR hassle.

The software projection maps an environment to the screen based off the perspective of a user. If a user is looking straight at a cube and walks right, the simulation will 'move' as well exposing the right side of the cube, simulating perspective.

The users hands (and feet) are tracked and rendered onto the screen, allowing the user to press buttons, hit beach balls, and more.

Extrapolation

The project as a whole is a collection of useful assets and demos to create your own applications. Potential use cases for this product include digital signage, education, and a lot more. It is very easy to get up and running with the code that we provide.

For example, this project could be used for eye catching advertisements. As you pass by a TV on the wall, it acts as a virtual display case. This could also be implemented as a game, allowing users to hack and slash their way through a hoard of beach balls.

How we built it

Using the Microsoft Kinect V2, and a simulated environment in Unity, the program finds the best suited human and begins to track them. The tracking relies on the Kinect API at a low level, but the "3-D" abilities of this program come from code written in C# by us.

Challenges we ran into

Calibrating the FOV and camera settings was the worst challenge. It was also difficult to time the beats correctly.

Accomplishments that we're proud of

It works amazingly well for being made in one weekend, and the 3D effect is much more convincing than we were expecting.

What we learned

Hogan learned a lot about the Unity engine as a whole, especially about 3D modeling, animation, and effects that can be applied. He also learned to use C# for the first time.

Matthew was re-familiarized with Unity after a long break and probably wrote better c# than 90% of the c# he has written in the past. He got do deal with a lot of super fun bugs in the Unity Linux beta.

Caleb learned about projection mapping and the Kinect API, and that it is definitely possible to sleep on cement if you are tired enough.

What's next for Headset-free Pseudo 3-D Experience

This technology could be used in digital signage, education, and other interesting applications. The code is all open-source, so it is now up to the community to see what they come up with.