This project was inspired by the show "Avatar: The Last Airbender". We thought it would be fun to control elemental powers with our hand gestures, so we decided to simulate it on the Hololens augmented reality platform. This was our first time using the technology, and we had to reshape the idea in our brainstorming process. For example, we found that we couldn’t override certain gestures, such as holding a palm face up. Once we got an idea of the constraints of the device, we were able to better adjust our project. We chose the Hololens because of its unique ability to conjure illusions with our bare hands.

What it does

BendAR is a HoloLens AR simulation that gives the power to conjure and shoot fire and water with your bare hands, and then tests element mastery against turret attacks. Turrets will aim and shoot the player, who can dodge the attacks and destroy its bullets and the turret itself. Just put on the HoloLens, and put your hand into the field of view to make gestures that conjure up fire and water (pointer finger up, tapped) and shoot them (release/throw gesture).

How we built it

We used Unity and the HoloLens SDK for development, Git and GitHub for version control, C# for the coding and Blender for the 3D models.

Challenges we ran into

HoloLens is still very new, so there is not much support and documentation out there to fix its' issues. It took the first 4 hours just to set up a HoloLens development environment. We randomly got errors with deployment where we had to restore to an alternate version. We were stuck on an error (Visual Studio CS0006 Metadata File) that showed up out of nowhere at 10 PM and was fixed at 1 AM. At one point we messed with our fireball prefab, which ended up breaking and took over an hour to restore. Helen's desktop with our turret model and scripts randomly shut down at 2 AM and finally turned back on at 3:30 AM. At this point, no error is a surprise, but every error is a heart attack.

For our own issues, creating fire and water that looked good and properly scaled was difficult. There is no way to feasibly create unique gestures, so we had to think of ways to make conjuring/shooting elements fun with only simple (and wimpy) tapping and pinching gestures.

Accomplishments that we're proud of

We made a game in HOLOLENS. It was incredibly difficult to fix so many software issues, which made us wonder how much we could develop by the deadline. But in the last 8 hours, we managed to pull through and accelerate development from a simple tracking fireball to shooting fire and water at turrets while dodging!

What we learned

We learned how to develop with the HoloLens/Mixed Reality SDK, create custom particles and materials for our "elements", and apply physics to our elements.

What's next for BendAR

We would like:

  • more elements
  • better particle and sound effects
  • smoother tracking
  • a way to "charge up" attacks
  • endless mode with many turrets
  • improved turret model and behavior

Final note: the project's ZIP file is larger than the maximum 35 MB allowed, so please access it through the Github repo. Or follow the Google Drive link below. Thank you!

Share this project: