Our inspiration is a tale of our laziness on the couch with the need to throw trash away. We had a clear line of sight of the trash can, but it was closed. In that moment, we wondered about how we could make things easier with the use of technology. That's when we realized that the application space that intersected augmented reality (AR) and IoT devices had the potential to change the way humans interacted with surrounding objects in their daily lives. The ability to control objects from afar with simple hand gestures or physical buttons on a controller promotes a world of endless possibilities in making smart homes smarter, which in turn makes humans more efficient with their available resources. Thus, the ideation of Magician came into realization.

What it does

Magician is a proof of concept for an AR interface with IoT devices. Essentially, the AR headset is supposed to provide a way to interact with IoT devices wirelessly. Whether that be through turning things on or off, or activating custom functions in a few simple gestures, Magician makes the magic happen. We also created a dashboard to view stats about your current smart system and how many interactions it has had.

How we built it

We used the Magic Leap SDK and Unity to create a basic interface to interact with a dumb trash can in augmented reality. This dumb trash can is connected to a raspberry pi and servo to open or close the trash can. We made our trash can smart by connecting the raspberry pi with a Flask API to interface with Magic Leap event handlers when certain gestures are actuated during the AR session on the Leap.

Challenges we ran into

Unity is quite difficult to use and AR applications are challenging to develop. We ran into countless issues with getting our AR application to show up in the world that the Magic Leap device illustrated. Furthermore, development on the Magic Leap SDK was relatively difficult since we were running into issues with testing our product on Zero Iteration that slowed down our development process. We also had issues with setting up a headless raspberry pi that connected with all of our devices on the TreeHacks network.

Accomplishments that we're proud of

We can open and close the trash can from across the room on an AR device or across the world through our Flask API! We are also especially proud of making our crazy idea come to reality, when nobody on our team had any experience with developing AR applications.

What we learned

We learned how to setup a headless raspberry pi and developed an API that connected the pi with other incoming devices such as the Magic Leap. We also learned how to develop AR applications in Unity and deploy them on the Magic Leap.

What's next for Magician

In the future, we would like to allow Magician to interact with smart services, such as Amazon Alexa's and other smart devices. We would also like to enable voice assistance on Magician, so we are not constrained by gestures in the AR space.

Share this project: