Our initial motivation was to build an Augmented Reality headset prototype using a RaspPi4, depth camera, transparent OLED, and bluetooth headphones!

What it does

The purpose of the device was to recognize hand gestures in order to play, pause, and skip audio tracks, as well as increase/decrease volume for music on the go.

How we built it

First we wrote code to recognize hand gestures from a hand pose model on a live video feed and output the action that we needed in text once the appropriate gesture was recognized. Next, we created a database to sync these action to our Spotify API in order to control our music player. Finally, we physically attached our depth camera to the RaspPi4 using velcro and connected our transparent OLED to the Pi using a Qwiic cable.

Challenges we ran into

Once we had the hand pose models and gesture recognition running on our PCs in Javascript, we noted that the RaspiPi4 couldn't run the code. We weren't sure whether to proceed with Tensorflow or start from scratch using OpenCV. We spent several hours trying to figure out the difference between the two and how to combine them. We were also unable to connect the RaspPi to our headphones via bluetooth which we thought would be trivial.

Accomplishments that we're proud of

-Recognize hand gestures and use this data to control an MP3 player (play/pause, skip, volume up, volume down). -MP3 player ran the Spotify API for an infinite selection of instant music. -Connected Intel RealSense D435i (depth camera) to Raspberry Pi4 for advanced gesture recognition. -Create a GUI for a transparent OLED display to interface with our Rasp Pi4.

What we learned

We mainly learned about different image processing techniques. Before the hackathon, we thought of object detection as a general concept. As we built our prototype, we realized that OpenCV is a library for computer vision and Tensorflow is a framework for machine learning. Although at a high level they appear to accomplish the same task, their specific toolkit is highly specialized.

What's next for Augmented Reality Headset

We will definitely continue working on this project! Today we ran out of time, but all team members want to learn how OpenCV combines with Tensorflow to extract meaningful position data from the environment. In the future, we will find a way to see 3D objects via our transparent OLED and interact with them using our hand gestures.

Share this project: