Handy Mouse

We have created a futuristic way to interact with your computer - control your device by simply moving your hand through the air, no VR headset or controller needed!

Comment

Inspiration

Inspiration for this project came from a general interest in computer vision and wanting to make something (somewhat) practical that after graduation would be a great conversation starter that could be used in a home or apartment for fun.

What it does

The product allows a user to control a Raspberry Pi with only their hands (rather than holding something like a mouse). The product watches the user and uses OpenCV to track the location of their wrist, which is then mapped onto a location on the screen where the cursor is placed. The product also allows the user to click the screen by closing their hand.

How we built it

To track the location of the wrist, we use OpenCV and Mediapipe to determine the location of the hand using a Mediapipe pre-trained model. The location of the cursor maps onto the wrist landmark's location within the camera frame, with some portion of the edge of the frame cut off to prevent the model from losing track of the hand at the edges of the camera frame. We use a Tensorflow pretrained model to identify the gesture that the hand is making and PyAutoGUI to actually click on the UI when the model detects the user is making a fist.

Challenges we ran into

The first challenge that we ran into was figuring out how to access specific landmarks within Mediapipe. The documentation for Mediapipe is relatively extensive and sample code is given which lets you start tracking your hand on screen relatively quickly. What was more difficult was figuring out how to get the location of only one landmark and how to ignore all of the others so that it ran much faster. Another major challenge was setting up the hardware so that it could run the code that we first wrote on our laptops. Downloading OpenCV and TensorFlow was particularly difficult, as we could not simply pip install the libraries. Instead, we needed to download the source code and compile each of them which took several hours each. We also ran into the challenge of having two versions of python on our Pi, and only the older version could see OpenCV on our Pi. Finally, we ran into issues of two corrupted SD cards throughout the project, which were major setbacks.

Accomplishments that we're proud of

We are proud that we were able to create something that the class seemed to enjoy and that can be used in our daily lives. We faced several challenges on the hardware and software side, and overcame (most of) them with determination and hard work.

What we learned

We learned that there are some incredibly powerful deep learning tools that are available to use by complete novices. We learned that installing software on the Pi is an absolute nightmare and that several idiosyncratic issues arise, especially with different versions of software and hardware.

What's next for Handy Mouse

Next, we would like to allow Handy Mouse to work over a local network. We would like to allow devices to have access to Handy Mouse's output so that any device can connect to the Pi running Handy Mouse and use it as an alternative peripheral over Wi-Fi. We would also like to find ways of smoothing out the movements mouse when being controlled by Handy Mouse, as it is a bit finicky in its current state.

Built With

Share this project:

Updates