HoverMouse

Inspiration

The inspiration comes from the fact that some people need to draw their illustrations during interviews or technical meetings, such as drawing a binary tree structure or a mathematical equation. However, mouse or trackpad are not natural as a drawing tool to humans, and not everyone has a tablet or paper and pen to draw on. Thus, we have thought of creating a new way so that people can draw and navigate their cursor in a natural way with great precision.

What it does

The HoverMouse is a system-level control app that allows user to control their cursor based on user’s hand gesture. User can move mouse around, do left/right click, scroll, and even take a screenshot using different gestures. The most interactive way is to open up a drawing pad and start drawing. User can also use the cursor in a regular way, such as open a new browser tab or select a file and delete it.

How we built it

We built the HoverMouse based on the open-source library MediaPipe, which does a great job in detecting hand gestures. The MediaPipe library maps 21 points on each of the hand it detects and we can program the specific patterns of gesture using the positions of those points. For controlling the cursor, we used Pynput to perform all the basic cursor actions.

Challenges we ran into

The main challenge is to make the control as precise as possible. It is hard to stabilise the mouse position when performing clicking whose gesture is touching the thumb with the index finger tip. Because the index finger is moving, the mouse might move a little bit when clicking, resulting in dragging rather than clicking. To resolve this, we have performed different noise reduction strategies and re-designed the gesture control of each cursor actions. Another challenge is to make sure the aspect ratio of the webcam and the screen are aligned properly.

Accomplishments that we're proud of

We are able to precisely open or close a tab, which involves in clicking really small buttons of x and +. We are able to scroll as well, and take a screenshot. The most interesting and exciting use case is to play Quick Draw using HoverMouse.

What we learned

We have learned how hand gesture or even poses, iris detections, can be achieved, as well as how to design the detection of different gestures. We also learned how to control cursors with native libraries using Python.

What's next for HoverMouse

We plan to add more cool effects and more gestures, such as swipe left and right. We can also make some good use cases in playing games as well, e.g. motion-based control games

Built With

• mediapipe
• pynput
• python