Manouse -- Webcam mouse

Inspiration

One of our team members wanted to make an alternative mouse input to play osu!, a rhythm game, without a mouse. We figured the webcam would be a fun place to start!

Manouse is a portmanteau of "manus", the Latin word for hand, and "mouse."

What it does

Manouse uses computer vision technology to map hand gestures to mouse control. We use OpenCV to create an outline of the hand, find it's position, and count the number of fingers. The position and number of fingers data can be mapped to various actions. Currently the position moves the mouse and finger count toggles clicking.

How we built it

We used OpenCV, win32api (to move the mouse), and Python

Challenges we ran into

We had to filter out faulty readings of the number of fingers, and develop robust algorithms and functionalities to compensate for inaccuracies.

Accomplishments that we're proud of

We are all first year Cal students, with the majority of us having little to no programming experience before this Fall.

What we learned

We learned how to use OpenCV to process frames captured by a computer's webcam, and using powerful algorithms, resolve meaningful information from the frames. We also learned how to use Python's win32api to control mouse position, mouse movement, and other core PC functionalities.

What's next for Manouse -- Webcam mouse

We look forward to expanding it's functionality, making a more friendly user interface, and improving it's hand detection and background filtering solutions.

Built With

opencv
python
win32api

Submitted to

Cal Hacks 3.0

Created by

I worked on using OpenCV to find the most appropriate/efficient method to filter out the background, extract the hand, and count the number of fingers from each of the webcam's frames. This functionality was then sent to our other teammates to implement other features (mouse control, demo game, etc.)

Rustie Lin
Blockchain Educator, Developer, Researcher; UC Berkeley CS
I worked with Rustie on the algorithms that would interpret the images from the webcam -- see his contribution for specifics.

jromagn
I applied our sensor algorithm to create a basic rock, paper, scissors game. The traditional gestures used in the game were recognized by the algorithm and OpenCV was also used to modify and display victory/defeat images.

Andy Zhang
I used Open CV to implement the code to identify and track hand movement and gestures. I also helped Andy with his implementation of Rock, Paper, Scissors.

Jonathan Kim
I worked on the function of the mouse, representing it as an object and creating methods to have the mouse move and interact with the screen according to the input from the camera.

Nadir Akhtar
Student at UC Berkeley. Developer and VP of Operations for Blockchain at Berkeley: Consulting.