HandyTrack

keyboard

Inspiration

Our team's goal for "HandyTrack" was to create an alternative way to interact with devices. Keyboards, touchscreens, and even voice control have become common input methods.

These input methods have one thing in common—they are all discrete devices (besides voice control) that users interact with, typically using their hands. With computer vision and AI as accessible as it is today, and our hands already central to how we interact with the world, why not use our hands themselves—gestures, to directly control devices? Why continue relying on traditional input devices when we can tap into the natural movements of our hands?

What it does

HandyTrack detects certain hand gestures that can be mapped to any arbitrary command. The gestures provide data that can then be used as arguments for those commands. Specific examples include a floating point number retrieved from the distance between the tips of the pointer and thumb, as well as common actions like "next" and "previous" mapped to air swipes left or right, respectively. Other metrics, such as facial detection, can be integrated with ease.

How we built it

We built HandyTrack using the open-source framework MediaPipe from Google. MediaPipe's pre-trained models for hand and face detection made it the ideal choice for our project. The hand gestures are implemented by tracking points of interest, or "landmarks," of the hand. MediaPipe's included hand models allowed us to compare certain points of the hand relative to others, which allows us to implement effectively any hand gesture. By analyzing the spatial relationships between these points of interest, we could map gestures to specific commands, such as swipes or pinches. This flexibility also allows for the potential to adapt the system to recognize new gestures as needed.

Challenges we ran into

One of the challenges we faced was that the distance between points on the hand would change if the hand moved closer to or farther from the screen. While we could have somewhat addressed this by implementing a static threshold—requiring the hand to stay a fixed distance from the screen—we decided against it. A static threshold would reduce the system's versatility. The ideal solution would involve sensing depth with IR imaging, but that was beyond the scope of our software project.

Another challenge arose when we tried to implement emotion detection using the same technique as hand gestures. However, we found that using simple cascades for facial detection proved to be more reliable and robust. This was especially true since facial expressions are more dynamic and complex than hand gestures. This technique can also be used for text input based on sign language. See the 'testing' branch for more info.

And of course the most difficult challenge of a project using Python and unfamiliar tools and libraries was dodging version errors, circular imports, pip problems, and inconsistencies across the teams' machines (it works on my machine anyone?).

Accomplishments that we're proud of

We came up with several ideas, pivoted from them, and moved away from our original project after sinking hours into it. Despite this, we stayed committed and completed our first-ever hackathon!

What we learned

During the hackathon, we learned valuable lessons in teamwork, problem-solving, and adaptability. Collaborating under time constraints emphasized the importance of clear communication and effective task division.

As freshman, we didn't have much experience learning new technologies in a short amount of time, so we learned the value of learning to learn (say that 10 times fast!). Overall, the experience taught us to think creatively in the face of obstacles, making our first hackathon both a great learning opportunity and an enjoyable adventure.

What's next for HandyTrack

The future of gesture based input systems includes controlling IoT devices, touch-free Spotify Playback, as well as touch-free operation of common interactions such as e-signatures.

In healthcare, HandyTrack has the potential to allow medical professionals to operate equipment and access patient data without physical contact, reducing contamination risks.

Overall, HandyTrack aims to make technology more accessible and intuitive across various domains.