Inspiration
This project came to be as the result of a grueling 12-hour-long League of Legends session: we noticed that, due to the extended pronation of our wrists, we were potentially causing the development of carpal tunnel, amongst other issues. Due to the pandemic, this is not a problem simply faced by gamers, but by an increasingly greater number of people, as jobs continue to be operated remotely and people find themselves working their hands away in front of computers. Inspired by the potential that computer vision has to solve this problem, we set out to create Wrist Watch.
What it does
Wrist Watch will track your dominant hand using a webcam and make sure that your wrist is straight at all times. If your wrist is bent for an extended period of time, Wrist Watch will then push a notification to remind you to straighten up your wrist, preventing the development of carpal tunnel.
How it's built
We primarily made use of OpenCV to detect when a hand is in an unhealthy position -- that is, a position such that your wrist isn't straightened. Aside from using the Python programming language, we did not use anything else.
Training a Cascade:
Taking advantage of the OpenCV machine learning-based approach, we set out first to build our own Cascade, which is a model that is trained using a lot of positive and negative images. We also take advantage of data sets that are under the Creative Common License as the negative images.
Tracking:
With a working cascade from the previous step, we set out to create the main application itself. OpenCV + Python has the ability to take in a webcam feed, and match frame by frame to see if they contain the specified object: a human's hand.
Here is an example of what it looks like in action:
Isolating the Object:
In order to make sure that we are able to only perform calculation related to the wrist, through a combination of images alterations and modifications, we were able to do just that:
Notification System
Wrist Watch leverages the built-in system notifications on different operating systems in order to alert the user when their wrist is bent too far to the left or right, allowing it to run entirely as a headless background process. This allows us to conform to the user's system notification preferences, increasing the likelihood that users will continue to use the application and prevent it from becoming a distraction.
Challenges we ran into
Most of the main challenges we encountered came from OpenCV. Specifically:
- The lack of documentation forced us to rely on often outdated guides, along with a good mix of trial and error, to learn more about OpenCV and how it worked.
- We had many issues getting OpenCV to work on a Linux-based system, mainly due to compatibility issues. There were libraries that we needed to work with OpenCV that were not available with Ubuntu's default package manager, so we spent some time trying to figure out how to get some of the needed packages. These challenges alone took many hours to solve (approximately 4-5 hours). However, after resolving these challenges, everything (for the most part) worked out in our favor.
Accomplishments that we're proud of
We're proud that we were able to create a machine learning application like Wrist Watch in such a short time! Despite the increased time pressure of having to gather data for positive and negative datasets and wait for the model to train, along with the trial-and-error process of properly processing images, we were still able to complete most of our goals for Wrist Watch.
We're also proud of our ability to maintain consistent communication throughout the entire project. The virtual hackathon experience was new for all of us, but we were able to use tools like Git and Discord effectively to work together on this project.
What we learned
One member of our team has very little experience with Python, so this project gave him the opportunity to gain an idea of how powerful Python is. Three of the members of our team had no prior experience with OpenCV: As a result, Wrist Watch was both a challenging but also incredibly fulfilling project. With the increasing prevalence of computer vision in mainstream products like cars and phones, we believe this project has been an important educational experience for all of us!
What's next for Wrist Watch
Moving forward, we'd like to expand Wrist Watch's functionality, primarily by improving the efficiency of our detection algorithm and adding new features. As of right now, the various image processing techniques are relatively taxing on the system -- for a background process, we'd like to minimize the RAM usage! In addition, we hope to add a system by which users can train the algorithm to better recognize their unique hands (akin to Apple's Face ID), and increase our dataset size (potentially by allowing users to opt-in to share images with a database) to allow for better detection.
In addition, we would like to improve the deployment of our project: currently, we do not have automatic installation of dependencies. In the future, we hope to have a contained application on the Windows Store and Mac App Store to increase the ease by which users can install Wrist Watch.
We believe that, with the increasing number of companies choosing to have their employees remain at home even as the pandemic begins to subside, good ergonomics will become an even more important part of physical wellbeing. Wrist Watch is only the first step in improving desk ergonomics in our increasingly digital future.
Log in or sign up for Devpost to join the conversation.