fitnessCV

Inspiration

Coming into HackISU, our team wanted to focus our project around computer vision and machine learning. In addition, we believe that the medical field is one of the places advances in machine learning could be the most beneficial. By combining these two ideas, we came up with the basic idea for fitnessCV.

What it does

At its core, fitnessCV is an interactive experience that allows the player to both better their physical and mental health through playing various mini-games.

How we built it

Using Python and its openCV and tensorflow libraries, we trained our own neural network to locate and track hands, as well as discerning whether the hand is open or closed. Once our model was sufficiently trained, we used a webcam to send a live video through our neural network. Then, our network records the location and state of the hand and writes that information to a text file. After this, our actual game, which is coded in Java, reads this information to correlate the movement of the hand to a cursor in our game. Then, by movement and the opening or closing of the hand, the player navigates the menus of our project and can play the various mini-games in fitnessCV.

Challenges we ran into

One of the major challenges we encountered was training our neural network to consistently track our hands. We found that this was mostly due to inaccurate data sets, since we initially trained with pictures we found off Google Images. Then we realized we needed to retrain with more relevant data. Thus, we took approximately 180 photos of ourselves with our hands in various positions as our new data set, which solved our major issues with our first attempt.

Accomplishments that we're proud of

The biggest accomplishment we made as a team was collectively learning how to create a functional neural network. On a smaller scale, we are also very proud of how we were able to integrate both Java and Python into our project and have them effectively communicate, so all team members would be able to work in a programming language they are knowledgeable in.

What we learned

As a team, we learned how machine learning works, with emphasis on neural networks. By completing this project, we were able to get a firm grasp on the concepts of machine learning and ultimately learn how to create our own, unique neural network to suit our needs. To be more specific, we learned quite a bit about convergence, which is when the loss for training goes down, but the data doesn't converge. This then relates to how changing our batch size can change the overall performance and accuracy of our model.

What's next for fitnessCV

We would like to further expand the catalog of available mini-games within fitnessCV so that any user would be able to get a comprehensive workout, whether it be mental or physical, and work towards living a healthier lifestyle.

Built With

• java
• opencv
• python
• tensorflow