PoseCheck

Inspiration

With the majority of our population spending most of their time sitting at desks, working, and staring at screens, our posture has gotten progressively worse. Adding the pandemic and its work-from-home lifestyle, many people's posture and subsequently, health, is being damaged. A study by the NIH quantifies that over 40% of the work-from-home population experienced lower back pain during quarantine and over 20% experienced neck pain. Moreover, 80% of Americans experience back problems at some point in their lives. In order to combat this growing issue, we decided to come up with a web app that can monitor and notify the user of their posture, enabling a healthier lifestyle.

What it does

PoseCheck is a web app that runs in the background while you are working. After allowing the app to use your camera and waiting a few seconds for calibration, PoseCheck will be continuously monitoring your posture throughout the session. We compile the results and display them in our graphs every few seconds as well as provide real-time feedback, such as through the visual representation of our model’s output. Our log-in system also allows for the user to track the posture over the course of many sessions to evaluate their progress.

We measure six different features to quantify the user’s posture:

• Proximity to the Screen: Measures how close your face is to the screen. Sitting in a way such that your laptop screen is very close to your face can cause long-term strain and damage your eye health.
• Slump: Measures distance from shoulders to ears. If you are slumping downwards, your shoulders will be closer to your ears. Slouching can cause your spinal discs to compress and increases the likelihood of back pain.
• Forward Tilt: Measures distance between your ears and your eyes. If the ears move to become higher than the pupils, the user is leaning their head further forwards. This can place undue strain on the back of your neck.
• Head Tilt: Measures difference in heights of your eyes. Leaning towards one side can place extra pressure on your neck, and can cause back and shoulder pain over long periods of time.
• Shoulder Tilt: Measures difference in heights of your shoulders. Leaning towards one side, or placing extra pressure on one shoulder can cause back and shoulder pain over long periods of time.
• Shoulder Width: Measures the distance between your left and right shoulders, scaled for proximity. Narrowing your shoulders can be a sign that you are slouching forwards, or pushing your shoulder blades back. Either of these can cause staring and back pain over long periods of time.

How we built it

PoseCheck uses a Flask backend and a MongoDB database to store users and past sessions. On the users’ session page, the webcam takes a picture at a set interval of time and passes it onto a convolutional neural network to determine the key points of the users’ pose, which includes their shoulders, nose, ears, neck, and eyes. Using OpenPose and numerous other CV and ML libraries, we calculated the key points and bounding boxes of the users’ faces. Once they were determined, we analyzed the six features described above and determined a score for each. The scores for each image were sent to the frontend where we used Charts.js to create graphs displaying the features over time. In addition, we used Tensorflow.js to provide the user with a real-time visualization of the key points that our model focused on.

Challenges we ran into

Due to the large size of our pose detection model, we were initially not able to push it to GitHub to deploy on a Heroku server and were also facing severe performance issues due to the limited RAM available in the free tier of Heroku. After spending some time searching for solutions, we made the decision to switch to AWS even though the process for deploying our app on an EC2 instance was much more complicated. We successfully managed to make the switch between Heroku and AWS and bypassed our earlier problems.

We didn’t have much experience with Charts.js prior to this, so it took some time to figure out how to create the graphs and display them correctly on the session page. We also had some trouble using TensorFlow.js but we figured out how to properly load and use the models for our real-time visualization.

Accomplishments that we're proud of

• Successfully integrating the web app with our pose detection model and analysis and displaying the graphs in a concise, easy-to-view manner.
• Adding a tensorflow.js model to detect key points of poses to a live webcam so that the user could view how the graphs and feature values were updating.

What we learned

• We learned how to work more extensively with JavaScript and certain libraries in the language, especially Chart.js and TensorFlow.js.
  
• We also learned how to deploy a web app on an EC2 instance in AWS.
• We learned how to use base 64 encodings to send image data from a web app to a flask backend using POST requests, enabling us to use Python for processing and decrease computation on the client-side.

What's next for PoseCheck

We plan on integrating a push-notification alert system to allow for the user to be notified when their posture is worsening, in case they are not paying attention to the app. We also plan on creating a simple and advanced mode to allow the user to be provided just a general overview in the simple mode, and then when switched to advanced they can get all the details. Finally, we are working on creating an overall metric to combine each of our six features together to make it more understandable, especially for the simple mode.

Built With

• charts.js
• cnn
• flask
• mongodb
• python
• tensorflow