• 

  PostureAI Screenshot

PostureAI - AI powered posture improvement

Inspiration

As more and more people work from home, individuals are often spending 9+ hours a day in front of their laptop. This is leading to an epidemic of back, wrist, and neck injuries due to repeated stress from improper ergonomics and posture.

We wanted to build an effective, convenient, and intelligent solution to help individuals actively improve their posture while working from home.

What it does

We implemented a Computer Vision system that calculates the angle (pitch, yaw, and roll) and position of the user’s head relative to a pre-calibrated position in order to determine if they need to correct their posture.

After the user calibrates, a timer is started -- while the timer is running the user is free to minimize the window and continue with their work. After the timer runs out, the posture verification process begins. If the user's posture is bad, they are notified with an audio notification, which plays until they fix their posture. If the posture is good, the timer begins again and the process is restarted! To protect the user's privacy, no facial image data is shared anywhere outside the app -- you can even use the system without an internet connection!

The system focuses on posture rather than position, so the user is still able to move left and right comfortably as long as their posture (i.e. angle/tilt of head) stays consistent

The app works best when the user is viewing the camera straight-on, such as when the user is using a laptop, however it also has very strong results when the camera is at an angle to the user.

How we built it

React

The app is a static website that is built using React and Typescript. We felt that it was very important to protect the user’s privacy, and wanted to ensure that no facial recognition data is shared over the Internet. Therefore, the app is able to work without an Internet connection once loaded, and can even be downloaded to your desktop.

Firebase

We used Firebase to deploy the system, utilizing Github Actions for CI/CD. Because of this, every pull request will be deployed with an automatically generated link for testing and sharing purposes. Furthermore, every push to the master branch will lead to an automatic redeployment of the app to production.

face-api.js

We used an NPM library called face-api.js to extract facial landmarks using computer vision. This library is a shallow wrapper around Tensor Flow (more specifically TensorFlow.js), which uses neural network machine learning models.

Challenges we ran into

A challenge we ran into was performing the calculations necessary to determine the user’s posture. This involved significant use of trigonometry and linear algebra in order to calculate the user’s position, tilt, yaw, and roll based only on the visual position of their eyes, nose, mouth and jaw. We planned out equations and strategies on paper before converting them to algorithms and code.

While developing the frontend, one challenge we ran into was managing all of the concurrent moving parts such as the webcam view, user calibration, and the mapping of the face mesh onto the live video before calibration.

Accomplishments that we're proud of

We’re proud of developing different techniques that leverage trigonometry and linear algebra concepts in order to implement posture checks. Each of the posture checks required us to consider a subset of 68 detected points of the user’s face and to calculate angles and distances that were then compared to the user’s calibration. In the end, we are all proud of how all the pieces came together to form the final product.

As a group, we were very productive and efficient and as all our group members worked well together. All teammates worked hard and motivated each other, resulting in an amazing learning experience.

What we learned

One major skill that we learnt through this experience was the use and implementation of Computer Vision models purely on the frontend. We learned how to leverage face-api.js and Tensor Flow in order to extract key landmarks from the user’s webcam view.

What's next for PostureAI

In the future we'll be adding analytics to the user experience. Users will be able to see statistics such as how long they kept proper posture or how many times they were reminded to fix their posture. Historical data will also be provided so users can track their progress from day to day and see their long term trends. Another future addition will be different types of challenges to encourage and motivate users to keep up their proper posture.

Built With

• ai
• chakra-ui
• computer-vision
• face-api
• firebase
• react
• tensorflow
• typescript