With the use of technology being integrated into our everyday lives, there is an increasing number of people suffering from bad sitting posture all around the world. This leads to back pain, neck injuries, and more negative side effects.
Our team and probably the majority of hackers at this event face bad posture issues while sitting all day on our laptops, so we wanted to take a stand to fix this. We needed a solution that would be responsible with reminding us to fix our posture, that was convenient for us to use no matter where we were located, and was easily accessible to all users. Thus, we decided that creating a web application that used your laptop’s webcam would be the optimal solution - it is easy to take with you wherever you go and it is easily accessible for all users anywhere. This led to the creation of Posturizer.
What it does
Our product begins by authenticating user accounts - you can simply sign in with your google account. We have programmed this web app to check every 10 seconds to see if users are sitting in a good posture or not, and if not, it will give tips on how they can improve their posture. It takes a photo, identifies one of the five classes we are using - slouching forward, leaning back, leaning to the right or left and good posture and then will provide the appropriate advice _ by speaking back to you _. The application will also give the users metrics on their progress of improving their posture such as how they’ve been doing for the past couple months, weeks, and days, as well as their frequencies of postures for the current day and yesterday.
How we built it
For the backend, we used azure custom vision trained with a manually collected data set and labelled our primary data into five classes: front, back, right, left, good. These postures dictated the five posture positions we were using for our product. For the frontend, we used react.js and next.js with the firebase database to store previous user data/metrics in order to include their analytics and summaries on the dashboard and provide further insights. The frontend calls the API from Microsoft Azure AI to communicate with the backend which then spits out the class ‘front’, ‘back’, ‘left’, ‘right’, or ‘good’ with the respective probability. When the user is exhibiting bad posture, we have recorded audio that will tell them to align themselves according to their current position. Lastly, on the user’s dashboard there are visualizations to show their metrics and progress over time, created with ant design.
Challenges we ran into
For simplicity, we began by using a single class classification rather than a multi class classification when identifying the posture; this was challenging as these classes are not entirely mutually exclusive. One of the main challenges was collecting a wide variety of high quality and diverse training data - we needed different faces, backgrounds, clothing, lighting and collecting that dataset manually is a tedious process. Once we had this data collected, the next challenge we faced was improving the overall accuracy of the model from 40% to 91% through fine-tuning the model, transforming and augmenting the database and digging through the azure custom vision documentation. With transferring this data onto the frontend side, we had to determine how to keep resolutions consistent, and keep it user friendly with all of the extra features (dashboards, analytics, etc).
Accomplishments that we're proud of
With this project, considering the time constraint and lack of data, we were extremely proud of reaching an accuracy of 91% with detecting the five sitting postures. With the features of our product, we are proud of being able to preserve privacy by blurring out the faces of the people in the training model and as well removing photos of the users after converting them to data for analytics. Lastly, using Microsoft Azure AI and Firebase to create a unique product that would benefit our team as well as other hackers was an achievement.
What we learned
From this incredible experience building an impact product, we came away with many learning points. First and foremost, we learned of the power of great ideas. The potential impact and importance of an idea was key to rallying and staying motivated for the hack. Through being motivated by small accomplishments and ultimately by the excitement of reaching our end goal, we were able to work diligently and collaborate. With an idea we all resonated with and truly wanted to solve, we were all fueled with a common purpose. In addition, we realized the difficulties of gathering primary data to train an accurate model. It was our first time doing preliminary data collection and we had to go to great lengths to gather data that was diverse and unbiased to train an accurate model. Finally, we learned to create a user friendly experience for the user with a simple interface, rather than cluttering it up with many features that would be difficult to understand. All in all, this hackathon has been an amazing learning experience!
What's next for Posturizer
If we were to take our product to the next phase, we would begin with collecting more data of different people, different settings, pictures with more noise and preprocess the images so that we can focus on the postures in all settings and retrain our model. We could extend this to a multi class classification so it can detect other postures such as if you are leaning to the right but also forward; this could also increase our accuracy for postures that are bordering between two classes.
On the consumer end, we could add additional analysis features for users on the dashboard to track their progress over periods of time. The product could have better personalization as each person’s spinal structure is unique, and calibration to the individual user can help with personalized feedback and suggestions. The voice reminders could also be personalized with different accents, genders, and other preferences. This product could also be converted into a chrome extension or application rather than a web application for convenience.