SlüchAI

MainBoard

Inspiration

As developers and students who have worked mostly from home during the past two years of COVID, it was super easy for us to forget if we had an improper sitting posture during the majority of our day. As we enter a new era of hybrid and in-office work, we wanted to build a solution that will notify us when we are slouching or leaning to the sides or down so that our future selves (and back) will thank us!

What it does

SlüchAI uses our webcams or phone cameras (where ever it may be relative to your position), and uses PoseNet model from ml5.js. PoseNet a machine learning model that allows for Real-time Human Pose Estimation. Furthermore, we have built and trained our own Neural Network that detects if you are: Slouching to the Right, Slouching to the Left, Slouching to Down, or most importantly, Sitting upright!

How we built it

Model Design

Data Gathering Process

Using PoseNet, we were able to get the (x,y) coordinates of our body points (left eyes, right eyes, nose, wrist, legs, arms, etc), this data has 17 coordinates, which had to be flattened to 1 dimension, so every frame, it returned 34 data points. Using this, we "emulated" the possible positions: (slouch left, right, down, sitting upright), and recorded these data points for 10 seconds each. The p5.js library was used to capture the camera live feed, which refreshes at 60 frames per second, thus, every possible position had: 10 sec * 60 fps * 34 points per frame = 20400 data points This is more than sufficient! Then, this dataset was stored in Google Cloud Storage so that our servers can have easy access. This data will be the standard to train our neural network!

Training Process

Using ml5.js Neural Network, we use the data as training data, and let it train to classify. We ran several statistical tests in order to come to an epoch number that is the most optimized to reduce losses: Model Training

Quick Demo!

Instead of reading about it, watch how it works below!
Quick Demo of SlüchAI
Try it here!
Lightweight solution on Github Pages

Challenges we ran into

Importing the weights.bin file in google cloud storage into the web application. Finding the least required number of epochs to minimize losses.

Accomplishments that we're proud of

We are proud of working as a team to learn new frameworks and technologies. When figuring out how to approach this project, we knew there was going to be a lot of moving parts with technologies we have never used before. Upon taking this challenge, we are proud to have created a working solution in such a short amount of time.

What we learned

Training our neural network with large datasets (10-20MB), importing the model (model.json, metadata.json, weights. bin) into our classifier

What's next for SlüchAI

We want to add a feature for end-users to TRAIN the neural network with their own configuration and slouching types (slouched however you want!). Currently, our model doesn't take into the fact that users may have different camera setups and types, so a quick pipeline that runs the user through this process, and sets the config to their own model. We also hope to create our own customized analytics page so users can track their slouching habits and how they’ve progressed overtime!