Shrimply: Real-time Posture Monitoring

Inspiration

Poor posture is inevitable for students and professionals who spend hours at their desk every day, but most solutions are easy to forget about and lose track of. We wanted to create a simple and accessible way to keep people aware of their posture in real time and help them build healthier habits with constructive feedback and reminders.

What it does

Shrimply is a real-time posture monitoring application that uses computer vision to analyze the user's posture using their camera. It provides feedback along with a posture score (with the screen colour corresponding to good, satisfactory, or bad posture) and explains the reasoning behind that score.

How we built it

We built Shrimply with a Next.js frontend that handles the user interface and the live camera feed that is sent to the backend. The backend was developed in Python, and uses MediaPipe for real-time computer vision, detecting body landmarks to compute posture metrics and scores that are sent back to the frontend to display real-time feedback to the user.

Challenges we ran into

A challenge we faced with facilitating real-time communication between the frontend and the backend. We used WebSocket communication to send webcam frames from the frontend to the Python backend for processing, then sending posture metrics and results back to the frontend. Making sure this process was fast and synchronized correctly took major collaboration and debugging issues related to real-time communication and updating the UI accordingly. Another challenge we faced was quantifying posture, extracting vectors using the mediapipe computer vision library, normalizing the vectors, extracting angles between the relevant vectors, and developing boundary values for the different ratios to determine the user's posture status.

Accomplishments that we're proud of

We're proud of building a functional posture monitoring system within the 24-hour timeframe using only a webcam. Integrating MediaPipe with our Python backend and facilitating effective communication between the frontend and backend to deliver frames and display instant, effective feedback was a big accomplishment. We're proud of pushing ourselves to try something new with using computer vision data and turning it into a simple posture score for users.

What we learned

We learned a lot about working with computer vision and handling data from a live webcam feed. We also gained experience debugging frontend-backend interactions and managing communication efficiently.

What's next for Shrimply: Real-time Posture Monitoring

We'd love to keep working on Shrimply! Next, we'd want to: \ - Keep track if posture history and historical analytics \ - Add additional metrics to provide more detailed feedback (such as distance from the screen, time spent at your desk, etc) \ - Introduce login/signup functionality