Everybody is at least generally aware of the negative impact a bad posture can have over a long period of time. However, most people do not know how to tackle this issue. Therefore we came up with the idea of an AI/image recognition based Posture Coach which can help people to improve their posture. Ultimately reducing health care costs, thus freeing up resources that can then be used to save the penguins.
What it does
It analyzes your posture through the webcam and records keyboard activity in order to establish a positive feedback loop to improve your posture and give you a detailed insight into your workplace habits. We also would like to motivate users to be rewarding their good posture performance through incentive programs which could include donations to environmental change organizations or health care benefits.
How we built it
We create a Vue.js Single Page Application which runs in the browser and feeds the webcam into the Tensorflow.js PoseNet model. The model was trained/configured separately and later plugged into the SPA. Then we take the output from the model and aggregate and visualize it in Charts using Apex Charts. Also we build a custom keylogger which does rolling window aggregation of the keyboard activity and streams it to the Vue.js application through WebSockets. After we had a functional proof of concept we gradually improved on the design and usability.
Challenges we ran into
The amount of input data was initially too large which caused massive performance issues in our app. Therefore we had to offload the data stream aggregation.
- Performance when running the pose estimation model in the browser
- Keyboard logging cross-platform compatibility
Accomplishments that we're proud of
- Build an actually functional and beautiful app in just one weekend
- It actually helped us to improve our posture during the hack
- Coming up with more meaningful metrics than we expected
- Solving a real-world problem that can truly help people
- Working together with people we didn't know beforehand
What we learned
- How accessible pose estimation is for real-word use-cases
- How to use Tensorflow in the browser (without GPU or strong hardware)
- How to split tasks within the team according to their skillsets
- How teaching others can improve the overall success of the team
What's next for Posture Pinguin
- Optimizing the performance using a native application (not in the browser), utilizing hardware (firmware).
- Building it into an actual product by integrating more sensors and metrics
- Creating a financially viable Business model and launching the product
- Connect to an environmental change program e.g. save the penguins