Won 6th Place Overall and Best Use of echoAR!
Try it Yourself!
Want to try out YogAR Yogi yourself? Feel free! It's avaliable for you to try at https://andrewdimmer.github.io/yogar-yogi. Note that due to time constraints, AR is only supported on iOS 11.or higher.
Inspiration
Modern life is stressful. Long working hours, poor diets, inactivity, and increasing social isolation in the digital age have all contributed to rising rates of anxiety and depression. The global COVID-19 pandemic, along with the lockdowns and social distancing used to combat it, have made all of these even worse. Health care workers on the front lines of this crisis have been hit especially hard.
Research has shown that yoga can relieve stress, and help reduce anxiety and depression.
Unfortunately, in-person yoga classes are one the activities that lockdowns have stopped. Learning any new skill can be frustrating and stressful, especially one like yoga that requires balance and physical coordination. Learning something like this all on your own can be downright overwhelming. That’s exactly the LAST thing that people need right now!
That’s why we created YogAR Yogi. YogAR Yogi is designed to help people learn and practice correct yoga forms in an easy, stress-free manner. After all, reducing stress is what yoga is all about, and it’s exactly what people need in these extra-stressful times.
What it does
The YogAR Yogi app uses TensorFlow to display a wireframe human image which positions the arms, legs, and torso into yoga poses. Players stand in front of their device’s camera and try to imitate each yoga pose. The web app takes the player’s image, and checks it against the app’s generated pose. The app will detect the accuracy of the yoga pose and display it as a percentage.
How we built it
There are a number of different components of the app itself. First, let’s start out with the front end website. Everything is built in React and Typescript, and styled with Material-UI. This allows us to easily rescale and rearrange the layouts of various pages to allow us to accommodate any device a user accesses the website from, whether it is from a computer or mobile device.
Next, we have the camera access. The first thing that we can do is access the webcam directly from the device the user accesses the site from. We actually use two different systems to access the webcam: first, we use the React-webcam npm module to assess if the webcam is on and we have the requisite permission to access it. This was selected because we had built-in React callbacks that allowed us to display messages to the user based on their camera status without leaving the React component that they were in. When we went to hook up TensorFlow’s PoseNet, however, we ran into some issues getting points out of the React-webcam module, so we built our own webcam system using the getUserMedia request in the JavaScript DOM to bypass parts of React and allow us to pass camera data directly into TensorFlow.
When it came to calibration, all of the data was collected via PoseNet, and the handlers are pretty standard React. We did do a lot of work to verify that there was only one active instance of anything running PoseNet at the same time. Likewise, when it came to generating and displaying the yoga poses, all of that was standard TypeScript libraries; we wrote our own algorithms to handle all of the generation and display. We wrote our own scoring algorithm, comparing the distance of the user’s points to that of the model, and giving a performance score based on accuracy.
For the 3D yoga models, we used EchoAR as our CMS and CDN, and used Apple’s ARKit to view the models, because we have Apple devices. The beauty of hosting and pulling our models from EchoAR is that the AR experience works cross platform on any device, and we just change which viewer we are using to display the file from EchoAR.
What's next for YogAR Yogi
- Improve the scalability of yoga poses for users of all shapes and sizes
- Include intermediate and advanced yoga workout routines
- Include more 3D models of yoga poses
Log in or sign up for Devpost to join the conversation.