PoseMe

Landing Page showing salient features of the platform
Login page for existing users
Registration page for new users
Route to create a new session/room
Dashboard showing different people participating, and challenging with unique poses
Using tensorflow.js we are finding the pose similarity between two instances
Entity Relation Diagram (We used cockroachdb)

Inspiration

Covid19 made us realize the importance of having a social circle and the impact isolation can have on our mental health. Although restrictions have greatly relaxed across the world, people still undergo periodic quarantine when they become infected with Covid19 or have been exposed by someone. Furthermore, in the post-pandemic world, workplaces are becoming increasingly remote and employees may not have the opportunity to build relationships with co-workers outside work. Our team built a simple physic activity game which allows people to connect together remotely in a healthy way.

What it does

PoseMe adds a competitional component to exercising. Users can create and join rooms of up to 4 people. The customizability of the room creation feature includes room name, duration, level of difficulty, and an option to keep/remove music. In a room, users will have turn and make poses and other users will have to mimic the pose to the best of their abilities. Our machine learning algorithm will judge the accuracy of the pose performed and award points at an interval of (undefined). During the game, players can see the scores change and at the end of the game, the final scores will be displayed and uploaded to a leaderboard.

How we built it

The frontend includes the home, login, registration, session and game room pages, all of which were built with React. Twilio was used to facilitate real-time video and audio streaming between the frontend and NodeJS backend. Snapshots are were captured from videos sent by Twilio and utilized by ML models in TensorflowJS to identify poses players make, which are then compared with video streams from other players. The model will output scores which are used to keep track of who is winning the game. CockroachDB was used to store user session data and images.

Challenges we ran into

Our biggest hurdle was setting up the video streaming service. We initially tried to use WebRTC, but encountered difficulty setting up the peer to peer connection. We decided to look up examples from previous teams in the past and utilized Twilio instead. Our teammates were also scattered around the globe so synchronizing the timeline was a challenge. In addition, one of our teammates caught Covid and was unable to attend the Hackathon.
Tailwind’s approach of being component-agnostic made for a “fun” time trying to get the design to look good. It was a low-friction experience designing the home, join, and create room pages - however, creating the room page became quite messy. It took a lot of determination and toying around with the flexbox and grids to solve this,
Some other issues that arose are included (but not limited to): Some of our internet connections dropping, Heroku rate-limiting us, the CSS grid being dysfunctional, and the music feature being buggy at times

Accomplishments that we're proud of

It was fun to build a project from scratch with people we met only few days ago. It was a great learning and collaborative experience and each of us learned new things.
We are very glad and happy to have a web program that remarkably executes our vision for this app. Many aspects were required for this program to be what it is, from devising a rough plan of what we want to accomplish to training and creating our machine learning model judge. Being able to create a frontend that’s built on the charm and appeal of minimalism and have it seamlessly tied in with a robust backend that is able to link those processes together. Being able to integrate the Twillio API on our application is another awesome accomplishment. Our ML judge was quite accurate thanks to the massive amounts of data and training we poured onto our model.

What we learned

We learned how to use Twilio and the requirements to build a video streaming connection.
Communication is key to success.
Downscaling can be the right approach when time is of the essence.
You have to be calculated what you should be persistent with. A minor bug doesn't mean as much as a codebreaking bug.
Trying to learn Tailwind during a hackathon is a recipe for pain and distress.
using twilio API for video, WebRTC
Learned to use CockroachDB.

What's next for PoseMe

We would like to make it a fun platform where anyone can come and play, exercise, do yoga etc. Our next step is to perform refactoring, followed by deployment to a scalable infrastructure. We will continue our research in this delightful area and shall make it one of its kind free and open-source platform for leisure time games.