Seeing that physical fitness was a rather under-served aspect of quarantine, we were very excited to pursue an opportunity that would both increase interpersonal engagement while getting people up and moving - ultimately improving both mental and physical fitness.

What it does

Our application connects individuals through two major approaches: community and competition. By allowing users to search through groups nearby that share similar interests to themselves, users are better connected to those who they can relate to. Within these groups we offer the ability to play various minigames, whether it is lifting a metaphorical elephant up a mountain (aka do a large number of squats and pushups to successfully accomplish the task) or racing against a metaphorical zebra (aka run and/or bike for a large distance), the objective is not only to complete the task but to do so with as much group involvement as possible. The more evenly spread involvement is, the higher score rewards are. Additionally, as teams work together to complete challenges, their continuity streaks (how many days in a row is a significant portion of the group active) will contribute to additional score increases.

Aside from this collaborative aspect, however, we also introduce a competitive side to the platform with challenges. Challenges can be hosted by anybody, and they can be set to either public or private. The concept of these challenges is to compete against everyone else in the group for the specified exercises. For example, if the exercises are to run, bike, and do squats, the user who does the most overall (has the highest score) wins the tournament (gets additional score bonus + profile awards).

To encourage fair and safe competition, PhysiPal also includes a GCP-hosted ML model that receives live camera footage from the mobile application and analyzes form to verify push-ups while also providing feedback if form is incorrect. It also tracks distances using GPS for accurate measurements during runs and biking. Through these features, we create a community of users who are interested in both fitness as well as social interconnectivity.

How we built it and Challenges we ran into

This application was primarily built with Flutter in the front-end. The back-end was a mix of GCP products (including Firebase), python, websockets, pytorch, and open-cv integrations. The front-end was quite challenging, and contains quite a few large holes when compared to the original UI/UX plan. This was primarily due to Flutter being a relatively new framework for many of us. The backend consist of 2 endpoints. The websocket control plane endpoint between server and clients, and the RTMP data plane endpoint. The RTMP end point is managed by an RTMP server, and the code here consists of the websocket server. All communications between client and server are handled in JSON messages. The protocol include 2 fields, type of the message, and an optional data. After connection, the client is expected to send key type (without optional data) to request the stream key for RTMP end point, and the server will respond with key type and the content of key in data field. Then, the cient is expected to establish connection to RTMP end point and serving the video stream (possibly from the camera feed of a mobile device). After RTMP connection is established, the client is expected to send type connected (without optional data) to inform the server about RTMP connection. Upon receiving the packet, the server will begin inference subroutine (running on a separate thead), and continuously send the current result to client with type push, and the corresponding statistic of the workout based on the machine-learning based evaluation of the video stream. The RTMP stream and websocket connection can be terminated at any time by client. After the termination of websocket connection, the ML evaluation will be terminated immediately. With RTMP clients from mobile devices, this system allows real-time machine learning based analysis of workout, and the client can display those statistics as needed by application. The design of using a server has many limitations, and the decision of such design is entirely due to the convenience of implementation of such ML inference system on server environment. With more mature support of machine learning infrastructures on mobile device (such as OpenCV and pytorch libraries), this entire system can be integrated to the mobile device itself without server dependency. This will enable lower latency, less bandwidth/battery consumption, and offline usage.The back-end had lots of trouble in terms of making the analysis of exercise done in real-time, making efficient data structures for larger queries, and learning how to use GCP to host the ML model.

What's next for PhysiPal

Next up, we aim to complete the application. There are quite a few holes (both in functionality as well as design) from our original thought processes. Additionally, the idea itself has room for growth. By incorporating more exercises and challenges, PhysiPal aims to appeal towards a larger audience. The application can even be expanded to involve sponsors of various sorts, and transform the platform into one that has individuals pursuing fitness, supported by sponsors who in turn donate funds to charities.

Share this project: