J.A.C.K. AI

Inspiration

Currently, physical therapy is not easily accessible to underrepresented segmentations such as rural patients, elderly patients, cost sensitive patients, disabled patients, and patients with a language barrier. Additionally, as a physical therapist, you will teach your patients the proper form for their exercises and make sure they are performing each movement correctly. But once they are back in their home environment, how do you know whether they are still doing their exercises the right way—or at all?

What it does

J.A.C.K. AI uses a trained computer vision model to analyze and monitor a patient's physical therapy exercises. The model is trained on correct and incorrect forms of exercises and identifies not just if a patient is doing an exercise incorrectly, but how they are doing an exercise incorrectly. We then provide users with personalized AI-generated feedback with GPT-4.

How we built it

Our project is an interconnected system of 3 different AI models. Our first model was a pose detection model that determines the x and y coordinates of the different movements of our bodies. We then process these coordinates through a series of algorithms and pass them into our manually built AI model that detects whether you are doing the exercise correctly or not. We will then extract information from this model and feed it into OpenAI's GPT API to make it give us comments/advice about how we performed the exercises.

Challenges we ran into

A major hurdle was gathering a suitable dataset to test and train our data on. As a result, we spent considerable time developing an easy and accessible frontend to convert our pose detection classification into csv files to be processed. In terms of training our models and displaying the recommendations to users, we also faced challenges in fine-tuning parameters for our AI as well as utilizing the OpenAI API.

Accomplishments that we're proud of

We are proud to have presented our idea to Y Combinator as part of the YC Pitch Challenge. Additionally, we are proud of the setbacks we have overcome and continuing to see an ambitious idea through in such a short amount of time.

What we learned

Throughout this entire hackathon, we have learned a lot about computer vision, web development, and most importantly working together as a team to come up with the best solutions and deliver the best results for our project. Although we all come from different backgrounds and different schools, we have been able to work very closely together and plan out the details of every single step of our project. All in all, we have not only been able to expand our technical skills but also learn so much about collaboration and communication.

What's next for J.A.C.K. AI

We will immediately reach out to physical therapists around the world to get our product on the market. On the technological side, we will begin classifying more exercise movements and training our AI models on a variety of different parameters that specify how movement exercises are incorrect.