Inspiration
Our project was inspired by how important performing exercise with proper posture is. Any exercise with the wrong posture not only degrades its effectiveness, but can also lead to potential injuries. We also noticed that learning a proper posture by ourselves is quite challenging. With how good AI models are at synthesizing large amount of information together for valuable information, we thought it would be great if there were to be an AI agent that looks at your posture and gives real time feedback on areas of improvements. Hence, our project PosePerfect was created.
What it does
Our project let's the user perform a specific exercise pose, such as push-up and deadlift. Users are free to film themselves facing the front or sideways (although results may vary depending on the type of exercise). Then our program samples the some part of the footage and identifies joints of the user. After the join data has been required, we send it to Cohere's API in order to generate accurate feedback of the user's posture. The feedback can be read out loud with text-to-speech model, so that the user does not have to read off the screen when performing an exercise routine.
How we built it
We started researching about Cohere's API and then used it for generating feedback for the user. To implement the Text-to-Speech feature, we considered various options including Gemini and different Hugging Face models. As more advanced models required embeddings and more complexity to handle, we found a way to use a simpler model but made repeated calls in order to make the audio sound more natural with little sacrifice on complexity.
Challenges we ran into
We have faced multiple challenges in making the project. Interestingly, some of the aspects of the project that we expected to be challenging turned out to be okay. The real struggle came from the fact that most of us were not very experienced in using NextJS or React. Something so simple like creating a React component was confusing. In addition, we also had trouble with playing audio on the browser, as the discrepancy between the server and clients were causing difficulties in achieving this.
Accomplishments that we're proud of
- Successfully integrating Cohere's API to provide meaningful feedback based on real-time posture analysis.
- Implementing a functional text-to-speech feature, despite the challenges of handling AI-generated audio playback in a web environment.
- Overcoming our lack of familiarity with Next.js and React, learning and adapting quickly to build a working product.
- Creating a proof of concept that demonstrates how AI can assist users in improving their exercise form with minimal setup.
What we learned
- Planning early is very crucial for a hackathon, especially if it is less than 2 days as if you take more time planning, it will just take up the time you have left to develop your project.
- Using AI is greatly helpful to do things which would be state-of-the-art years ago and took many years of research or weeks/months of coding which can be achieved within a few hours today.
What's next for "Title"
- Better feedback, specifically showing what the correct position of the pose should be by drawing lines on the screen based on the user's current position in the video.
Built With
- cohere
- huggingface
- nextjs
- posenet
- react
- typescript
Log in or sign up for Devpost to join the conversation.