ClimbAI

• 

  This is a preview of the overlay generator, which overlays multiple climbs to view mistakes.

Inspiration

As avid climbers, we always look to improve our climbing skills, and teach others how to climb. However, learning how to climb and improve by yourself is extremely hard, as we learned in our first few months of climbing. We had the ability to do the climbs, but we couldn't figure out how to set up our movements in order to reach the top. We couldn't constantly rely on our friends to show us how it's done, so we decided to use our coding skills to allow us to improve.

What it does

ClimbAI uses computer vision to analyze the locations of one's joints throughout a climb, as well as the angles of each joint, allowing one to rewatch the footage with this data to see how to fix their mistakes. Once the file is uploaded, we supply AI recommendations on how to improve your climb, and help you reach the top.

How we built it

Using online documentation, we first started with a basic react site, and worked on the formatting, but for logistical reasons, we decided to switch over to raw HTML, CSS and JS to complete our website. We used the MoveNet library to analyze the videos, and drawing functions in JavaScript to transform the output into a visual representation onto the person climbing. We originally allowed the user to download the edited video with the angles and the lines, but we removed this and replaced it with a video player. We used bootstrap to apply cosmetics to the website, and we used Arize AI for original testing, but decided against it due to its limitations on providing accurate recommendations on how to climb, and the difficulty to create a dataset to train a model on.

Challenges we ran into

One main challenge we ran into was analyzing the climber using computer vision. After hours of trying to code it ourselves with no avail, and testing other libraries online with no luck, we stumbled upon the MoveNet library created by Google, allowing us to accurately locate the joints during a climb, which is very difficult given the maneuvers that we

Accomplishments that we're proud of

We're very proud of the progress we made, and that we were even able to analyze a person's movements during their climb, and supply AI recommendations.

What we learned

As this is our first hackathon, we learned a lot about how to code, but even more about ourselves. In the beginning, Akash was more centered around web development, and Sid was more centered around AI, but we completely flipped the script for this hackathon, focusing on our weaknesses, allowing Sid to learn how to code a website from scratch, and Akash to learn how to implement computer vision.

What's next for ClimbAI

With future resources, we can utilize tools such as Machine Learning and Augmented Reality to calculate more precise statistics such as center of mass, 3D angles, jerk, a "beta" on how to hold the rocks for a climb, thoughtfully generated advice, and even a generated video of a human that completes the climb in the most optimal way, helping one visualize the exact path. We also plan to make the overlay video generator publicly available, which allows users to create their own overlayed videos, and see their improvement between climbs.

Built With

• alphapose
• arize
• bootstrap
• css
• git
• github
• html
• javascript
• movenet
• openpose
• php