Fix My Jumper

Inspiration

Both me and my partner love basketball. However, as we progressed through high school, we realized that we started to have less and less time to attend practices and our time with an actual coach began to decrease. But, our love for the game wouldn’t let that stop us. We decided to build an automated AI-powered shooting form coach that would give us tips and suggestions on how to improve our jumpshot, so we can practice the game we love without having to waste time commuting to practices.

What it does

The app records the user in real-time taking a jumpshot then it audibly gives the user suggestions on how to improve their shooting form through TTS. This way, the user doesn’t have to walk back to their device to read the feedback and can continue shooting. The app is accessible and extremely beneficial to all skill levels and can help all passionate basketball players improve their shooting.

It is an AI-powered basketball coach that uses your camera and the Gemini API to analyze your shooting form and give live audio feedback.

How we built it

We built it in VsCode and used Flask to build the backend and HTML to build the frontend. The app opens the camera using OpenCV and uses MediaPipe to track the positions of the shooter’s joints, such as the positions of the elbow and wrist. Then, when the system detects that there is a sudden change in wrist velocity and that the wrists goes above the shoulders, it indicates that a shot is being taken. The app then begins automatically recording calculations, such as the elbow angles and the elbow extension range, as well as the raw positional data of the joints and then passes the data to the Gemini API to analyze the shot mechanics. Once the API responds, the response is then converted to an MP3 file using Edge-TTS. The camera feed, API text response, and MP3 file are displayed using Flask, HTML, and JS.

Challenges we ran into

Initially, the project was going smoothly. However, after we encountered our first big challenge when trying to optimize the API usages. Originally, we were planning on sending all of the MediaPipe joints data to the Gemini API. But, we soon realized that this would take the API too long to respond and that we would probably quickly hit the rate limits. So, we decided to only send the data when the user was actually shooting. However, this came with its own challenges as we had to figure out how to actually write a shot detection algorithm that could accurately tell when a shot was happening, no matter the shooting form. We spent a considerable amount of time solving edge cases where the user does an unorthodox or irregular jumpshot (for example, how to detect a granny or grandpa shot or if the person uses both hands to shoot the ball).

Accomplishments that we're proud of

We are proud of creating an app that can accurately detect when a user shoots and deliver instant, real-time feedback that helps basketball players of any level improve their shooting.

What we learned

We learned how to integrate completely different technologies, like computer vision, generative AI, and TTS, into a single application. We also learned how to use raw joint coordinates and turn them into meaningful information (such as indicating a jumpshot has started).