ForeSights

• 

  with AI Coach

• 

Inspiration

Golf can be pretty intimidating, especially for beginners. It has a lot of technical details and requires a ton of practice to get better, and even experienced players often struggle to perfect their swings without access to high-level coaching or pricey equipment. ForeSights was born out of a desire to make this journey easier and more approachable for golfers at any skill level. By blending advanced technology with expert advice, we aimed to create a tool that helps players improve while giving them immediate, easy-to-follow feedback they can use right away in their practice. Our goal was simple: to make golf analytics both accessible and practical, so anyone can work on their game and see real progress.

What it does

ForeSights uses computer vision, a large language model, and websockets to provide a webapp that gives immediate feedback on how to improve your golf game. The information is then interpreted by a virtual coach which provides actionable feedback to work on in your next shot. All this information is right at your finger tips with the web app interface by showing the tracking system in realtime.

How we built it

By leveraging an Intel RealSense D415 Depth Camera, we are able to accurately track a golf club in 3D space to calculate the direction a ball would go. Through the combination of the LiDAR camera and the RGB camera, we used a YoloV5 model to detect the golf club head. Once identified, the model would track the distance over time to find the speed and the LiDAR camera measures the distance from the top of the golf club head to the bottom and determines the depth from these positions to calculate the angle.

For the virtual coach, we trained a LLM model on golf manuals and blogs using Cohere. This fine tuned model is then displayed to the user using the speed and angle metrics given by the RealSense camera.

Lastly, our web interface displays realtime tracking by using websockets, daily streak tracking to keep consistency, and snapshot statistical reviews.

Challenges we ran into

We tried to train two separate PyTorch models to use as a classifier but once we started training our models we quickly found out that it would be too time consuming and we would be better off using models that were already trained and ready to use. However, a minor issue with this was that most golf swing models were trained on people who are right handed and our testing club was for left handed people, the model was significantly better at detecting the club when it is on the right side. To fix this, we treated the left handed club like a right handed one and saw great improvements in the models accuracy. Another challenge going into this project was the unfamiliarity with some of the technologies and proved to have a steep learning curve for some members because of the complexity.

Additionally, we were overly ambitious and wanted to include a three.js 3D model to simulate the golf ball flight online. This proved to be more challenging than expected and was not able to be implemented due to the complexity. Lastly, the team came with varying technology, some on MacOS and some on Windows which created issues with using the RealSense camera which only had compatibility with specific systems. This also affected our attempts to train our own Model because of the lack in compute and no NVIDIA GPU.

Accomplishments that we're proud of

We are proud to have completed an ML heavy project with training and implementing an LLM, attempting to train CV models, and implementing real time tracking. We started off extremely ambitious with a high bar for submission and are happy to say that we have reached a point of satisfaction. Because we were aggressive with our choice of technology including the first-time use of LIDAR, personally trained cohere LLM, 100% deployed live-feed websocket and attempt at a 3js, the team learned exponentially more than we would have by staying within our comfort zone. It is rare in hackathons to have the opportunity to clean up and fully deploy a product and that was something the team is extremely proud of,

What we learned

At the start of this project, each team member had different levels of experience with YOLOv5 and Retrieval-Augmented Generation (RAG). During the hackathon, we learned how to fine-tune a large language model (LLM) using Cohere and integrated this LLM into our application using RAG, enabling it to fetch real-time data from our JSON files containing golf swing information. We also implemented the YOLOv5 model alongside an Intel RealSense D415 Depth Camera to classify our golf club and capture data inputs. Throughout the process, l team members gained a deeper understanding of backend development and successfully deployed the project using Vercel where websocket allowed us to have live updating video of the golf swing and had our coach “Professor Par” give us tips on how we could improve.

What's next for ForeSights

With more time and resources, a more stable and robust server connection and potentially using a microcontroller to run the Intel RealSense camera. Additionally, we could train our own models to develop a better golf club tracking system.

(https://www.foresights.ca/)

Built With

• cohere
• css
• fastapi
• html
• javascript
• ngrok
• opencv
• python
• vite
• yolo