FlopFinder

Inspiration

Soccer is the most popular sport in the world, enjoyed by billions of viewers globally. Unfortunately, there are often many controversial and outcome-altering calls made every game. Referees can simply make the wrong decision, leaving fans and players frustrated. Flop Finder uses computer vision technology to make objective, decisive, and accurate calls.

What it does

FlopFinder is an AI-driven platform that objectively determines whether a soccer play was a genuine foul, a warning call, a yellow card, or a red card. By uploading a video clip of the play, users will receive an instant analysis of the players’ movements, with the AI system evaluating key metrics - velocity, acceleration, torso angle, player contact, and reaction time - to classify the play. Beyond just giving a final verdict, FlopFinder also provides a detailed, referee-style explanation that helps users understand the reasoning behind the decision.

How we built it

We began by utilizing the YOLO(you only look once) computer vision model to extract pose skeletons from players in the uploaded video, essentially tracking each player’s movements frame by frame. From these skeletal representations, we then extracted critical parameters such as velocity, acceleration, torso angle, contact, and reaction time, assigning each a weight based on its importance in determining the call. This data was then fed into a Random Forest model that produced a weighted score corresponding to one of four outcomes: flop, warning, yellow, or red card. To mimic a referee’s breakdown, we integrated a large language model that generates human-like explanations (based on the extracted parameters). The entire model was developed in Python, with Streamlit powering the web application that connects users with our analytical engine.

Challenges we ran into

One of the main challenges we ran into was finding good data to train our model on. Since a dataset of classified soccer clips doesn't readily exist, we went and manually recorded and labeled clips we found online to build our dataset. Additionally, these clips would often contain lots of players not involved in the foul, so the corresponding skeletal models would have a lot of unnecessary data causing our model to get confused.

Accomplishments that we're proud of

We’re particularly proud of building a tool that not only distinguishes between legitimate fouls and flops but also explains its decisions in plain language that anyone can understand. Seeing our skeletal tracking reliably monitor player movements and our LLM generate insightful, referee-like feedback has been incredibly rewarding. Also, its really cool to have developed a model that we could actually use for ourselves, for example in our intramural soccer games.

What we learned

While building FlopFinder we learned a lot about computer vision - specifically how pose estimation works and how to make sifting through clips computationally efficient. We also learned a lot about Random Forest classifiers, and how an ensemble of tree models work together to produce a single result. Lastly, we learned how to host our app on Streamlit so that we can share it with anyone who wants to use it!!

What's next for FlopFinder

Our next goal is to implement a form of ball detection, since the position of the ball relative to the players is crucial to making calls, and something our current model would definitely benefit from. For example, if a player makes contact with the ball first, and then comes in contact with an opposing player, it should be classified as a clean tackle. Our current model may struggle to understand such scenarios. Additionally, we want to broaden our dataset even more. Ultimately we want to create a more streamlined process for gathering clips - perhaps using APIs to scrape match commentary data on flops, penalties, and other tackles, and then automatically clipping and labeling videos at the time the offenses were committed.