Inspiration
It began as an argument on a familiar topic one too many Sunday afternoons: a team faced with 4th and 2 from the 45, and everyone on the couch has an opinion on what the coach should do. While the data has long demonstrated that NFL coaches punt and kick far too often, that knowledge is rarely useful in practice, outside of broad strokes. That is, given a situation like the one above, it is not immediately obvious what the right choice is. I set out to create a tool that could take a 4th down situation as input, and return a recommendation as output.
What it does
4th Down Decider takes as input a 4th down situation (distance, field position, point differential, time, timeouts for both teams), and returns a recommendation of what the coaching staff should do: go for it, kick the field goal, or punt. For each of these three potential actions, the tool trains a model on five seasons of NFL plays (2025-2021), estimates the expected points added (EPA) for taking that action in the situation presented, and recommends the action that would be estimated to produce the highest EPA. Wherever two actions' EPA estimates are within 0.5 points, the tool states that you should flip a coin between the two options with the highest EPAs.
How I built it
Using play data from nflverse (a github repo) , I filtered the 2021-2025 seasons to snaps on 4th down, labeled them according to the action taken (go for it, punt, or field goal), and constructed features from that data (distance, field position, point differential, time and timeouts remaining, red zone, two minute drill). For each of the three potential actions, I then trained a Random Forest regressor on these features, predicting the EPA for taking that action, optimized using GridSearchCV to tune the depth of the trees, their minimum leaf size, and the number of estimators. A Random Forest Regressor is a machine learning model that uses many decision trees to predict a number. It combines the results from all the trees to make a more accurate prediction. Using this model, I then estimate the EPA for each potential action, and return the one with the highest value, or a note if two are particularly similar. To better interpret the model, I also exported the decision trees used by each as images for inspection (both for my own curiosity, and for the user's benefit). The tool itself is a simple HTML/CSS page, with a python script handling the modeling and recommendations.
Challenges I ran into
The issue I fought with most while building this tool was identifying and working around the implicit biases in the data. Coaches punt nearly exclusively from their own deep territory, and go for it on 4th downs almost entirely on short distances. As such, the models for these actions have very little data to work with, outside of their primary use cases. When building the models, I added explicit checks to remove potential actions from consideration if the situation was too far outside the realm of what coaches typically do, and therefore what the model has data on.
Aside from issues with training data, I also discovered a bug while testing the tool: during training, I calculated the "two minute drill" feature based on how much time was left in the half, and in the game. When making predictions, the tool only used the time left in the game. This inconsistency meant that, during testing, the model would occasionally encounter a situation that it did not properly understand. When working on a project that makes use of consistency between training and test sets, it is important to remember that feature definitions must also be consistent.
Accomplishments that I'm proud of
I am particularly proud of being explicit and transparent about the inner workings of the tool. Every recommendation comes with an analysis of the EPA for each potential action, as well as an inspection of the decision trees the model uses to make its recommendations. Users are not asked to trust the tool blindly - they can see for themselves why a recommendation is made. In addition, I managed to put together a working recommendation engine from raw data in the time provided, including optimizing the model and caching the trained model and relevant data for later use.
What I learned
I learned a great number of lessons while working on this project, but the most important one was understanding that while modeling is frequently a large part of a data science project, it is rarely the most important one. The analysis of training data to identify potential issues, and the creation of guardrails to deal with them, is frequently the more difficult, and overlooked, part of the process.
In addition, I now have a far greater appreciation for why professional 4th down models (like the public Win Probability models created by various analysts) work the way they do, and what goes into making one. I am now aware of the amount of domain knowledge that goes into making such a tool, beyond raw modeling.
What's next for 4th Down Decider
As mentioned, I would like to update the tool to instead use a conversion probability model, along with an expected value calculation for each potential action. This would likely generalize far better to unique or unexpected situations. For the same reason, I would like to add team-specific tendencies to the mix, instead of relying on averages across the league. In addition, I would like to tie the tool to a sports API, so that it can populate a live game situation automatically, and make suggestions as the game is played. Finally, I would like to test the tool on a season of games that it did not train on, to see how its suggestions compare to what coaches actually do.
Log in or sign up for Devpost to join the conversation.