Go Deep!

• Title: Summarizes the main idea of your project.

Go Deep!

• Who: Names and logins of all your group members.

Caleb Moran (cmoran5), Andy Burris (aburris3), Vanessa Alexander, (valexan3)

• Introduction: What problem are you trying to solve and why?
  • If you are implementing an existing paper, describe the paper’s objectives and why you chose this paper.
  • If you are doing something new, detail how you arrived at this topic and what motivated you.
  • What kind of problem is this? Classification? Regression? Structured prediction? Reinforcement Learning? Unsupervised Learning? etc.

We are implementing an existing paper called DeepQB by Brian Burke (https://assets-global.website-files.com/5f1af76ed86d6771ad48324b/5f6d394ebce99d0d6cdb767c_DeepQB.pdf). This paper aims to bring deep learning techniques to football analytics, primarily looking at how quarterbacks (the most important position in the sport) make split-second decisions on the field. We chose this paper because we are all student-athletes and sports fans. It’s fascinating to try to understand the mental processes of elite athletes. Potentially the insights from this implementation can help us in our own sports! The problem calls for regression in estimating expected yards and classification for outcome or receiver target probability.

• Related Work: Are you aware of any, or is there any prior work that you drew on to do your project?
  • Please read and briefly summarize (no more than one paragraph) at least one paper/article/blog relevant to your topic beyond the paper you are re-implementing/novel idea you are researching.
  • In this section, also include URLs to any public implementations you find of the paper you’re trying to implement. Please keep this as a “living list”–if you stumble across a new implementation later down the line, add it to this list.

There is an interesting paper called POINTWISE (http://www.lukebornn.com/papers/cervone_ssac_2014.pdf) that uses player-tracking data for the NBA to analyze expected possession value. The paper’s authors designed a probabilistic model to value player actions. The paper discusses how trying to fit their model to the immense amount of data was quite challenging—something that DL can often succeed at. Similarly to the football paper we are implementing, this research can determine player offensive value.

• Data: What data are you using (if any)?
  • If you’re using a standard dataset (e.g. MNIST), you can just mention that briefly. Otherwise, say something more about where your data come from (especially if there’s anything interesting about how you will gather it).
  • How big is it? Will you need to do significant preprocessing?

Our data comes from Kaggle, where the NFL has posted a publicly available subset of player tracking. https://www.kaggle.com/competitions/nfl-big-data-bowl-2023/data It is of a reasonable size. We will need to do some pre-processing to handle the CSV files. There are a lot of columns we can drop to reduce the file size. We also will want to filter out all non-passing plays.

• Methodology: What is the architecture of your model?
  • How are you training the model?
  • If you are implementing an existing paper, detail what you think will be the hardest part about implementing the model here.
  • If you are doing something new, justify your design. Also note some backup ideas you may have to experiment with if you run into issues.

We will use Tensorflow and Keras. The architecture from the paper is:

• An input layer of 230 nodes
• A dense hidden layer of 256 neurons
• A second dense hidden layer of 128 neurons
• A dense output layer, with activation dependent on the type of target variable the model is being asked to estimate

We also will use batch normalization, relu, and dropout. I think it will be challenging to avoid overfitting, so we will need to be careful with that. We also will need to effectively determine our hyper parameters. The original paper uses a grid search to find them.

• Metrics: What constitutes “success?”
  • What experiments do you plan to run?
  • For most of our assignments, we have looked at the accuracy of the model. Does the notion of “accuracy” apply for your project, or is some other metric more appropriate?
  • If you are implementing an existing project, detail what the authors of that paper were hoping to find and how they quantified the results of their model.
  • If you are doing something new, explain how you will assess your model’s performance. What are your base, target, and stretch goals?

We want to see the model predict who the actually targeted receiver was with significantly higher than a 20% rate (the naive uniform distribution). The author of the original paper achieved accuracy of 59.8% on this metric. ”Further, when quarterbacks target the receiver as predicted by DeepQB, the completion rate is 74%. This compares to a completion rate of 55% when quarterbacks target any other receiver. These results indicate that the model can successfully make inferences using the tracking data, and is capturing some degree of understanding of each pass play.” We plan to run these two experiments with our implementation.

• Ethics: Choose 2 of the following bullet points to discuss; not all questions will be relevant to all projects so try to pick questions where there’s interesting engagement with your project. (Remember that there’s not necessarily an ethical/unethical binary; rather, we want to encourage you to think critically about your problem setup.)
  • What broader societal issues are relevant to your chosen problem space?
  • Why is Deep Learning a good approach to this problem?
  • What is your dataset? Are there any concerns about how it was collected, or labeled? Is it representative? What kind of underlying historical or societal biases might it contain?
  • Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?
  • How are you planning to quantify or measure error or success? What implications does your quantification have?
  • Add your own: if there is an issue about your algorithm you would like to discuss or explain further, feel free to do so.

While there are no surface level biases in the player tracking data, who these quarterbacks are demonstrates serious racial bias. Black quarterback prospects are consistently undervalued by NFL franchises and the media, leading to fewer opportunities. There is a myth that Black QBs are only “running backs” or cannot play the game at a high mental level.

The major stakeholders in this project are NFL teams, who are interested in assessing their players and those on other teams. But another serious stakeholder is the fans, particularly those who play fantasy football or gamble. Providing them with misleading information through algorithmic errors could cost them major amounts of money.

• Division of labor: Briefly outline who will be responsible for which part(s) of the project.

Caleb is responsible for all forms & assignments, and managing the project at a higher level. Andy is handling pre-processing. Vanessa is in charge of the loss functions. All of the group members will contribute to implement the model’s training and testing functions.

• Methods:

Our data differs from the original paper (because the original set was unavailable to the public), but we use NFL Big Data Bowl player tracking from 2021. We pre-process it to isolate the instant the pass is thrown, and to include only information relevant to choosing a pass target. Then the data goes through three Dense layers. Between each is ReLU, dropout, and batch normalization. The final activation type determines which variant of the model we run (receiver target prob., expected yards, or pass outcome prob.).

• Results: Our model was trained for 10 epochs with a batch size of 128 and the Adam optimizer at a learning rate of 0.001. After training, our model which uses softmax activation to represent the receiver target variant, achieved a validation accuracy of over 40% and a validation loss of approximately 1. The original paper achieved a validation accuracy of just under 60%. We believe this difference is the result of that paper having approximately five times as much data as we did.

• Challenges: Our main challenge was with pre-processing the data. We had to use a different set of player tracking information than the paper, and this meant we needed to invest significant effort into maximizing what the model could learn from it by pre-processing. Matching the player IDs and parsing the natural language text were rather challenging tasks.

• Reflection: Our project was a valuable learning experience. Because of various pre-processing issues, the model was not as successful as we had hoped. As such, we only achieved our base goal of implementing the target receiver probability variant. Given more time, we would implement the other two variants. Over time, our approach became more focused on how we would better pre-process to decrease loss and increase accuracy. Originally we were hoping to increase those by tuning hyperparameters, but this was not fruitful. Had we done this all over again, we perhaps would not have chosen this project because of the limitations of the dataset.

Built With

• python
• tensorflow