Panoptic Pigskin

• 

  Overhead

Inspiration

Growing up, I wanted to be an NFL quarterback. I wanted to see what they saw: the pass rush closing in, the safety rotating, a throwing window that's open for half a second. But every game I watched was from the outside looking in. Broadcast angles show you the play; they never put you inside it. I wanted to experience football in first person.

What it does

Panoptic Pigskin turns NFL All-22 footage into a 3D replay you can step inside. From just two camera angles, it reconstructs the field of all 22 players, then renders the scene with Gaussian splatting. You can watch the play from any spot: through the quarterback's eyes as he drops back, from a safety settling into his zone, or from a free camera hovering over the line.

It also generates a report for every play that grades each player's execution. Each grade is backed by evidence from the reconstruction, such as a late read, a blown gap assignment, a route rounded off by two yards, and shows what the player could have done instead.

How we built it

Everything is written in Python, using the following libraries:

1. Detection: YOLOv8 locates players in frames of both feeds
2. Identification and Tracking: EasyOCR reads player jersey and tracks player ID
3. Camera Calibration: Triangulate camera position, heading, and zoom through common field landmarks: hashes, 10-yard markers
4. Body reconstruction: SMPL-X frames over dual-camera resolved COCO joint positions
5. Rendering: Gsplat accelerates rasterization of gaussians

Challenges we ran into

The core problem was two cameras versus infinite viewpoints. Almost every angle you'd want to watch from is one nobody filmed, and players constantly block each other from both cameras. I filled that gap by relying on what's known about football, not just what's visible: the field's exact dimensions, SMPL-X body priors that keep poses anatomically plausible, and physical limits on how far a body can move between frames. Combining those constraints let me reconstruct the parts of each player the cameras never saw.

Accomplishments that we're proud of

Accurately solving for 22 unknown bodies given only two camera angles and a handful of quality assumptions.

What we learned

When it comes to computer vision, progress comes from a lot of experimentation. Trial and error gives you more options to succeed with data to back your solution.

What's next for Panoptic Pigskin

I want to scale from single plays to every play in the NFL. Then, I want to go back in time. Older film is grainier and less consistent, but I want to reconstruct 1980's football directly from the eyes of Joe Montana and Dan Marino.

Beyond fans, Panoptic Pigskin is an incredible tool for coaches and athletes to watch film. New insights can be drawn from different perspectives, and letting players relive their own plays is incredibly valuable.

Built With

• coco
• easyocr
• opencv
• python
• scipy
• smpl-x
• yolov8