Beyond The Frame

GIF

Inspiration

A football move can be over before you notice the run, positioning, or space that made it possible. Replays let us watch again, but I wanted to make those moments interactive. Beyond The Frame began with a question: What if pausing a match let you explore the play from a player’s perspective? The goal is to help viewers move from watching the game to understanding the decisions within it, using an experience designed for the Fire TV remote.

What it does

Beyond The Frame turns football footage into an interactive tactical workspace. Viewers can:

Choose a player and switch between tracked players using directional controls.
Follow their movement through synchronized highlights, trails, and a moving follow camera.
Freeze a moment and choose among visible passing options.
Explore a hypothetical pass animated over frozen player positions, then replay the original footage.
Compare passing lanes under adjustable ball-speed and defender-reaction assumptions.
Ask tactical questions, including “What should I do next?” and “How is this player moving?”
Open a field view when calibration is available and save moments worth revisiting. The tactical assistant works offline using local rules and measurements from the selected frame.

How I built it

I started with my football-tracking notebook and developed an offline Python analysis pipeline using Ultralytics YOLO, PyTorch, OpenCV, and BoT-SORT. The pipeline produces timestamped player tracks, camera-cut boundaries, pitch calibration, and approximate field positions. It also creates tentative appearance groups from jersey colors. For the bundled demonstration, I processed all 358 frames of the supplied clip. The run produced 41 anonymous track IDs, with accepted field mapping on 292 frames. Tracking IDs can split after occlusions or camera cuts, so they do not represent verified player identities. I built the interactive client with HTML, CSS, and JavaScript, then packaged it inside a Java Android WebView host as a signed Fire OS APK. Heavy analysis happens beforehand, keeping playback and interaction independent of cloud credentials.

Challenges I faced

The first challenge was making player selection truly interactive. A rendered tracking video contains visible boxes, but it cannot independently switch the selected player. I generated raw coordinates so the application could draw its own synchronized overlays and follow different tracks. Another challenge was calibration. Broadcast cameras move, players disappear behind one another, and some frames do not contain enough reliable pitch information. The application preserves missing-data states and breaks trails across camera cuts. Passing analysis also required restraint. A clear-looking lane does not guarantee a successful pass. I used an explicit geometric model and interpretable labels rather than inventing completion percentages or certain outcomes.

Accomplishments

I brought the notebook into a working sports-focused experience with player switching, a smoothed follow camera, tactical questions, passing decisions, and replay comparison. The application includes offline analysis, a signed APK, and eleven core tests covering synchronization, tracking gaps, calibration, geometry, and data validation.

What I learned

The most useful analysis makes its reasoning visible. A highlighted player, a movement trail, and an explanation tied to the current frame can make a tactical idea easier to understand. I also learned that television interaction needs deliberate focus navigation, readable overlays, and predictable Back behavior. Every important action needs a clear path from the remote.

What’s next

My next milestone is completing Fire TV runtime validation. I also want to improve tracking continuity, appearance grouping, and calibration across more football clips. Longer term, I want to expand the tactical questions while keeping every explanation connected to visible evidence. Current validation: The application flows have been tested in the browser, and the Fire OS APK has been built and signature-verified. Fire TV device or cloud-device runtime testing remains outstanding.