REPLAY FORGE

Go inside the play. Change the decision. Compare what could happen.

Comment

Inspiration

A basketball game pauses for a timeout, and the argument on the couch begins. One person thinks the defender should have stayed with the shooter; another thinks he should have helped in the paint. Everyone has watched the same replay, but they are still trying to imagine what the players could see and how another decision might have changed the possession. Replay Forge grew from the idea that viewers should be able to step inside that moment and explore the argument together.

What it does

Replay Forge turns a recorded basketball possession into a stylized 3D replay that viewers can explore from different angles. The current demonstration begins with licensed broadcast footage and keeps that footage connected to the reconstructed players, court and timeline. A viewer can pause at the decisive moment, move the camera courtside or overhead, and examine the space around a defender.

From there, the viewer can give a supported instruction to change the defensive decision. Replay Forge creates a separate simulated continuation and lets the viewer compare it with the reconstructed original at the same point in time. The alternate gives people a way to explore their idea while preserving the original play as the reference. The browser experience also provides a return to the source footage.

How we built it

The reconstruction begins with a Python vision pipeline that detects players, estimates their poses and follows their movement through the video. Camera registration and court calibration place those observations into a shared court coordinate system. Ball detections are linked across frames, and the reconstruction records where it uses estimated depth or fills a short gap in the observations.

A Node.js companion loads the resulting possession and handles commands tied to the viewer's current moment and selected player. The WebGL viewer renders the court, animates the reconstructed tracks and displays the alternate branch for comparison. A Vega WebView wrapper brings this experience to the television development environment. Keeping the video, scene and commands on the same timeline makes it possible to move from watching a play to discussing a specific decision.

Challenges we ran into

The hardest part has been maintaining a coherent possession through camera movement, occlusion and incomplete observations. Player tracks can fragment when bodies overlap, and a single broadcast view leaves depth ambiguous. The pipeline therefore carries the supporting observations and estimates into the replay so that those decisions remain inspectable. Television playback required careful work on video compatibility and focus routing. The final Vega Virtual Device demonstration shows the licensed source pictures, reconstructed court, camera changes, a simulated defensive alternate, ghost-original comparison and a visible return to the source. The source pictures use silent JPEG derivatives at their original presentation times. Native movie decoding and source audio still need further work. The visible source return uses the clearly labelled simulated-assistant host API; remote Back delivery was not confirmed. The reconstructed movement is inferred, the alternate is simulated, and the demonstration uses local assistant controls.

Accomplishments

The project connects real basketball footage to an interactive world where a viewer can change viewpoint, direct an alternate decision and compare the resulting continuation. Bringing those pieces into one experience makes the technical work immediately useful: the viewer can investigate the exact moment that started the conversation and see their proposed change play out.

What we learned

The reconstruction experiments showed how much the experience depends on selecting observations that are useful for the particular sport. Ball-specific detection improved the available evidence, while court-line refinement improved the scale of the reconstructed scene. The integration work also showed that every camera move and instruction needs to retain its connection to the source moment, especially when the viewer switches between the original and an alternate.

What's next

The next stage will improve player identity, ball recovery and responsive movement in alternate possessions. The broader experience will let viewers save a moment while the game continues, revisit it through a persistent library and discuss it through a contextual voice companion. Those improvements will make Replay Forge feel like a natural part of watching a game with other people.

Built With

Share this project:

Updates