Ghost Broadcast

Any game. Any Camera. Since 1997.

Comment

Inspiration

We love Basketball and it has given us many memories both happy and sad. We wanted to create a way to relive them in the most immersive way possible.

Thirty seasons of pro basketball live as text. Every game since 1996-97 has a play-by-play log: the shooter, the spot, the shot type, the rebounder, the clock. We wanted to watch them in 3D on the TV, from any seat in the building: Kobe's 81, Jordan's last shot in Utah, last night's game. Every game, as it was played.

Movement is the wall. Public tracking covers about half of one season: SportVU logs from 27 October 2015 to 23 January 2016, 636 games. Our open-source engine, pbp2track, carries that half-season across thirty seasons by motion matching. Each logged possession takes the closest tracked possession, pinned to the facts its log records. On 40 held-out games the rebuilt shooter sits a median 3.38 ft from his tracked release. A random tracked possession with the same result misses by a median 17.11 ft. The log's own calibration floor is 3.32 ft. On AWS, any of 37,925 games rebuilds on request in under 20 seconds.

Sports researchers call model-generated players ghosts. Ghost Broadcast is a broadcast of them, measured against their tracking. We built it in one week, 16 to 23 October: the data on Kaggle, the engine, the rebuild service on AWS, a Fire OS app for the TV and a phone to steer it.

What it does

Ghost Broadcast is a Fire OS app that plays basketball games in 3D on the TV. Where public tracking exists, it plays the measured game. Everywhere else, it rebuilds the game from its log, on request, in the cloud.

Tracked and rebuilt games. 2015-16 games play from optical tracking: ten players and the ball at 25 Hz, ball height included. Any game since 1996-97 is rebuilt from its event log alone. pbp2track borrows tracked movement and pins the logged facts: the shooter on his spot at the release, the ball through the rim on a make, the rebounder at the ball. Free-throw trips are staged at the line, so Kobe's 81st point happens on screen. Rebuilt games read MODELLED, and numbers drawn from modelled positions carry a ≈.

Any game, on request. The home screen's third row opens a picker over all 37,925 logged games since 1996-97. Press OK and AWS rebuilds it while the TV counts the possessions. The 2016 Finals Game 7 went from request to playable in 17.6 s. A January 1998 game in Chicago took 13.2 s, and a May 2026 playoff game 16.4 s. A rebuilt game is cached, and every TV opens it in about 1.5 s.

Court Sense. One question drives the glass: is this a good shot for this player, right here, right now? On Christmas Day 2015, Curry catches 24 ft out and rises from 25. The panel reads GREEN LIGHT: 48% for Curry from that spot, 40% for the league, 1.43 points a shot. Pick any player and the camera circles him. On the ball he sees his open teammates. On defence he sees the man who needs a body and whether doubling the star pays: DOUBLE CURRY? SEND TWO.

360 REPLAY and seven cameras. A made three or a dunk earns a replay. It slows to quarter speed into the release, freezes the ball off the shooter's fingertips and walks the camera 140 degrees behind him. Prime Video's Prime 360 builds this season's plays in 3D from arena cameras. Ghost Broadcast orbits 2015-16 games from their tracking coordinates. Rebuilt games get a straight replay marked MODELLED. Six cameras sit on the remote: broadcast, courtside, overhead, rim, the player's own eyes and a 360 follow. The phone adds a seventh, a free orbit.

The phone and the booth. Scan the TV's QR code and the phone becomes a second remote. Pick a player, or hold to talk: "360 on Curry", "stay with Steph and show me who's open". Rules resolve common phrases at once ("show me Steph's view" in 1 ms). Claude Sonnet 4.6 on Amazon Bedrock reads the rest from AWS Lambda in about 1.1 s. Claude also writes the booth from the event log, and a fact guard drops any call naming a player outside the play. Two voices call it in English, or isiZulu captions run on request. The D-pad drives the whole app, and six games ship inside the APK for offline play.

How we built it

The data, on Kaggle. Of the 636 SportVU archives, our play-by-play source covers 614. Five are broken, and 609 join on the event number, with 99.8% of logged events mapped to tracking frames. A Kaggle kernel cuts those 609 games, 49.1 million frames, into 108,602 possessions, normalised so the offence attacks one rim.

pbp2track: motion matching (Python, numpy, pandas). The engine builds each lineup from the substitutions, then rebuilds every logged possession in four steps. Retrieval queries the library on the outcome and picks the tracked possession with the lowest weighted cost over shot spot, shot family, possession length, the previous possession and recent reuse. Role assignment casts the logged parts first: the shooter, the assister, the ball handler on a turnover and the rebounder. A permutation search places the other players by least travel. A time warp of 0.75x to 1.35x fits the logged clock. Pinning walks the shooter onto the logged spot over 60 frames and eases the ball onto its tracked flight. Blended joins and 25 Hz output finish the game. Pinning the log's facts is how half a season reaches thirty.

The proof. We held out 40 games, rebuilt each from text alone, and compared 4,548 shots with their SportVU tracking. The baseline plays a tracked possession with the same outcome, unpinned.

  • Shooter error at the release, median: 3.38 ft rebuilt, 17.11 ft baseline. Mapped onto the court, the log's own shot coordinates carry a median 3.32 ft error: the calibration floor.
  • Shooter error, 90th percentile: 13.1 ft rebuilt, 32.6 ft baseline.
  • Player speed: 6.26 ft/s rebuilt (KS 0.028) against 6.28 tracked. The baseline runs 4.91 (KS 0.214).
  • Closest-defender gap error: 2.99 ft rebuilt, 3.65 ft baseline.
  • Team spacing: 503.7 sq ft rebuilt, 508.7 tracked.

Analytics. Shot zones follow the official definitions. An 11-feature logistic xFG model trains on 96,977 tracked shots on Kaggle, and the TV reproduces scikit-learn's probabilities to within 6.8e-6. Beta-binomial zone skill adjusts make chances per player. On top of the model sit expected points, open-teammate lanes, who guards whom and a double-team value.

Renderer and TV app. Vite builds TypeScript and three.js 0.186 on WebGL2 into one HTML file for the TV's WebView, in High, Medium and Low tiers. One faceless Quaternius CC0 body in ten likeness-free looks plays every player. The app follows Amazon's React Native for Fire TV path: Expo SDK 57, react-native-tvos 0.86 and react-native-webview 13.16, tested on the Android TV emulator (API 34, 1920x1080).

AWS. One CloudFormation stack holds the live side: an API Gateway WebSocket relay, a Node.js 22 Lambda and DynamoDB rooms with a 6-hour TTL. Bedrock serves Claude Sonnet 4.6 through the EU inference profile in eu-west-1. The phone page sits on S3 behind CloudFront. bash infra/deploy.sh deploys it all.

Ghost Broadcast on AWS

The rebuild service. The TV sends {type: "rebuild", game_id}. The relay checks a DynamoDB jobs table: a cached game answers at once, and a game in progress gains the room as a listener. A new game starts a Python 3.12 Lambda with pbp2track and the AWS SDK for pandas layer. It reads the season's Parquet file and holds a 2.2 MB index of 87,301 clips in memory. Each clip the motion matching picks arrives with one S3 range request into 1.75 GB of int16 tracks, so a game reads about 150 clips and only those. The rebuilt game goes to a games bucket behind CloudFront, with progress over the WebSocket. The tracked clips stay in a private bucket. Only rebuilt, MODELLED games are served.

Challenges we ran into

The data wall. A probe kernel joined SportVU to the play-by-play, the join held. The public copy stops on 23 January 2016. Every rebuilt game, from the 1998 Finals to last April in Phoenix, borrows its movement from possessions played between October 2015 and January 2016. We searched the Data Exchange catalog through AWS Marketplace for more data and it returned two Data Exchange listings, both market reports. A search of the Registry of Open Data on AWS for basketball returned zero files. So we decided to use machine learning fundamentals to get around the bottleneck and it kicked off our innovation.

Thin text. A log line carries a shooter, a spot, a shot type and a clock. The log and the tracking disagree: the scorer logged Curry's Christmas three at 29 ft, and the tracking puts the release at 25.0 ft. Tracked games read the tracked number. A rebuilt possession could end with the ball still in the air, so the app carries every shot to the rim.

Fire OS from a Windows PC. The demo video must show the app on a Fire TV device or the Fire TV/Vega simulator. Our test bed was one Windows 11 PC. The Vega Virtual Device runs on macOS and Ubuntu, and the Android emulator offers generic Android TV images only. So we took Amazon's React Native path for Fire OS on the Android TV emulator. On 19 October, with four days left, we weighed Vega once more: WSL, a reboot, an untested setup and a port of our WebView app. We stayed on the emulator.

The quick start. On day one the guide's npx create-expo-app <name> -e with-tv crashed on Windows. npx degit, a longer Gradle timeout, one removed package and an npm override got the TV app running. Inside the APK the WebView blocks module scripts and fetch() over file://, so the renderer ships as one HTML file and loads every asset through XMLHttpRequest.

The TV remote. Short presses arrived as a lone key up, and the focused WebView fired its own OK and media keys. The app now catches the lone key up, and the page acts only on keys the app forwards. The WebView takes focus on load and on resume. Amazon's pre-submission test for Vega apps caught a Back trap, and Back now walks out to the launcher.

The emulator's GPU. In 5 of 6 Medium launches on the emulator, a multisampled scene buffer swallowed every draw: a black court under a live HUD. The warm-up now frees those buffers. All four launches after the fix rendered.

Accomplishments that we're proud of

  • A game rebuilt from text puts the shooter a median 3.38 ft from the tracked release, within 0.06 ft of the 3.32 ft calibration floor. Player speed matches the tracked distribution: 6.26 ft/s against 6.28, KS 0.028. All of it comes from one public half-season.
  • Any of 37,925 games since 1996-97 rebuilds on AWS in under 20 seconds and serves every TV from the cache.
  • The glass matches the record. League 2015-16 FG% and 3P% read 45.2 and 35.3, against 45.2 and 35.4 published. Kobe's 2005-06 season matches to the shot: 978 of 2,173, and 180 of 518 from three. Tracked shot distance sits a median 1.42 ft from the official figure.
  • xFG reaches an AUC of 0.643 against 0.611 for distance alone, on 96,977 shots.
  • A 3D broadcast with a 360 REPLAY freeze, in one offline Fire OS APK built on a Windows PC.

What we learned

  • Building 3D motion from limited data.
  • The log sets the floor. A rebuild pinned to the log reaches the log's own accuracy, and broadcast footage would lower the floor itself.
  • Constraints carry the motion. The shot spot, the clock and the roles turn retrieval into reconstruction.
  • On a TV, focus is the product, and every key needs one owner.

What's next

The goal: every game, as it was played. Today a rebuilt game borrows its movement from one public half-season, and the shooter sits 0.06 ft above the floor the log allows. The next gain comes from data.

  • Licensed tracking. Cameras have tracked every game since 2013-14, and that tracking is licensed. More seasons would widen the motion library and test the rebuild outside the season it learned from.
  • Broadcast footage. Calibrated to the court lines, a frame from any camera would place every player on screen. pbp2track would pin each one the way it pins the shooter today and fill the players off camera from its library. A pose model such as WHAM (CVPR 2024) would give each player his own body shape and pose. The frame would also set the clock: logged events sit a median 23 frames, about 0.9 s, from their tracked moment. The 2015-16 tracking stays the yardstick.
  • A developer path on AWS Data Exchange. The app, the engine and the AWS side are built. Data is the one input a developer has to be granted. We want one sample season of today's tracking under a research or developer licence agreed with the rights holders, then the seasons since 2013-14, then archive footage on the same terms.
  • The product. Vega OS, built on a Mac or Ubuntu machine. Measured frame rates on Fire TV hardware. Last night's games by morning, on an EventBridge schedule. A learned refiner for the joins. More booth languages. Then any sport with an event log and one season of tracking.

Built With

Share this project:

Updates

Submission history