-
-
Training environment for the neural network
-
-
Inspiration
I work at Tennis Channel, and I have a confession: I do not fully understand tennis.
I can watch a rally, but I often cannot explain why a player chose a particular shot, how an opening was created, or what I should notice next. Traditional broadcasts are excellent at serving existing fans, but they speak a language beginners are already expected to know. Search only helps if you know what to search for, a knowledgeable friend cannot sit beside you for every match, and private coaching can cost $60–$100 per hour.
That gap became personal when I started playing on public courts during the pandemic. I remember feeling like a baby deer learning to walk—even when nobody was watching. I live in Los Angeles City Council District 10, and I know I am not alone. Young players, volunteer coaches, and new fans across Districts 8, 9, and 10 may be able to reach a public court or stream a match, yet still lack access to the knowledge that makes tennis understandable.
Baseline treats understanding as a form of digital access. We set out to make a tennis broadcast that teaches instead of assuming: a Learner’s Feed.
What it does
Baseline lets someone watch tennis with an AI coach. The working prototype provides a shared live-style match feed with a three-minute lookback, plus an upload flow for recorded matches.
For every point, Baseline:
- finds the exact point interval;
- watches and listens to that point with tennis-specialized AI;
- turns the analysis into a short beginner lesson explaining what happened, why it worked, and what to watch for;
- speaks the lesson in a consistent coach voice; and
- answers follow-up questions using the point currently on screen as context.
A viewer can pause, rewind, seek to a point, and ask questions such as “Why crosscourt?”, “What is a winner?”, or “What should I copy?” The experience is built around the match rather than around a generic chatbot.
How we built it
We built the complete prototype in one day on a single NVIDIA DGX Spark.
The product uses a responsive browser interface and a FastAPI backend. FFmpeg creates analysis clips and media proxies, while a persistent vLLM service runs NVIDIA Tennis Nemotron locally. Each point is analyzed with its broadcast audio and three seconds of post-point context so the model can use player reactions, calls, and the immediate aftermath—not only sparse visual frames.
Finding point boundaries became its own computer-vision system. Professional broadcasts usually stay on a wide court camera during a rally and cut away when the point ends. CourtNet-v1 samples the video at 2 FPS and predicts 14 court landmarks plus a court-line mask. Sustained court visibility becomes a candidate point interval, and a review interface lets a human adjust, add, or delete boundaries before analysis.
CourtNet has 1,629,327 trainable parameters, starts from random initialization, and uses no pretrained backbone. We generated all training images and exact labels procedurally with a deterministic Three.js court renderer, randomizing surface, camera, lighting, wear, shadows, occlusion, overlays, blur, and compression. We trained a 1,000 / 2,500 / 10,000-image ladder and deployed the 10,000-image checkpoint. Across our two unique evaluation videos, it recovered all 18/18 incumbent point intervals with no extra final intervals; on the human-labeled video, its one changed boundary was closer to the human marker.
The coaching layer converts technical analysis into concise learner language. Breeze TTS 2 provides an approved, consistent original coach voice. Playback uses the video as the master clock, so pausing, buffering, and seeking also pause or reposition the independent coaching audio. The prototype is deployed through a stable Cloudflare tunnel at baseline.rerouted.dev.
Challenges we faced
Reliable point detection
Our first attempt asked a large multimodal model to locate points directly. It missed real points and invented others. A smaller, purpose-built court detector was both faster and more controllable.
Then we discovered that the detector we initially tested had no declared software license. Rather than ship uncertain code or weights, we replaced it with our own architecture and a fully synthetic training pipeline. The hardest part was not achieving high synthetic validation accuracy—it was crossing the sim-to-real gap. The 1,000-image model looked good on synthetic validation but detected 0/8 real intervals. The 2,500-image model suddenly recovered all eight, and the 10,000-image model improved boundary accuracy further.
Broadcast cuts inside a point
A close-up can briefly interrupt the wide camera even though the rally or point sequence should remain one interval. Treating every cut as a boundary produced tiny fragments and overlapping analyses. We added temporal post-processing that can merge short camera-cut fragments while enforcing a final minimum interval of five seconds. Human review remains available because unfamiliar broadcast styles can still fail.
Getting the ending right
Tennis Nemotron sometimes made the wrong call when a clip ended on the final shot. Extending analysis three seconds beyond the exact UI boundary helped, but the biggest lesson came from audio: on one difficult point, video-only analysis named the wrong winner, while the same video with broadcast audio correctly understood the ending. We now preserve exact point boundaries for playback while supplying a separate context-rich clip for analysis.
We also stopped treating a generative model’s guess as official truth. Nemotron explains visible tactics; future scorebug OCR and Sportradar alignment will own score, server, and winner facts. That separation makes the system more honest and easier to improve.
Synchronizing two media streams
The coach voice is a separate audio element from the match video. Without explicit synchronization, it continued speaking when the viewer paused or buffered. We made the video the source of truth: coach audio pauses at the exact offset, resumes with playback, and restarts the relevant lesson after a seek settles.
What we learned
- Access is not only connectivity. A stream can be technically available and still be inaccessible if it assumes knowledge the viewer never had a chance to learn.
- The smallest model with the clearest job often wins. CourtNet only needs to recognize court geometry; the larger tennis model only needs to explain a bounded point.
- Context beats confident guessing. Three seconds of aftermath and the audio track mattered more than simply asking the model harder.
- Synthetic data can replace uncertain dependencies—but validation must be real. Perfect synthetic metrics did not guarantee transfer; the failed 1,000-image run was as important as the successful models.
- Trust requires explicit evidence ownership. Detection, tactical explanation, learner editing, voice, and official match facts should each come from the component best suited to own them.
- Timing is part of the product. A correct lesson delivered at the wrong moment—or audio that ignores pause and seek—still feels broken.
What’s next
Next, we would connect live match ingest, scorebug OCR, and Sportradar’s official event feed through confidence-gated temporal alignment; broaden CourtNet evaluation across more tournaments, camera angles, and community footage; and test the Learner’s Feed with youth players and volunteer coaches in Los Angeles Districts 8, 9, and 10.
Our goal is simple: turn watching tennis into a way to learn tennis.
Log in or sign up for Devpost to join the conversation.