ApexTactics AI

Turns a basketball clip into a live coach's breakdown — deterministic geometry translated into tactical tokens, then streamed into real coaching commentary by an LLM.

Comment

Inspiration

Every "AI sports analytics" demo we'd seen did the same thing: dump raw player coordinates into an LLM and hope it reasons correctly about distance, velocity, and spacing. It never does — language models don't have a native sense of physics. Feed one Player_1: (45, 120) and it'll hallucinate whatever number sounds plausible.

We wanted to build the thing that actually fixes that, instead of another wrapper that hopes the model gets lucky: a translation layer that makes the physics legible to the model before it ever has to reason about it. Sports gave us the perfect testbed — specifically the high pick-and-roll, a play with a clean, well-defined structure (screen → defensive rotation → drive) that we could ground entirely in real geometry.

What it does

Upload a phone clip of a high pick-and-roll. ApexTactics AI:

  1. Tracks every player and the ball frame-by-frame (YOLOv8 + ByteTrack).
  2. Calibrates pixel space to real court-metric space via a homography from four user-clicked court corners.
  3. Runs that tracked geometry through a deterministic heuristic engine — not an LLM — that emits semantic tokens the instant real, hand-set thresholds are crossed: SCREEN_SET when the ball handler comes within 1.5m of a teammate, DEFENSIVE_COLLAPSE when the defense's convex hull area compresses 25%+, DRIVE_TO_RIM when the ball handler hits 6 m/s toward the basket.
  4. Streams that token sequence to an LLM (via OpenRouter) with one job: explain why that exact sequence of real events led to the outcome, in coaching language.

The result is a two-pane live view: your video on the left, and the tactical tokens plus streamed coaching commentary on the right — appearing in real time as the pipeline works.

How we built it

  • Computer vision: YOLOv8 for player/ball detection, ByteTrack for persistent multi-object tracking, a rolling smoothing filter to absorb short occlusion glitches.
  • Team assignment: jersey-color k-means clustering on sampled torso pixels, since nothing in the tracking output natively distinguishes offense from defense — whichever cluster held the ball most often across the clip is labeled offense.
  • The heuristic engine: pure NumPy/SciPy (ConvexHull, kmeans2) — every threshold is explicit and hand-set, computed once in calibrated court-metric space, never passed to the LLM as raw numbers for it to guess about.
  • Reasoning layer: OpenRouter's unified, OpenAI-compatible API, so the model backing the commentary is swappable via one environment variable with zero code changes.
  • Frontend: Next.js + Tailwind, streaming the backend's NDJSON response live into the UI as tokens and commentary arrive.

Challenges we ran into

Almost everything that could silently break in a real-time CV-to-LLM pipeline, did, at some point:

  • Team assignment was never actually implemented at first — every tracked player was getting dumped into "offense," which meant DEFENSIVE_COLLAPSE could mathematically never fire, on any clip, no matter how good the tracking was. We caught this by adding a diagnostic panel that reported exactly what the engine was seeing per frame, rather than guessing blind at why tokens came back empty.
  • Ball detection was the hardest part. The default YOLOv8 nano model, at the default confidence threshold and the default 640px inference resolution, essentially never saw the basketball at all in real phone footage — a distant, fast-moving ball is genuinely one of the harder small-object detection cases. Fixing it took three separate levers together: a bigger model, a much lower ball-specific confidence threshold, and doubling the inference resolution.
  • A pixel-vs-real-world coordinate bug quietly made ball-handler assignment meaningless — comparing raw pixel coordinates against post-calibration, real-meter court coordinates, which silently produces a "nearest player" that isn't actually nearest to anything.
  • Free-tier LLM routing churn. Model slugs we relied on got deprecated or rate-limited mid-build, which pushed us to build automatic retry-with-backoff around OpenRouter's own reported wait times, rather than hardcoding a single model and hoping it stays available.
  • A subtle React race condition where two concurrent stream requests (from React's own development-mode double-effect behavior) could both write into the same commentary state at once, interleaving two separate LLM completions into unreadable text — fixed by explicitly gating every state update on which request was still the active one, rather than trusting the network layer to clean up after itself.

Accomplishments that we're proud of

  • A heuristic engine that's genuinely tested against hand-built mock coordinate data with real assertions, not visual inspection — which caught a real off-by-one bug in our temporal lookback logic before it ever reached a live demo.
  • Getting ball detection from a hard zero to a real, working detection rate through actual diagnosis rather than guesswork — we built a debug panel that reports frame counts, detection rates, and confidence scores, which is what let us find and fix each bottleneck in the chain instead of blindly tuning parameters.
  • A reasoning layer that's provider-agnostic by design — swapping the underlying LLM is a one-line config change, not a rewrite.
  • Being honest, in our own documentation, about exactly where the system's limitations are — calibration accuracy, team-assignment assumptions, ball-detection recall — rather than overselling a demo.

What we learned

The hardest part of "AI + sports" isn't the AI. It's everything upstream of it: getting real-world tracking data clean enough, calibrated correctly, and split into the right semantic categories that the reasoning layer can actually trust. Every bug we hit was in that translation layer, never in the LLM itself — which validated the entire premise of the project: if you hand a language model clean, pre-computed, ground-truth facts instead of raw numbers, it does a genuinely good job of reasoning about them. The work is making sure what you hand it is actually clean and actually ground-truth.

What's next for ApexTactics AI

  • Extend the same translation-layer approach to more set pieces — isolation drives, off-ball screens — now that the pick-and-roll pipeline is proven reliable.
  • Multi-camera fusion for full 3D spatial mapping, removing the single-fixed-angle constraint.
  • Server-side caching of computed token sequences so a reasoning-layer-only failure can retry instantly without re-running the full CV pipeline.
  • A classical color/blob-based ball detector as a fallback path for footage where general-purpose object detection still struggles with a small, fast ball.
  • Season-long player tracking for longitudinal development data, moving from single-clip analysis to an ongoing coaching tool.

Built With

Share this project:

Updates