Inspiration

Many games, like local leagues, school games or niche sports have no commentator. We want every match to have a caster which can announce game events, in almost any language.

What it does

Connect to the video and press play, computer vision will follow the game, one/two AI casters announce events with real voices, possibility to switch languages mid-sentence. It covers basketball, football and ultimate frisbee.

How we built it

Computer vision: YOLO detectors with tracking and pitch keyboints (TensorRT on NVIDIA, CoreML on Mac). Writing: Claude Haiku writes commentary. Judging: a decider model chooses what to say and what to not say. Voices: ElevenLabs streams them, with per word timings for language switching. Server: FastAPI server with 5 second buffer so every event announcement is precisely timed.

Challenges we ran into

Language switching mid-sentence without sentence starting over, jitter on tracking boxes, under different light distinguish players of different team, getting models to run on both Windows and Mac, keeping speech in sync with the video.

Accomplishments that we're proud of

Live translation at once switching mid-sententce. Casters who banter, if two - they call each other by name and back rival teams. Live football tracking at about 40 frames per second on laptop GPU, one platform runs all three sports.

What we learned

Timing matters more than wording: a great line that is late feels wrong, it needs to be timed. plenty of ux details, like warm up process which is important.

What's next for Courtsight

More sports and leagues, real life pilot with FK "Žalgiris".

Built With

  • blood
  • claude
  • codex
  • elevenlabs
  • sweat
  • tears
Share this project:

Updates

Submission history