Inspiration
I run Clann, an app that reviews BJJ sparring rolls chess.com-style. People love the review, but reading it isn't the same as having a coach next to you pointing at the screen. So I wanted a coach with a face who has already watched your roll, and who you can just talk to.
What it does
You pick which fighter you are and a coach (four personalities, each with their own face and voice), then talk:
- "Break down the Americana": slow-mo up to the moment, then a broadcast-style freeze (the gym goes grey, the two fighters stay in colour), YOU/HIM outlines on the real bodies, a glowing limb and a labelled arrow ("your right knee → his left elbow").
- "Show me that in 3D": the real frame lifted into 3D with a depth model, so you can spin round the actual position.
- Side-by-side of two moments, a "ghost" of your better shape, and a 10-second clip you can save.
- Alongside: an eval graph across the roll, 💀 for every tap, and every move rated brilliant to blunder.
- At the end, a Call of Duty-style after-action report: stats, XP, medals and your drill for the week.
How we built it
- Voice and reasoning: Gemini Live inside a LiveKit agent. The coach "presses buttons" on the page over LiveKit RPC (show_moment, freeze_at, highlight, arrow, spotlight, show_3d, compare, ghost, save_clip).
- Face: Anam real-time avatar.
- Body tracking: YOLO11 pose + segmentation on every frame, with bystanders filtered out. Gemini Flash checks who's who against each fighter's photo at the key moments, and the coach only draws where the tracking was verified.
- 3D: Depth Anything V2 estimates depth for the frame, and the browser renders it as a point cloud in three.js.
- The move-by-move review of the roll comes from Clann, my existing app, through its public share API. All the code for the interactive coach was written during the hackathon (the repo's first commit is Saturday 16:26).
Challenges we ran into
- Two bodies tangled on a mat is the worst case for pose tracking: the tracker sees one blob. I filtered out bystanders, used Gemini to check identities against the fighters' photos, and only allow drawings at moments I'd verified.
- My first 3D (a separate mannequin per fighter) looked like two dolls standing apart. Lifting the real frame with a depth model fixed that.
- Getting a voice model to drive the video reliably meant giving it a small set of clear tools and a "director" order: replay → freeze → draw.
Accomplishments that we're proud of
python
- You can talk to a coach with a face, and it actually drives the video: it jumps, slows down, freezes and draws on the right person.
- Real footage in 3D, from a single phone video, with no special camera.
- It feels like a game: a chess.com-style review during the call and a Call of Duty-style report at the end.
What we learned
Overlays only build trust if they sit on the right person. Being honest about where the tracking is reliable mattered more than drawing everywhere. And voice-first changes everything: athletes ask "why did I lose that?" in a way they never would by typing.
What's next for interactive jiujitsu coach (clann)
- Bring the live coach into the Clann app for every reviewed roll, not just the demo one.
- Verify the tracking automatically across the whole roll, so the coach can draw anywhere.
- A real coach's face and voice: my own coach has agreed to trial it.
- Use athletes' corrections as training data to make the review better.
Built With
- anam
- depth-anything
- fastapi
- gemini
- javascript
- livekit
- mediapipe
- python
- three.js
- yolo
