Playback

Playback gives amateur basketball players pro coaching. Its agentic AI pipeline uses Google Cloud, Gemini, and ElevenLabs to turn raw footage into personalized heatmaps and tactical insights.

Comment

Inspiration

Amateur and pickup basketball players rarely get the tactical feedback and coaching that professionals rely on to improve their game. Watching a recording of your own pickup game is helpful, but without a trained eye, it's hard to identify spacing issues, defensive lapses, or optimal shot selection. We were inspired to build Playback to democratize access to elite sports analysis by giving everyday players a "coach in their pocket" that provides personalized, data-driven, and actionable insights without needing to hire a coach. Our design inspiration came from premium fitness apps like Nike Training Club to ensure the experience felt both professional and highly engaging from the moment you open the app.

What it does

Playback is an AI-powered basketball coaching app that analyzes amateur game footage and generates a comprehensive, interactive dashboard. Users simply upload a video of their game and provide a brief description of what they are wearing. Our agentic AI pipeline processes the video to identify the player, track their movements, and analyze their performance. The app then generates an immersive dashboard featuring:

  • Positional Heatmaps: Visualizing where the player spent the most time on the court and their movement tendencies.
  • Tactical Analysis & Replay: Highlighting key moments alongside the video timeline, such as open looks, great setups, or defensive lapses.
  • Voice Coaching: Providing personalized audio feedback and recommendations, mimicking a real coach's post-game breakdown.

How we built it

We split the architecture into a decoupled frontend and backend, synchronized via a strictly defined data contract (contract.json), allowing our team to work in parallel.

Frontend: Built with Next.js 15 (App Router) and Tailwind CSS. We focused on delivering a premium, dark-mode sports aesthetic with responsive micro-animations for the upload flow and processing dashboard. Backend & AI Pipeline: Powered by a Python FastAPI server, the core of Playback is an 8-step agentic orchestrator. We utilized Gemini function calling to chain multiple services together autonomously:

  • Google Cloud Video Intelligence: For complex person detection and tracking.
  • Gemini Vision: For frame-by-frame tactical breakdowns and generating strategic coaching insights.
  • ElevenLabs TTS: To give our AI coach a realistic, commanding, and encouraging voice.

Challenges we ran into

We ran into persistent timeout and connection aborted errors when handling large video uploads to Google Cloud Storage, requiring us to dive deep into our Google Cloud authentication and pipeline connection strategies. Additionally, marrying the dense analytical data from the backend to a clean, premium frontend required careful management of state. We had to design an engaging animated processing overlay just to keep users invested while the heavy multi-stage AI analysis compiled in the background.

Accomplishments that we're proud of

We are incredibly proud of the seamless integration between our multi-modal AI models. Successfully passing positional data from Google Cloud Video Intelligence to Gemini Vision for tactical understanding, and then translating that into human-like audio via ElevenLabs, felt like a massive win. Furthermore, upgrading the frontend to a high-end UI elevated the entire project from a basic hackathon prototype to an application that feels genuinely polished and ready for consumer use.

What we learned

We learned a tremendous amount about agentic tool calling and how to effectively orchestrate an LLM's sequence of operations. We also deepened our knowledge of Google Cloud services, specifically handling large media files and cloud-native authentication securely. On the frontend, we discovered how crucial thoughtful UI transitions and strict mock-data contracts are when parallelizing development between backend and frontend developers.

What's next for Playback

For the next phase, we plan to integrate a ChromaDB-backed Retrieval-Augmented Generation (RAG) system to ground our AI coach in a deep knowledge base of historical basketball plays, drills, and professional coaching methodologies. We also want to expand the model to support multi-player team analytics and integrate social features so players can share their coaching breakdowns and highlights directly with their pickup group.

Share this project:

Updates