Inspiration
Every one of us has spent hours shooting alone, missing, and guessing why. Private shooting coaches are expensive, and YouTube breakdowns of Steph Curry's form can't tell you what you are doing differently. We wanted a coach that watches your shot, compares it to the pros, and tells you exactly what to fix: one that fits in your pocket.
What it does
Peek is an iPhone app that coaches your jump shot.
- Record a shot with your phone propped up on the sideline.
- Ask Coach, our AI chat, anything about it: "How does my release compare to Steph Curry's?"
- Get real fixes. Peek tracks your body in 3D, lines your release up with one of ten NBA shooters (Curry, Lillard, Kyrie, Ray Allen, Anthony Edwards, Luke Kennard, Luka, DiVincenzo, LaMelo, SGA), and Coach turns the differences into plain-English cues like "hold your follow-through" or "keep your elbow under the ball."
- Track every session. Clips save on the phone with slow-motion replay, shots you tag as made or missed, and a side-by-side skeleton comparison you can open in AR.
How we built it
iPhone app. Expo and React Native in TypeScript, with expo-camera for capture, expo-video for replay and SQLite for on-device session history. We designed every screen in Figma first.
On-device pose tracking. We wrote a custom Expo native module in Swift on Apple's Vision framework (VNDetectHumanBodyPose3DRequest, plus 2D body and hand pose). It finds the release frame automatically by watching for the wrist rising above the head with the arm extending. Then it aligns the shot to the same release time and body frame as the NBA references.
Shot Coach pipeline (Python). For deeper analysis, a Mac worker runs RTMPose (via rtmlib) for 2D keypoints and PoseMamba to lift them into a 17-joint 3D skeleton. It smooths the motion with a zero-phase Butterworth filter and clips each shot to the window from 1.0 s before release to 0.5 s after (46 frames at 30 fps).
Measuring form. From the 3D joints we compute 12 release metrics, including elbow angle at release and set point, knee dip, dip-to-release time, release height, trunk lean, elbow flare, wrist snap and follow-through hold. Joint angles come from the vectors between joints:
$$ \theta = \arccos\left(\frac{\mathbf{u}\cdot\mathbf{v}}{\lVert\mathbf{u}\rVert\,\lVert\mathbf{v}\rVert}\right) $$
To compare whole motions, we line up the elbow, knee, wrist-height and hip-rise curves against an NBA template. We take the root-mean-square error at the best small time shift ℓ (a few frames either way), so a slightly faster or slower shot isn't penalized:
$$ d(a,b)=\min_{\ell}\sqrt{\frac{1}{N}\sum_{i}\left(a_i-b_{i-\ell}\right)^2} $$
Left-handed shooters are mirrored so everyone is compared against the same right-handed templates.
Coach. An AI agent built with LangGraph and Claude (Anthropic). A form-analyst subagent returns structured findings, validated with Zod, from the measurements, and Coach writes them up as short, specific tips. Chat history persists in Postgres.
Backend. A Fastify API on Vercel, Supabase for auth and Postgres, private Vercel Blob storage for uploads, and FFmpeg to cut each tagged shot into its own clip.
Challenges we ran into
- One camera, no depth. A single iPhone video is 2D, but shooting form is 3D. Lifting 2D keypoints into 3D, and correcting for the camera angle, took the most iteration.
- Finding the exact release. Every comparison depends on the release frame. Detecting it automatically from noisy keypoints, without the player tagging it, took a lot of tuning.
- Apples to apples. Shots differ in speed, handedness and camera position. Aligning everything to the release, mirroring lefties and searching small time lags made the comparisons fair.
- Heavy models, light servers. PyTorch pose models can't run on Vercel's serverless functions. So we moved inference to a Mac worker and kept a fast on-device path for guests.
- 36 hours. None of us had built a computer vision pipeline before, so we scoped hard: one drill (the catch-and-shoot three), done well.
What we learned
- How modern pose estimation works end to end: 2D keypoint detection, 3D lifting and signal smoothing.
- How much of good coaching is alignment. Comparing at the same moment, from the same angle, matters more than any single number.
- How to build a tool-using AI agent that grounds its advice in real measurements instead of generic tips.
- How to ship native iOS code inside an Expo app, and how to wire auth, storage and background jobs across Supabase and Vercel.
What's next
More drills (free throws, pull-ups, layups), more sports from the same capture-and-coach loop, and a progress view that tracks how your form changes over time.