JNKS

JNKS is an AI coach that watches your (missed) shots so your friends don’t have to. Real-time feedback, form fixes, pro comparisons, and voice coaching that takes you from brick to slick in a click.

Comment 4

JNKS: The AI Coach Turning Bricks into Buckets

Inspiration

JNKS isn't just a clever name; it's the heartbeat of our team (comprised of the initials of our four founders). We came together with a shared belief: Physical fitness is the most undervalued pillar of healthcare. In a world increasingly dominated by sedentary entertainment, we wanted to bridge the gap between "gaming" and "getting active." We realized that many people quit sports like basketball because they lack the feedback to improve. By building an AI coach, we’re building a tool that promotes long-term musculoskeletal health through proper form and cardiovascular wellness through gamified movement, not just building a sports app.

How We Built It

We didn't just want JNKS to work; we wanted it to feel like a premium, high-performance product.

The Frontend

To achieve the polished look on our landing page, we used a cutting-edge stack:

  • Next.js 16 (React 19) & TypeScript: Providing the robust architecture needed for complex state management.
  • Tailwind CSS 4: For rapid, responsive styling that keeps our UI lightweight.
  • Three.js / React Three Fiber: We integrated a 3D basketball model to create an immersive, interactive environment.
  • GSAP & Framer Motion: These were used to handle the fluid transitions and high-end animations that give the app its premium feeling.

The Computer Vision Pipeline

Our analysis engine is a multi-stage pipeline:

  1. Pose Estimation: Using MediaPipe, we track a 33-point skeletal map of the user in real-time.
  2. Ball Tracking: We custom-trained a YOLOv8 (Ultralytics) model on thousands of frames of pro shooters like Steph Curry and Kevin Durant.
  3. Pro Comparison Logic: We compare the user's joint angles to the pro dataset using Cosine Similarity to calculate how close the user's form is to the "Gold Standard."

Mathematical Breakdown

Joint Angle Computation

For three landmark points a, b, c (angle measured at vertex b):

$$\vec{u} = a - b, \quad \vec{v} = c - b$$

$$\theta = \arccos\left(\frac{\vec{u} \cdot \vec{v}}{|\vec{u}| |\vec{v}|}\right)$$

This is the interior angle at joint b in degrees. Applied to elbow (shoulder–elbow–wrist), knee (hip knee–ankle), shoulder (hip–shoulder–elbow), and hip (shoulder–hip–knee).

Elbow Angle Score

Deviation from the biomechanical ideal of 90°:

$$S_{\text{elbow}}^{\text{MP}} = \max!\Big(0, 100 - 2,\big|\bar{\theta}_{\text{elbow}} - 90\big|\Big)$$

where $$\bar{\theta}_{\text{elbow}}$$ is the frame-weighted average elbow angle across all clips.

Stance Score

Piecewise scoring based on knee angle range [110°, 130°]:

$$S_{\text{stance}}^{\text{MP}} = \begin{cases} 95 & \text{if } 110 \leq \bar{\theta}_{\text{knee}} \leq 130 \ max!\Big(0, 95 - 2d\Big) & \text{otherwise} \end{cases}$$ where $$d = \min!\big(|110 - \bar{\theta}{\text{knee}}|, |\bar{\theta}{\text{knee}} - 130|\big)$$.

Weighted Average Across Clips

Each clip i contributes proportionally to its frame count ni:

$$\bar{\theta} = \frac{\sum_{i} \bar{\theta}i \cdot n_i}{\sum{i} n_i}$$

Hybrid Category Score

For measurable categories (elbow, stance):

$$S_{\text{cat}} = \lfloor 0.4 \cdot S_{\text{cat}}^{\text{MP}} + 0.6 \cdot S_{\text{cat}}^{\text{Gemini}} \rfloor$$

For vision-only categories (follow-through, release point):

$$S_{\text{cat}} = S_{\text{cat}}^{\text{Gemini}}$$

Reference Comparison — Frame Similarity

For each frame pair, the per-joint angular error is computed:

$$\Delta_j = |\theta_j^{\text{user}} - \theta_j^{\text{ref}}|$$

For wrist height (normalized coordinates), scaled to angular equivalent:

$$\Delta_j = |h_j^{\text{user}} - h_j^{\text{ref}}| \times 150$$

The frame score maps average error linearly to [0, 100] with a 45° ceiling:

$$S_{\text{frame}} = \max!\left(0, 100 - \frac{\bar{\Delta}}{45} \times 100\right)$$

Challenges We Faced: The Real-Time Wall

The biggest hurdle was latency. Running a 33-point pose detection alongside a custom YOLOv8 model while trying to render 3D elements in the browser nearly melted our machines.

We quickly learned that "Real-Time" is a game of trade-offs. We had to optimize our OpenCV frame processing and implement a worker-thread strategy to ensure the UI didn't lag while the "Brain" was thinking. We learned that in the real world, a 95% accurate model that runs at 30 FPS is infinitely more useful than a 99% accurate model that runs at 2 FPS.

What We Learned

This project was a masterclass in optimization. We walked away with a deep appreciation for the engineering that goes into the "seamless" services we use every day.

More importantly, we validated our mission: Healthcare is more than medicine; it's movement. By using the entertainment of basketball as a hook, we can encourage users to stay active, correct their posture, and prevent the injuries that come from improper technique.

What's Coming Next

We were able to use this framework to create a similar version of the app for yoga to show that this pipeline can be applied to other activities. The only difference is that yoga does not have "professionals" to compare to. A link to that repo can be found down below.

Built With

+ 16 more
Share this project:

Updates