Inspiration

Iris is a member of UW Figure Skating and spends much of her free time on the ice. Like many figure skaters, she takes private lessons with a coach who corrects her jumps, spins, and mistakes she might not notice herself. But outside those 30-minute lessons, she's practicing independently.

She can spend hours repeating the same movement, knowing something feels wrong but not knowing exactly where, when, or why. So she records her practices, but seeing a mistake doesn't necessarily mean understanding how to fix it. Even existing AI tools struggle, often misidentifying jumps, miscounting rotations, or giving generic advice unrelated to her actual technique.

That's the gap we wanted to address at DubHacks.

We built With., an AI-powered figure skating app that analyzes skater's movements and providing personalized, technique-specific feedback between lessons. Our goal isn't to replace coaches, but to build something that works With. skaters and their coaches, extending that support beyond their time together on the ice.

Skating is very hard to analyze from video. A triple Lutz is about half a second in the air at up to 6 revolutions per second, and what separates a Lutz from a Flip is which blade edge she takes off from. General vision-language models were never trained to see that. We wanted a tool that actually understands skating.

How we built it

The key idea: don't ask an LLM to see skating. Give it measurements. We built a custom harness that chains specialized models together:

Pose: MediaPipe extracts a 2D skeleton from each frame. A VLM (Perceptron) tracks the main skater so we follow the right person on a crowded rink and also helps in clip segmentation.
Custom skating model: we trained a 7.1M-parameter temporal model on competition data (MMFS and FineFS), using the UW Hyak and Tillicum GPU clusters. It detects jumps, spins and step sequences, and predicts the jump type, rotations, falls and Grade of Execution (GOE).
ISU scoring and physics: each element is scored the way judges do, and jump height comes from air time $t$:

$$ \text{TES} = \sum_i \left(\text{BV}_i + \tfrac{g_i}{10}\text{BV}_i\right), \qquad h = \frac{g\,t^2}{8} $$

Here $\text{BV}_i$ is the element's base value from the ISU table and $g_i \in [-5, 5]$ is the predicted grade.

LLM last: Gemini, with NVIDIA Nemotron as a fallback, gets the structured results and ISU rules and turns them into coaching that points to timestamps.

A FastAPI backend serves a Next.js app and an iMessage agent built on Photon, so skaters can text in a clip from the rink.

Results

On our benchmark of hard skating-analysis tasks, frontier foundation models scored 40–50%. Our harness scored 83–85%, and our scores land within about 0.2 points of the official judges' scores for Olympic skaters. On held-out test data, the skating model identified jump type correctly 98.5% of the time and counted rotations correctly 97.1% of the time.

Challenges

3D training data, 2D phone video. Models trained on 3D motion capture returned garbage on real clips. We retrained on 2D skeletons with added pose noise so they hold up on real phone video.
Skating breaks pose estimators. Spins, motion blur and crowded rinks confuse pose models, so we combined VLM person tracking with the skeleton.
Keeping the LLM honest. Early versions made up corrections. We now give the LLM only measured evidence and ISU rules to work from.

What we learned

A small domain-specific model that measures, paired with an LLM that explains, beats a frontier model working alone. Making training data match what users actually upload mattered more than model size. And because ISU judging is formal enough to write as equations, we could check ourselves against real judges.

Beyond skating, our custom harness may prove to have widespread applications in complex motion comprehension.