Inspiration

At the Winter Olympics and X Games, snowboarders launch 30 feet high executing triple and quadruple corks with up to 2160° (6 full rotations). Judging and broadcast analysis still rely on subjective visual estimates from distant 2D monitors:

Human Eye Limits: Distinguishing 180° differences or subtle off-axis cork tilts in real time may be inaccurate.
Camera Ego-Motion: Rapid telephoto camera panning masks or exaggerates an athlete's physical rotation.
Zero Sensor Suits: Athletes cannot wear cumbersome IMUs, suits, or optical markers in competition.

Snow VAR extracts ground-truth 3D biomechanics, isolates true rotation in world space, and provides automated trick classification and Olympic judging directly from standard 2D broadcast video.

What It Does

Scouting & Ingest: Automatically detects scene cuts and ranks super-slow-motion replay clips where the athlete is sharpest.
Sensorless 77-Joint 3D Pose: Tracks the athlete (YOLO11 + ByteTrack) and calls the NVIDIA 3D Body Pose NIM to reconstruct a full 77-joint Nova skeleton in 3D camera space—with zero markers or wearable hardware.
Camera Ego-Motion Subtraction: Computes inter-frame background homographies to cancel the broadcast camera's pan, tilt, and zoom, isolating the rider's true physical rotation in world space.
Airborne Segmenter (Model A): Uses a temporal 1D CNN to find exact takeoff and landing timestamps.
Factored Trick Classifier (Model B): Decomposes complex aerials across 5 decoupled heads: spin magnitude (180°–1800° +), sense (CW/CCW), stance (regular/goofy), axis (flat/cork/double cork), and grab (indy, mute, melon, etc.).
Olympic FIS Score Prediction (Model C): Evaluates trick difficulty against 4,995 real World Cup and Olympic jumps (543 official FIS score sheets), predicting landing probability and judge panel marks (0–100, 7.8 MAE).
Omniverse USD Digital Twin: Exports camera-compensated skeleton motion directly into text USD (.usda), running natively in NVIDIA Omniverse, Isaac Sim, and Blender.
Interactive VAR Dashboard & Nemotron Coach: A React/Three.js console with synchronized 4K video scrubbing, 3D orbitable skeleton inspection, telemetry curves (RPM, rotation accumulation, knee flexion), and an AI reasoning coach powered by NVIDIA Nemotron-3 Ultra (550B).

How We Built It

NVIDIA 3D Body Pose NIM: Monocular 2D-to-3D pose estimation returning 77 Nova joints via gRPC on NVCF / self-hosted GPU.
NVIDIA Nemotron-3 Ultra (550B): Generates structured biomechanical coaching reports (nvidia/nemotron-3-ultra-550b-a55b via NVIDIA API) validated by a strict runtime JSON schema.
NVIDIA Omniverse & Isaac Sim: Parametric synthetic trick generation with Omniverse Replicator and Blender 4.3 (MPFB 2.0 Nova-77 rig) to establish ground-truth physics benchmarks.
Universal Scene Description (USD): Generates text .usda files conforming to UsdSkelAnimation, validated via USD's skinning resolver.
Computer Vision & Kinematics: YOLO11, ByteTrack, OpenCV homography ego-motion cancellation, and quaternion-based SO(3) rotation integration.
ML & Serving Stack: PyTorch, scikit-learn, FastAPI, React 18, Vite, TypeScript, Canvas 3D, and Nebius Cloud GPU instances (NVIDIA L4).

Key Challenges & Breakthroughs

Camera Pan Distortion: Telephoto pans induce massive false rotation. We isolate background feature points to invert camera motion (𝐑{world} = 𝐑{cam}⁻¹𝐑_{rider}) while preventing rotation accumulation across camera cuts.
Conformed Slow-Motion: 120+ fps footage saved into 30 fps broadcast containers distorts velocity. We normalized all ML features by timebase and torso proportions to make classification independent of video frame rate.
Avoiding "Confident Wrong Numbers": 3D pose estimators can produce smooth yet inaccurate output. We built snow synth (a synthetic physics benchmark) and snow label (a frame-by-frame human annotation tool) to publish verified error margins.

Accomplishments

Zero-Hardware 3D Kinematics: 77-joint metric poses extracted from standard broadcast clips without athlete suits or markers.
Sub-2° Rotation Accuracy: Verified on synthetic ground-truth benchmarks.
Olympic Contest Scoring: Trained on 4,995 real FIS contest jumps, predicting judges' panel marks within 7.8 MAE.
Omniverse Ready: One-click generation of camera-compensated .usda digital twins.
Robustness: 105 automated unit and integration tests running with zero network or GPU dependencies.

What's Next

Edge Deployment: Run real-time inference on NVIDIA Jetson AGX Orin for on-mountain terrain park setups.
Multimodal Commentary: Combine visual tokens with 3D joint telemetry for automated natural- language play-by-play.
Action Sports Expansion: Extend from snowboarding to skateboarding, freestyle skiing, surfing, and BMX.