SwingLab

Inspiration

What set off this project was a set of computer-vision demos that analyze other sports and exercises, mainly Jeremy Park's vision-demos repo, which includes a pull-up tracker and a rock climbing tracker. While brainstorming, we came across Qazi Ammar Arshad's pickleball pipeline, which tracks ball movement and has a live stats overlay.

We decided on baseball due to the lack of cheap and easily accessible tools for training a swing. Learning to hit a baseball is hard, and it's harder without a coach. Most beginners never get real feedback on their swing. Pro and college hitters get biomechanics labs with motion-capture suits and force plates.

Then we found Driveline Baseball's OpenBiomechanics dataset: 677 real swings from 98 mostly college hitters, recorded in a lab with motion capture, with the bat speed and exit velocity of every swing. The demos that inspired us judge form with hand-written rules and stop at showing numbers. With Driveline's data, we could learn what the best hitters actually do, measure your swing against them, and have Gemini turn the gap into advice you can act on.

What it does

SwingLab is an AI hitting coach for anyone with an iPhone 12 Pro or a newer Pro model.

Film your swing. Use the Record3D app, which captures color video plus LiDAR depth.
3D rebuild. SwingLab rebuilds you in 3D and finds every swing in the recording automatically. It measures the mechanics that drive bat speed: hand speed, shoulder turn, hip turn, leg drive, front-leg block, contact point, stretch and step-through. These mechanics were built from the Driveline dataset.
Players get a score (0–100). It shows how close your swing, based on each of those 8 factors, is to the top hitters in the Driveline lab data, adjusted for your height, weight, age and level.
Coach report. Gemini acts as a coach and writes a report based on SwingLab's analysis that explains the weak points of a player's form. It will then suggest what to work on, drills, a training plan and exercise videos. For example, if a player's hip flexibility is an issue, it will suggest hip-strengthening exercises and stretches.
Ask the coach. Ask Gemini anything about the swing, how to improve, or how the player feels afterwards.
Ball detection tells hits from misses. An object-detection model (YOLOX) looks for a real baseball in the video. If it sees the ball fly off the bat, the swing is a hit and also gets a contact score. If the ball goes past, it's a miss. With no ball, it's a practice (dry) swing, so you can train with or without one.

The viewer shows the annotated video synced with a 3D replay, a kinematic-sequence chart (hips, chest, hands), the factor panel, your key measurements against the lab range, your history over time, and an upload flow with live progress.

SwingLab is meant to be run locally, so anyone can download it and run it on their own computer. Players keep their own videos, and their data stays private.

How we built it

We started with the Driveline dataset. Each of its 677 swings was recorded in a lab with motion capture at 360 frames per second, along with the swing's bat speed and exit velocity. Since a phone films at a much lower frame rate, we cut the lab data down to the same number of frames per second, so our models learned from the same kind of footage a phone produces. We also put every hitter, lab or phone, into the same frame of reference, with lefties mirrored into righties so everyone can be compared. From there we tracked how the hips, chest and hands rotate and speed up, found the key moments of each swing (load, foot plant and contact) and measured the mechanics. To make sure our numbers were right, we checked them against Driveline's own published measurements. Even at phone frame rates, we found the moment the bat meets the ball within 50 milliseconds, faster than a blink, on all 677 lab swings, and our hip-rotation speeds lined up closely with Driveline's lab measurements.

Next, we needed to decide what an elite swing looks like. Bigger, older and more experienced hitters naturally swing harder, so we took height, weight, age and level out of the equation and called the top third of what was left "elite." Our bat-speed model predicts within 3.1 mph on average, compared with 3.8 mph for just guessing the average, and it was always tested on hitters it had never seen. To score a swing, each measurement is compared with the elite hitters. For example, a hand speed of 5.8 m/s against an elite 7.0 m/s is 83%. Those percentages are averaged into the 8 factors, and the factors are combined into one score, with leg drive counting the most, followed by shoulder and hip turn. We set the scale so only near-elite swings reach the 80s, and anything the phone couldn't measure is skipped rather than guessed.

On the phone side, the Record3D app gives us color video, a LiDAR depth map and the phone's position 60 times a second. MediaPipe, Google's pose model, finds 16 points on the body in each frame and estimates its 3D shape, and the LiDAR depth tells us the body's real size and where it's standing. Phone data is messy, so we added clean-up steps. We filter out bad depth readings when the hands pass in front of the hips, rebuild limbs the camera loses using their known length, and warn the player in plain words when a recording can't be trusted.

A player can get their recordings to the laptop by dropping the file in a folder, uploading it from their phone, or streaming straight from the phone over USB. Since one recording can hold several swings, SwingLab looks for moments when the hands move faster than 3.5 m/s and the shoulders turn at least 80° toward the pitcher, which keeps practice waggles from counting as swings. Each swing gets about 27 measurements, and the score shows up about 30 seconds after the recording lands, with the coach's report and an annotated video following about a minute later. LiDAR can't keep up with a ball at full speed, so to tell hits from misses we use the phone's color video instead. A tracker follows anything that moves like a ball, and YOLOX makes sure it's a real baseball and not a light in the background.

For the coaching, our code picks the one thing the player should work on first: their weakest factor and the measurement holding it back. Gemini watches the swing alongside those numbers and writes a plain-language report with a cue, a drill and a training plan, choosing from our library of 24 exercises, each with a video showing how to do it.

Everything is written in Python. A FastAPI server handles uploads and runs the analysis in the background while showing live progress, and the viewer uses three.js for the 3D replay next to the annotated video. The demo site is hosted on Vercel, and 133 automated tests make sure the pipeline keeps working as we change it.

Challenges we ran into

LiDAR can't see the bat or ball. The depth map is low resolution, and a bat head moves about a meter between frames at 30 fps. Instead, we predict bat speed from how the body moves and use the color video to spot the ball.
The body hides itself. Filming from the front, the back shoulder and hip disappear behind the body mid-swing. Our first real recordings had shoulder lines flipping backward and fake swings. We rebuilt the 3D step around MediaPipe's body shape, with LiDAR setting the size and position.
Streaming straight to the laptop. Our first live stream from the phone had a long delay and missed swings. We fixed it by keeping the last few seconds of the stream and saving each swing as its own recording, then showing the score first while the slower steps finish in the background. Now scores show up about 30 seconds after a swing.
Phone frame rates are coarse. At 30 fps, the timing between the hips, chest and hands moves in 33 ms steps. We trained every model at phone frame rate so lab and phone numbers stay comparable.
False ball detections. A lit window and a drifting light got called "balls," which turned dry swings into misses. Now a swing counts as dry unless YOLOX confirms a real baseball.
Hosting vs. privacy. A LiDAR recording can be 100 MB, but Vercel accepts only 4.5 MB per request. Instead of storing people's videos in the cloud, we made SwingLab run on your own computer, which also provides better data privacy.
Gemini produced hard-to-decipher text. At first, the coach's reports and suggestions were very technical and hard for new players to understand. We rewrote the prompts and created a response template to better guide Gemini's answers.

Accomplishments that we're proud of

Live capture from the phone's LiDAR sensor directly into the computer.
Building a model that detects contact and timing almost as accurately as Driveline's own lab data.
Creating Gemini outputs that are easy for new players to understand.
Privacy by design, with all data stored locally.

What we learned

Good form has to be defined from outcomes. Driveline never grades a swing as good or bad; it records bat speed and exit velocity. Body size alone explains a big share of bat speed, so "good form" had to mean what the hitters who swing hardest for their size do, not just who swings hardest.
Where the camera stands changes everything. Filming from the front, the back shoulder and hip hide behind the body, and the LiDAR reads the near side instead. Trusting depth joint by joint flipped our shoulder lines backward, so we learned to take the body's shape from MediaPipe and use LiDAR only for size and position.
Train on the data you'll actually get. A phone records far fewer frames per second than a lab. Resampling Driveline's 360 Hz data down to phone speed before training is what made lab and phone numbers comparable.
Gemini is a better explainer than a calculator. When we gave it exact measurements, plain-language fault phrases and a fixed report template, it coached clearly. Left on its own, it wrote technical text a beginner couldn't follow. We also check every number it writes against our measurements, so it can explain the data but never invent it.
Real recordings find problems synthetic tests miss. Night lighting, dark sleeves and a depth range that cut off at about 3 meters only showed up once we filmed real swings.

What's next for SwingLab

Pitching: record and analyze pitchers too. Driveline publishes pitching data in the same format.
Faster results: cut the wait from recording to report.
Better Gemini prompting: make it a sharper coach and a more careful analyzer.
More real-world testing: field sessions with real hits to tune the contact score and ball detection.