Watch how you Box

Landing Page
Customize Avatar (designed by me using blender)
Height and weight to estimate arm length and force of punch
Choose mode
Learn to box with a coach guiding you
Follow the coach
At the gym (play around)
Calibration

Inspiration

Learning to box without a coach is hard, because you can’t see your own form. We wanted a coach that sees your arm and talks to you, using things you already own: a smartwatch and a phone.

What it does

Watch how u Box tracks your lead arm with a Galaxy Watch on the wrist and a phone strapped to your bicep. A 3D boxer in the browser copies your arm in real time.

Train: a coach demonstrates about 10 moves and 6 combos. He calls cues out loud, waits for you to throw, then scores each try on form, speed and estimated power. Gym: a heavy bag swings and reacts to your hits, with called combos, rounds and a summary at the end. Free mirror: shadowbox and watch the boxer move with you. History: every session is saved. With an optional Gemini key, you get a full progress report that you can read or download. Pairing is simple: the phone scans a QR code, and the watch enters a 4-letter code once.

How we built it

Frontend: React, TypeScript, Vite and three.js, hosted on Vercel at watcher.fit. Relay: Cloudflare Workers with Durable Objects, one WebSocket room per session. Sensors: a custom Wear OS app (Kotlin) streams the watch’s orientation at about 100 Hz. A web page on the phone reads its motion sensors. Arm tracking: the laptop browser fuses both sensors into upper-arm and forearm directions, after a one-pose calibration that lines up their headings. Move detection: a rule-based detector reads elbow angle, hand speed and hand path. Character: a rigged character from Mixamo. The tracked arm bones are overridden every frame. Gemini API: writes the coach progress reports. ElevenLabs: voiced all 62 of the coach’s lines.

Challenges we ran into

Heading drift: neither sensor has a compass, so their headings drift apart. We solved it with a calibration pose. Machine learning didn’t hold up: we first trained a model to guess the whole arm from the watch alone. It scored well on test data but looked wrong live, so we switched to a second sensor on the upper arm. Combos: punches thrown back to back merged into one motion. We had to split them at the point where the elbow bends back into guard. Natural fidgets: small movements kept registering as moves, so we filter them out. Voice gaps: short clips had silence at both ends, so we record whole sentences and trim the silence automatically.

Accomplishments that we're proud of

Real-time arm mirroring from just a watch and a phone, with no camera. Setup with no IP addresses and no apps to install on the phone: scan a QR code and press Start. A coach that feels like a real trainer: he talks, demonstrates and waits for you. A heavy bag that actually reacts to how hard you hit.

What we learned

Sensor fusion and motion maths with quaternions. Hosting real-time WebSockets on the edge. Simple rules that match how real motion looks beat a model that only scores well on test data. Building with a sensor simulator let us move fast without wearing the hardware all day.

What's next for Watch how you Box

Track both arms (a second watch, or an Arduino over Bluetooth). Let you teach the app your own moves with a personal classifier. One-tap watch pairing through a companion phone app. Head movement and footwork, plus sparring rounds against the coach.