FireRoom

Inspiration

We wanted to make boxing practice more accessible and engaging at home. A phone already has a camera, and a TV provides space for immersive demonstrations and gameplay. FireRoom brings them together to turn the living room into an interactive boxing practice space.

What it does

FireRoom connects a smartphone camera to a TV experience through QR pairing. The phone tracks body landmarks, allowing players to practise jabs, crosses, and jab-cross combinations.

Coach Mode

• Translucent 3D ghost arms demonstrate movements at adjustable playback speeds.
• Guided lessons provide countdowns, practice windows, and movement feedback.
• Movement evaluation checks detected punch sequences, recovery, and the opposite hand’s guard.
• A virtual punching bag responds to recognized punches.
• A hands-free flow guides players through setup, demonstrations, practice, and feedback.

Coach Atlas

Coach Atlas provides evidence-grounded feedback and supports commands for replaying demonstrations, slowing playback, and reviewing performance. The implemented version uses deterministic offline coaching.

Arcade Mode

Players face an animated opponent with timed attacks, health, scoring, combination bonuses, and a finishing sequence. Phone pairing carries across Coach and Arcade modes.

How we built it

We built FireRoom using TypeScript, Vite, Three.js, and MediaPipe Pose.

MediaPipe processes camera frames on the phone. A relay sends timestamped pose landmarks to the viewer, keeping raw camera video out of the gameplay transport.

Deterministic movement rules identify punches and recovery. Shared data contracts connect tracking, coaching, animation, and gameplay.

Three.js powers the boxing environment, ghost-arm demonstrations, responsive punching bag, and opponent animations. We also built TV application wrappers and installed and launched FireRoom in the Vega simulator.

Challenges we ran into

The hardest challenge was distinguishing deliberate punches from ordinary hand movements and guard adjustments. Shifting landmarks, incomplete recovery, and uncertain camera observations could interrupt recognition or lesson progression.

Cross-device integration introduced additional challenges:

• Camera permissions and HTTPS access.
• Delayed pose messages and interrupted tracking.
• Reconnection and pairing continuity between modes.
• Simulator stability and remote-control behavior.

We developed recording, replay, and diagnostic tools to investigate these problems using captured pose evidence.

Accomplishments that we're proud of

We connected phone-based tracking, 3D demonstrations, guided lessons, coaching feedback, and Arcade gameplay in one prototype.

We also implemented a hands-free coaching flow with pause, replay, and retry controls.

Our production build succeeded and FireRoom was also installed and launched in the Vega simulator.

What we learned

We learned that detecting a punch and evaluating technique are separate problems. A recognized movement does not prove correct form, and uncertain tracking should lead to cautious feedback.

Recorded pose replay became essential for reproducing failures and checking whether changes affected previously working movements.

We also learned to distinguish automated software test results from real-world camera accuracy and device validation.

What's next for FireRoom

We plan to expand FireRoom with more personalized training and coaching:

• DTW-based movement comparison: Use Dynamic Time Warping to compare a player’s movement sequence with reviewed reference demonstrations while accounting for differences in pace.
• LLM-powered Coach Atlas: Extend the current offline coach with conversational AI that answers technique questions, explains observed feedback, and selects drills through validated lesson commands.
• Personalized progression: Use performance history to recommend practice areas and adjust lesson difficulty.
• Broader real-world validation: Test recognition, coaching feedback, and complete gameplay sessions across more players, phone cameras, and Fire TV hardware.

Built With

• mediapipe
• three.js
• typescript
• vite