MoveQuest

• 

  MoveQuest Logo

Inspiration

Fire TV makes a big screen easy to add to a living room, but most TVs don’t have a camera. We wondered: could the phone people already own become the camera, while the TV does the seeing? That idea became MoveQuest: turn the living room into a place to move and play together, without asking families to buy a console camera or wear a tracker.

What it does

MoveQuest turns body movement and hand gestures into interaction inside a Fire TV app. Choose a workout such as squats, jumping jacks, or arm raises, and the TV displays movement cues, rep counts, detected-player counts, and session progress.

The prototype also includes family-game flows for Freeze Dance, Mimic the Pose, and Gesture Trivia. Open-palm and closed-fist gestures can pause or resume a session; swipes and other hand signs provide simple in-app interactions and feedback.

How we built it

We built the TV experience in React Native, with a Kotlin Android module for receiving and processing camera frames. A companion Android phone app streams MJPEG video over the home Wi‑Fi network. On Fire TV, MediaPipe pose and hand-gesture models analyze the incoming frames; the TV app uses those detections for rep estimation, movement cues, and gesture responses. The phone acts as the camera, not the AI processor. Video stays on the local network and isn’t uploaded to a cloud service.

Challenges we ran into

The biggest challenge was building for a TV that has no built-in camera. We had to make the phone-to-TV stream simple to connect while keeping the computer-vision work on the TV. We also had to account for Fire TV’s limited processing resources, changing Wi‑Fi conditions, and remote-first navigation. Exercise scoring and movement feedback use prototype heuristics, so we worked to make the experience understandable without presenting it as medical-grade form analysis. Fire OS also limits what one app can control: MoveQuest gestures operate the MoveQuest experience, not YouTube or other apps.

Accomplishments that we’re proud of

We connected everyday phone hardware to on-TV pose and hand-gesture recognition, without requiring a dedicated camera accessory or cloud video processing. The TV experience can display live camera preview, estimate reps for several exercises, track up to four people temporarily, and respond to gestures all in one app. We’re especially proud that the design makes the TV the experience and the phone a simple camera companion.

What we learned

We learned that a good computer-vision demo is more than a model: camera placement, lighting, stream reliability, focus states, readable TV layouts, and clear feedback all matter. We also learned to design within platform boundaries instead of promising system-wide gesture control.

What’s next for MoveQuest

Next, we want to test across more Fire TV devices and real living-room conditions, improve tracking stability and exercise scoring, add guided visual demonstrations, and make calibration easier. We’d also like to expand the multiplayer games while keeping participation privacy-preserving and video processing local.

Built With

• amazon-fire-tv
• android
• camera
• computer-vision
• expo.io
• fire-os
• firetv
• gesture-recognition
• javascript
• kotlin
• local-network
• mcp
• mediapipe
• mjpeg-streaming
• mobile-app
• on-device-ai
• pose-estimation
• react-native
• real-time-processing
• wi-fi