• 

  QuestMove - The Moving Forest

Inspiration

QuestMove was inspired by my daughter.

She is full of energy. She loves to move, explore, imagine, and play — but like many children, she also loves spending time in front of the TV.

As a parent, that created a question I kept coming back to: what if screen time did not have to mean sitting still?

Instead of asking children to choose between movement and entertainment, I imagined an experience where the two could become the same thing.

What if the TV could invite a child to stand up, jump over a rock, duck under a branch, raise their arms to unlock a magical door, and physically become part of the adventure?

What it does

QuestMove turns a smartphone into a motion controller for a Fire TV adventure.

The player selects an avatar and enters an adventure such as The Moving Forest. Fire TV displays a QR code that opens the QuestMove Motion Controller on the phone.

The phone camera then uses pose detection locally to understand the player’s movement. No video needs to be streamed to the TV.

During The Moving Forest, rocks roll toward the player’s character. When the child physically jumps, QuestMove detects that movement and sends a lightweight movement event to Fire TV. The avatar immediately jumps over the obstacle and the adventure continues.

The current experience includes:

Fire TV adventure gameplay Girl and boy player avatars QR-based phone controller setup Real-time body tracking using the phone camera Physical jump detection Real-time phone-to-Fire-TV interaction Rolling obstacle challenges Replay using either the Fire TV remote or body interaction A child-friendly visual experience designed for TV viewing

The longer-term vision is for QuestMove to support multiple gestures and AI-driven adventures that adapt their story and challenges to the player.

How we built it

QuestMove is built as a multi-device experience.

The Fire TV application uses React Native and is based on Amazon’s multi-TV application architecture. It handles the game experience, navigation, avatars, animations, obstacles, and player feedback.

The smartphone acts as the motion sensor. The QuestMove Motion Controller runs in Safari and uses MediaPipe Pose Landmarker to process camera frames locally on the device.

Instead of transmitting video, the controller converts pose information into small semantic events such as:

jump

Those events are sent through a WebSocket connection to a lightweight relay and then delivered to the Fire TV application.

The current development pipeline is:

iPhone Camera → MediaPipe Pose → Movement Detection → WebSocket → QuestMove Fire TV → Gameplay

Challenges we ran into

One of the biggest challenges was making movement detection feel reliable enough for an actual game.

Detecting a person with computer vision was relatively straightforward. Determining whether that person had truly jumped, without generating multiple detections or confusing other body movements with a jump, required much more experimentation.

We built a temporal movement detector using standing calibration, torso displacement, velocity, and a jump state machine. We then physically tested it on an iPhone until five real jumps consistently resulted in five detections while movements such as stretching, tiptoeing, and crouching produced no false positives.

Another challenge was connecting a secure browser based phone controller to a Fire TV development environment. The iPhone controller runs over HTTPS and secure WebSockets, while Fire TV connects to the same development relay over its existing WebSocket connection.

We also learned that correct technology is only part of the experience. Initially, the game simply asked the player to jump. We changed the design so the obstacle now rolls toward the character first, giving the child a visual reason to react.

That small change made the experience feel much more like a game.

Accomplishments that we're proud of

The accomplishment we are most proud of is getting the complete physical interaction loop working end to end:

A child jumps in the real world → the phone recognizes the jump → a movement event travels across the network → the Fire TV character jumps over an obstacle.

We also built the system so that raw camera video and pose landmarks do not need to leave the phone. Only the final movement event is transmitted.

Other milestones we are proud of include:

Reliable physical jump detection on a real iPhone One physical jump producing exactly one game action Real-time response on a physical Fire TV Stick QR-based controller setup A working child-friendly adventure scene Separate girl and boy avatars A replayable Moving Forest gameplay loop Strong separation between computer vision, transport, and gameplay Automated tests covering the movement and transport layers

What began as a technical prototype has started to feel like an actual living-room game.

What we learned

We learned that motion-controlled gameplay is not only a computer-vision problem.

The technology must understand movement accurately, but the game must also clearly communicate when and why the player should move.

For example, simply displaying “JUMP” worked technically, but having a rock visibly roll toward the character created anticipation and made the same physical action feel meaningful.

We also learned the importance of keeping real-time gameplay deterministic. Computer vision should recognize the player’s actions, while the game engine remains responsible for timing, obstacle behavior, progress, and animation.

Another important lesson was privacy by design. Processing camera frames locally and sending only semantic movement events gives us a much cleaner architecture for a family-oriented product.

Most importantly, we learned that the best interaction is often the one that disappears. The phone should feel less like another screen to manage and more like an invisible sensor that lets the child focus on the TV and the adventure.

What's next for Quest Move

The Moving Forest is only the first adventure.

The next step is to expand the physical vocabulary of the game. Rocks will require the player to jump, low branches will require them to duck, and magical objects can encourage actions such as reaching or raising their arms.

That will allow levels to evolve from a single repeated movement into sequences such as:

Jump → Duck → Jump → Reach → Duck

We also plan to add an AI-powered Adventure Director using Amazon Bedrock. Rather than using generative AI for real-time physics, the AI layer can adapt story moments, dialogue, challenge sequences, difficulty, and encouragement while the core gameplay remains deterministic and responsive.

The current locally hosted phone controller will also evolve into a hosted QuestMove experience, where Fire TV can display a QR code such as:

questmove.app/join/ABCD12

allowing a phone to securely join a specific game session without local development infrastructure.

Beyond The Moving Forest, we envision multiple worlds, adaptive challenges, family play, richer characters, and adventures where movement becomes part of the story.

QuestMove’s long-term vision is simple: turn the living room into an adventure space where the child’s body is the controller.

Built With

• codex
• react-native
• typescript