Starting Screen
Starting Cutscene: Walking with Claude
Starting Cutscene: Claude is taken
Starting Cutscene: Claude is imprisoned
Humpty Dumpty's Level
Alice in Wonderland Level
Cinderella's Tower Level
King Triton's Level
The Little Red Riding Hood Level
Xen's Tower Level
Ending Cutscene: After finding all 6 objects
Ending Cutscene: Rescuing Claude
Ending Cutscene: Happily ever after with Claude
Ending Cutscene: The End
Ending screen

Inspiration

In 2026, lots of games can be played from anywhere, with close friends playing from thousands of miles apart. But reminiscing on childhood memories of games like Wii Sports and Xbox Kinect, we wanted to create an interactive game designed to facilitate in-person play and a deeper sense of connection. Our game's setting was inspired the fairy tale theme of the hackathon, especially the amazing opening video in which we witnessed Xen collecting various fairy-tale items to save HackTX. In our game, Jon (inspired by our teammate, Jonathan) is in the same situation, and it's each player's skill and quick-thinking that determines whether he reaches the same fairy-tale ending.

What it does

Jon and the Beanstalk is a two-player endless runner in the browser, played by two people standing in front of one webcam whose image is split down the middle. The person on the left controls Jon, swiping or physically moving their hand left or right to change lanes and up to jump, while the person on the right plays the Giant, aiming with their hand and closing a fist to drop an obstacle into a lane. As Jon runs, he collects items through a short sequence of story stages, and the Giant can throw walls, hurdles, oil, mud, and fireballs. If the camera isn’t available, the same game can be played with the keyboard.

How we built it

The whole thing is a static web app, built in TypeScript with Vite, Phaser for the game, and MediaPipe for hand tracking. Muse Image was used to generate almost every asset on screen, including the different player sprites for Jon, the obstacles, and the beanstalks. Aseprite was used to animate each image and turn it into something usable. Gemini API was used to make the AI controlling the other player when in single-player mode. We divided the work among vision, the runner, the Giant, and the UI, ran tracking in Web Workers so the frame rate wouldn’t collapse when the models were busy, and generated the sound in the browser instead of shipping audio files.

Challenges we ran into

Hand tracking is messy in ways that don’t show up until two people are actually standing there. Fast movement and hardware limitations can make the tracker lose a hand for a frame, or alternatively, closing a fist may fail to release an obstacle in the lane intended, so we spent a lot of time deciding when to trust a reading and when to ignore it. Gestures also landed later than keyboard input, which meant early versions felt fair on keyboard and unfair on camera. We had to slow the attacks down enough for someone to still dodge. Four people committing into the same repo at once made that harder, because a change in one area could cause merge conflicts or even silently break another unless we kept the shared interfaces small and stable.

Accomplishments that we're proud of

We got two people controlling a full game off one webcam, with a keyboard fallback when the camera fails, plus a short tutorial and a complete run with a story attached.

What we learned

The model was not the main problem. Most of the work was filtering bad tracking, generating and animating assets, keeping the two players from stealing each other’s half of the camera, and balancing the game around the delay between a real movement and the game noticing it. A lot of the work goes into what you wouldn't expect: the fine details, not core gameplay mechanics, that make the game cohesive and engaging.

What's next for Jon and the Beanstalk

We want to play it more at a normal standing distance from a laptop, then tighten calibration for different heights and lighting and add a few more Giant attacks once the existing ones feel reliable.