Let's Hold it!

T-Pose
Y-Pose

Inspiration

We were inspired by the classic “Hole in the Wall” TV game show, where contestants had to twist and shape their bodies to fit through moving walls. We wanted to recreate that same fun experience using AI and computer vision.

What it does

Let's Hold it turns your body into the controller. Using your webcam and AI-powered pose detection, the game gives you a pose to match. Your goal is simple: hold the pose perfectly until the timer runs out. If you break form or fail the pose three times, the run ends. Survive as many poses as you can and climb the leaderboard to see who holds out the longest.

How we built it

We built a FastAPI backend using MediaPipe Pose for real-time body tracking, and a Next.js frontend that streams webcam frames over WebSockets. Firebase handles authentication, lives, and leaderboard data. The backend processes each frame, sends pose landmarks, and the frontend calculates how well the player matches the target shape in real time.

Challenges we ran into

Building accurate real-time pose detection in the browser was harder than expected. MediaPipe behaves differently depending on lighting, angle, and frame rate, so getting consistent landmarks required a lot of smoothing, mirroring fixes, and custom logic. Creating a game loop that reacts instantly to body movement without false positives or jitter was also challenging. The game loop and coming up with something that makes sense to the user was something we pondered about and did take some time to brainstorm and eventually came up with what we currently have.

Accomplishments that we're proud of

We built a full, real-time, motion-controlled game using only a webcam, FastAPI, WebSockets, and Next.js, with no special hardware required. We’re proud that the game actually works smoothly, tracks multiple poses, speeds up over time, and maintains a global leaderboard with Firebase. Seeing the pose detection sync with the animations was a huge moment for us.

What we learned

We learned how hard it is to make AI-driven interactions feel natural. Pose estimation looks simple from the outside, but mapping landmarks into meaningful actions takes careful thresholds, geometry, and smoothing. We also learned a ton about WebSocket performance, game state timing, Firebase transactions, and how to build a full stack project that responds in real time. Claude is good at prototyping things out getting a sense of how the product will look like and then ultimately ending up with a clearer vision on what we wanted to get done. With hackathons in general, there is always the cost trade off where we are limited by time so there is really only a couple of things that are good features and kinda work, besides that its more a less pray to get a MVP and just ship ship ship.

What's next for Hole in the Wall

A mobile exercise mode. Coders sit all day, and this project showed us how fun it is to turn movement into gameplay. We want to add quick 30-second challenges, streaks, and a fitness-style mode where you can play on your phone and get a little exercise during breaks. We also want to tune the poses further, add custom levels, and support multiplayer rooms. After enough engagement taking this into the realm of entertainment trying to get streamers and youtubers to participate and compete against each other ultimately turns into something everyone can enjoy.

As a group we may not continue the idea but the thing we get most out of building this game is simply creativity where as we could have forced ourselves to use company sponsors tech to try and integrate it but as a group we rather be more creativity and do things with full creative freedom and turning ideas into reality as ideas focused on companies often tend to overlap with other people but as a group we had learnt to be creative and free.