windowed hell

Inspiration

We were inspired by Windowkill by torcado, especially its unique mechanic where the game window itself becomes part of the gameplay. The idea of your playable area constantly shrinking and having to shoot the edges of the screen to create more space felt chaotic, creative, and unlike anything we'd played before.

We wanted to build our own take on that concept while adding a completely different control scheme. Instead of using a keyboard and mouse, we challenged ourselves to control the entire game using hand gestures powered by computer vision + add a multiplayer option.

What it does

Windowed Hell is a fast-paced bullet hell game running natively on the QNX Real-Time Operating System.

As waves of geometric enemies close in, the playable area continuously shrinks from every direction. To survive, players must dodge incoming attacks, eliminate enemies, and even shoot the borders of the game window to push them back and reclaim space.

Rather than traditional controls, the game is played entirely through hand tracking. Your left hand controls movement, your right hand determines your aiming direction, and making a fist fires your weapon. The result is an intuitive, touch-free control system that makes the gameplay feel surprisingly immersive.

How we built it

A Windows laptop handles all of the computer vision using Python, OpenCV, and Google's MediaPipe. During a short calibration phase, it learns the player's neutral hand position and continuously tracks both hands to calculate movement, aiming direction, and firing gestures.

These inputs are packed into a compact binary packet and streamed over UDP to a Raspberry Pi running QNX. On the Pi, a dedicated networking thread receives the data and updates the game state while the main game loop continues rendering smoothly.

This separation allowed us to run computationally intensive AI vision without affecting the deterministic performance of the QNX game engine.

multiplayer works via 2 pis running udp.

Challenges we ran into

Our biggest challenge was getting the computer vision stack to run directly on QNX. We spent a large part of the hackathon trying to cross-compile MediaPipe and TensorFlow Lite for the Raspberry Pi, running into dependency issues, compiler errors, and platform-specific incompatibilities.

We eventually realized we were spending more time fighting the toolchain than building the game. Switching to a host-target architecture let us keep the game running natively on QNX while offloading the AI processing to a laptop.

Another challenge was input reliability. Our initial OpenCV skin-color detection worked well in testing but quickly became unreliable under changing hackathon lighting. Replacing it with MediaPipe's hand tracking gave us much more stable and accurate controls.

Accomplishments that we're proud of

We're proud that we successfully combined real-time embedded systems with AI-powered computer vision without introducing noticeable input delay.

Moving from text-based communication to compact binary packets made the controls fast and responsive, and seeing the player's movements mirrored instantly on the QNX-powered game for the first time was incredibly rewarding.

Most importantly, we built a game that feels genuinely different: turning hand gestures into an intuitive way to survive an increasingly chaotic bullet hell.

What we learned

This project taught us an important lesson about engineering under tight deadlines: sometimes the best solution is to rethink the architecture instead of forcing a difficult implementation.

Along the way, we learned a great deal about POSIX networking, binary data communication between different platforms, QNX scheduling, and integrating MediaPipe's hand landmarks into a responsive game control system. We also gained firsthand experience balancing AI workloads with the requirements of a real-time operating system.

What's next for windowed hell

Our long-term goal is to eliminate the host laptop entirely and run the full computer vision pipeline directly on QNX. Once the required toolchain support is available, we plan to integrate MediaPipe with the QNX Camera Framework so the Raspberry Pi can handle both vision and gameplay on its own.

We're also planning to expand the gameplay, additional enemy types, upgrades, and new mechanics that make the shrinking play area even more dynamic.