PewPew Sherif

• 

  PewPew Sherif

Inspiration

We wanted to explore accessibility in gaming. As people who enjoy playing games, we noticed how limited the options are for players with reduced motor control. PewPew Sherif was our attempt to bridge that gap by creating a game that can be played entirely hands-free, while also letting us explore computer vision, an area we’re both interested in.

What it does

PewPew Sherif is a hands-free version of Space Invaders controlled using facial gestures. The player moves the cowboy by tilting their head left or right, and shoots by opening their mouth. No keyboard or mouse is required.

How we built it

We used TensorFlow’s pretrained FaceMesh model to detect facial landmarks in real time. From these landmarks, we applied simple geometric calculations to determine head tilt direction and mouth openness.

The game itself was built using basic HTML, CSS, and JavaScript to keep the Space Invaders logic lightweight and easy to reason about. We used React to manage the application structure, run the FaceMesh model, handle webcam input, and pass control signals to the game.

Because the hackathon timeline was short, using a pretrained model allowed us to focus on integration and interaction rather than training a model from scratch.

Challenges we ran into

Learning FaceMesh and building the Space Invaders game individually was fairly smooth thanks to clear documentation and simple game logic. The real challenge came when integrating the two systems.

Sending real-time facial control data to the game loop without introducing lag was difficult. Early versions caused noticeable slowdowns in gameplay, especially when updating movement and shooting simultaneously. Debugging timing, performance, and communication between the face tracking system and the game required a lot of iteration.

Accomplishments that we're proud of

We’re proud of successfully getting the FaceMesh model working and understanding how to extract meaningful control signals from facial landmarks. This project was our first real dive into computer vision, and seeing it drive an interactive game in real time was extremely rewarding.

We’re also proud that, despite working with unfamiliar tools and concepts, we were able to deliver a fully working demo within the hackathon deadline.

What we learned

We learned a lot about real-time computer vision, browser performance, and integrating machine learning models into interactive applications. We also gained a better understanding of how small delays or inefficient communication can significantly impact gameplay.

What's next for PewPew Sherif

Next, we want to improve performance so controls feel faster and more responsive. We’d also like to add harder levels, smoother animations, and possibly more gesture options. Longer term, we’d like to expand on the accessibility aspect by supporting additional hands-free input methods and customizable control sensitivity.

Built With

• browser
• browserapi
• css
• facemesh
• javascript
• react
• webapp