Laptop Ninja

Inspiration

Pose estimation makes kinect-like gameplay possible without requiring expensive sensors. We wanted to make a project to showcase the potential of pose estimation for games.

What it does

Laptop Ninja allows you to ruthlessly attack cut fruit using your arms. Nothing more than a webcam is required.

How we built it

We used posenet-python for the backend (and, earlier, tensorflow.js posenet) to estimate pose and p5js for the game itself.

Challenges we ran into

The largest challenge was balancing latency with accuracy. At first, we wanted to keep Laptop Ninja easily accessible and with only a static backend, which meant it had to run on regular laptops with slow CPUs. One challenge with this, though, was that due to javascript's threading model, tensorflow.js would block the main thread causing the game to lag. Tensorflow.js also wouldn't play well with web workers. We ended up moving the pose estimation to a server on Google Compute Engine to reduce latency.

Accomplishments that we're proud of

Our biggest accomplishment was integrating the pose estimation into the game, which turned out to be much more complex than previously anticipated, due to javascript's threading model. Having the ability to play the game without a controller was a significant moment of pride for us.

What we learned

We learned the potential of AI, and how this method of detecting hand motion for video games can also be applied to games in Wii sports. Given enough time and improvement to machine learning algorithms, this could be an alternative to controllers, which would make interactive games more accessible.

What's next for Laptop Ninja

Given more time, we would set up a complete Fruit Ninja style game, including a 60 second timer and a well-designed menu. Given more time, we could also implement denoising algorithms for the pose estimation to make the game smoother and a udp-based connection to reduce latency further.

Built With

• flask
• gce
• p5js
• posenet
• tensorflow.js