StarGazer

Title Screen
Game Play Screen
Player Collision
GazeTracker in Action!

Inspiration

As computer science students focused on Human-Computer Interaction (HCI) and Computer Vision (CV), we wanted to combine our skill sets and interests to create a fun, unique, arcade-style experience. Moreover, as gaming enthusiasts, we wanted to create an accessible experience.

What it does

StarGaze is a 2D arcade-style game that is controlled only by the player's eyes. The goal is to fly through an endless asteroid field without crashing. Your vision, your voyage! An Ocular Odyssey!

How we built it

We decided to build StarGazer with a Python Backend which has the Gaze estimation algorithm implementations. We decided to build off of an existing open-source project called GazeTracking. GazeTracking leverages OpenCV to track eye position as well as estimating where the eyes are actually looking on the screen. We added our own package of capabilities on top to improve the functionality and usability of the gaze estimation algorithms.

Our Python backend passes gaze estimation coordinates to Unity via python socket. These coordinates serve as commands for our spaceship in-game. Unfortunately, the raw feed of real-time gaze estimation coordinates was quite noisy, thus making the movement of the spaceship clunky and unreliable. To combat this, we implemented a Kalman filter that takes the raw coordinate data fed through the socket and reduce the noise to produce more stable and responsive control.

The Unity game engine houses all of our game objects, environment, and logic written in C#. Beyond implementation of a gaze-controlled spaceship, we also designed the game to have varying levels of difficulty. The number of asteroids flying in per second, and their speed can be adjusted depending on skill level and desired challenge. The game also resets itself automatically after the player crashes, so it truly is a hands-off experience!

Challenges we ran into

Tuning and refinement of eye and gaze-tracking was an initial challenge. We wanted to make sure that the user did not struggle controlling the spaceship. The jittery nature of the gaze tracker's output made the spaceship move quite erratically. Having worked with Kalman Filters before, we found that this was a good solution that made control of the spaceship smoother and easier.

Another challenge that took us a considerable amount of time was piping the data through in a way that is usable by Unity and works in parallel with Python.