GameSense

• 

  GameSense Black Box CAD Render

• 

  Full setup

• 

  Headband

• 

  Setup

• 

  Breadboard

Inspiration

What it does

GameSense is a gaming headband that converts spatial audio into multi-directional, haptic feedback. In real time, players can convert the three-dimensional audio they may not be able to hear in-game into clear vibrations that indicate direction, intensity, and movement. It also includes head-tilt sensors that enable alternative player controls, allowing gamers with limited hand or arm mobility to move and interact using simple head movements, creating a more immersive and accessible gaming experience.

How we built it

GameSense captures a game’s spatial audio and processes it in real time with Python to determine the direction and intensity of sounds. These signals are mapped to vibration motors placed around a wearable headband, allowing players to feel where audio originates in the game world. A microcontroller drives the motors to produce directional haptic feedback that represents footsteps, gunfire, and other cues. We also integrated an IMU head-tilt sensor that translates head movement into in-game movement, providing an alternative control method for players with limited hand or arm mobility. Together, these systems turn sound into touch, creating a more accessible gaming experience.

Challenges we ran into

One challenge we ran into was utilizing non-intrusive methods to capture audio data without triggering in-game anti-cheat protocols. To solve this, we used a native 7.1 Surround Sound audio output combined with a Windows Audio Session API Loopback to capture audio streams at the OS level.

Accomplishments that we're proud of

One accomplishment we’re especially proud of is how seamlessly wearable we made GameSense. The headband is lightweight and comfortable, so after a few minutes players barely notice it’s there. At the same time, the vibrational audio creates an incredibly immersive experience. Achieving both all-day comfort and a memorable user experience was the key goal for us.

What we learned

Designing an accessibility tool requires forgoing the many assumptions we would normally take for granted. We learned to think critically about how people touch an interact with games, exploring new ways to turn sound into touch. Technically, we gained expertise in real-time signal processing, haptic feedback, and hardware debugging while navigating challenging constraints such as comfort and anti-cheat compatibility. Beyond the technology, we learned the value of empathetic design choices: creating an immersive and truly accessible product takes incredible amounts of iterating, testing, and adapting.

What's next for GameSense

We aim to expand GameSense's compatibility across MacOS, Linux, and major gaming consoles such as PlayStation, making it more accessible to more players. Our ultimate goal is to have a software that can convert any and all audio into understandable cues for everyone to understand. Beyond compatibility, we plan to refine the haptic experience, install customizable feedback profiles, and publish on open-source, pushing GameSense into the go-to immersive, accessible gaming platform.

Built With

• arduino
• c++
• esp32
• python
• wasapi