Gestura

Gloves that turn your movements and voice into game controls, so instead of learning the controller, the controller learns you.

Comment

Inspiration

We’re two high-school students, and Hack the North is actually our first hackathon, so we came into the weekend not really knowing what to expect or how hackathons even worked.

Our original idea was to build a small device that you could attach to almost any real-world object and turn that object into a game controller. A bottle could become a sword, a broom could become a staff, and basically anything around you could become part of a game.

While experimenting with that idea, we realized our hands are already the most natural controller we have.

That led us to Gestura: wearable motion-control gloves that let you control games using your actual hand movements and finger touches instead of a traditional keyboard and mouse.

What it does

Gestura turns hand movement and finger touches into game controls.

For our demo, we use Minecraft because its controls are easy to understand and show off physically.

The left glove controls movement, replacing keys like WASD, while the right glove handles camera movement and actions.

The right glove also has touch sensors on the fingers, giving us extra controls without needing physical buttons. Different finger touches can be mapped to actions such as interacting, placing blocks, switching items, or other game commands.

Instead of clicking a mouse button to break a block, you can physically swing your hand. Instead of reaching for a keyboard key, you can tap your fingers together.

We also experimented with using physical objects alongside Gestura. For example, you can hold a plastic sword while wearing the glove and physically swing it to trigger an attack inside the game. The sword itself does not need any electronics because Gestura is tracking the movement of your hand.

Our bigger goal is for Gestura to become a controller that can adapt to how different people want to play. Rather than being limited to one fixed control scheme, different gestures and finger combinations could eventually be mapped to different actions and games.

How we built it

We built two wearable controllers using ESP32-S3 microcontrollers, IMUs, and touch sensors mounted onto gloves.

  • Left glove: detects hand orientation and translates movements into keyboard inputs such as forward, backward, left, and right.
  • Right glove: uses IMU data to detect motion and rotation for camera control and physical actions such as swinging.
  • Finger touch sensors: provide additional inputs on the right glove for actions that need more precision than gesture detection alone.
  • Software: receives the sensor data, processes and filters the movement, and translates it into keyboard and mouse commands for the game.

We keep the movement and touch processing lightweight so the controls stay fast and responsive while playing.

Challenges we ran into

One of our first challenges was getting reliable motion data. Small changes in hand position can cause noisy sensor readings, so we had to experiment with calibration, dead zones, and sensitivity to make the controls feel natural.

We also had to make the touch sensors reliable. Since they are mounted directly on a glove that is constantly moving, we needed to prevent accidental touches from triggering random commands during gameplay.

We also learned very quickly that wearable electronics can be a little dangerous if you are not careful.

At one point, we accidentally shorted the positive and negative battery connections. The wire started smoking and our battery holder was basically toasted. Luckily, nothing serious happened, but it definitely got our attention.

After some emergency sketching, checking our wiring, and some slightly questionable-looking soldering, we managed to rebuild the power connection and get everything running again.

Another challenge was making sure normal hand movements did not accidentally trigger controls. Reaching for something or adjusting the glove should not cause your Minecraft character to suddenly punch something.

We also had to think carefully about latency because even a small delay between moving your hand and seeing the game respond makes the controller feel unnatural.

Being at our first hackathon was also a challenge by itself. We were learning not just how to build Gestura, but also how to manage our time, divide up work, make decisions quickly, debug under pressure, and keep moving even when something broke.

Accomplishments that we're proud of

We are proud that we were able to combine several different physical inputs into one wearable controller.

The gloves do not rely on just one sensor. We combine hand orientation, rotational movement, physical gestures, and finger touches to create different ways of interacting with a game.

Seeing a character move because you tilt your hand, watching an attack happen because you physically swing your arm, and triggering another action just by touching two fingers together was the point where Gestura started feeling like a real controller instead of just an electronics experiment.

We are also proud that we managed to recover from our hardware mistakes and keep building instead of losing half the project to one toasted battery holder.

Considering that we came into Hack the North as two high-school students at our very first hackathon, getting from an idea to a working physical controller in one weekend is something we’re really proud of.

What we learned

Gestura taught us a lot about combining hardware and software in a way that actually feels natural to a person.

We learned about:

  • ESP32 development
  • IMUs and motion sensing
  • capacitive touch inputs
  • filtering noisy sensor data
  • wireless communication
  • keyboard and mouse emulation
  • gesture detection
  • latency optimization
  • hardware calibration
  • soldering under slightly stressful circumstances

We also learned a lot about hackathons themselves: how quickly an idea can change, how important it is to get a basic version working first, and how much you can learn when you only have a short amount of time to build something.

One of the biggest things we learned was that getting data from a sensor is the easy part.

Making that data feel like a natural and responsive controller is much harder.

We had to think about how people actually move their hands, which movements should count as intentional actions, and how to prevent normal movements from accidentally affecting the game.

What's next for Gestura?

Minecraft is only our first demo.

Our original idea was to make almost any physical object become part of a game, and Gestura is our first step toward that bigger goal.

Next, we want users to be able to create and customize their own gestures and finger combinations, save different control profiles for different games, and switch between them easily.

We also want to explore using the same gloves for things beyond gaming, such as controlling robots

Built With

Share this project:

Updates

Submission history