Gestura

Gesture-based mouse built using FreeWILi and Vosk text-to-speech used to increase accessibility in using the web for disabled people, such as amputees and people with lost-mobility in their hands.

Comment

👋 Gestura: Gesture-Based Mouse & Voice Control

💡 Inspiration

At the FreeWILi workshop held during MHacks this year, our team was inspired by a demo of a Theremin-like device. It used hand gestures—detected via the FreeWILi’s accelerometer—to control pitch and volume. This sparked an idea: what if we could harness similar gestures for accessibility?

One of our teammates has a family member who suffers from severe Rheumatoid Arthritis, which makes traditional mouse and keyboard usage extremely difficult. We envisioned a system that would allow users to control a computer using just hand gestures and voice commands—no physical touch required.

Moreover, Gestura, which can be attached to the wrist or ankle, has a broader use for amputees and other disabilities to allow people to regain the joy of digital interaction once again.

🧠 What It Does

Gestura enables users to control their mouse using hand gestures and perform mouse actions via voice commands.

  • Hand gestures detected by the FreeWILi move the mouse across the screen.
  • Voice commands like:
    • “left click”
    • “right click”
    • “double click”
    • “scroll up”
    • “scroll down”
    • “hold”
    • “release”
    • “pause input”
    • “resume input”

This provides a touch-free, accessible experience for users with limited mobility.

🛠️ How We Built It

We used Python and developed collaboratively with Git and GitHub.

📦 Libraries & Tools Used

  • freewili: Interface with the FreeWILi accelerometer.
  • pynput: Control the mouse (movement and clicks).
  • sounddevice: Capture audio input via microphone.
  • vosk: Offline voice recognition using vosk-model-small.

🎮 Mouse Movement via Accelerometer

  • Captured X and Y acceleration using the freewili Python library.
  • Applied double integration to convert acceleration to position.
  • Used pynput.mouse.Controller.position to update the mouse location.

🎙 Voice Control

  • Captured real-time audio using the computer’s microphone.
  • Processed input through Vosk for speech recognition.
  • Mapped voice inputs to actions like:
    • click
    • right click
    • scroll up/down
    • pause/resume input, etc.

🚧 Challenges We Ran Into

  • Sensor Drift & Offset: The raw accelerometer data wasn’t zero at rest, causing significant drift after integration.

    • Fix: Introduced a deadband (e.g., \(a \in [-2000, 2000]\)) to treat near-zero acceleration as zero.

  • Erratic Movements: Large spikes in acceleration led to uncontrollable motion.

    • Fix: Added min/max bounds for both velocity and acceleration to stabilize gestures.

  • Screen Bounds: The mouse pointer would sometimes go off-screen.

    • Fix: Implemented position clamping to ensure the mouse stays within the screen (1920x1080).

🏆 Accomplishments We're Proud Of

  • Natural-feeling gesture control after some practice.
  • User feedback: A friend testing it smiled when the device worked exactly as intended.
  • Implemented a broad command set, from clicking to scrolling and pausing/resuming gesture input.

📚 What We Learned

  • The FreeWILi device is powerful and versatile for prototyping assistive tech.
  • Sensor data in the real world can be noisy—handling that is both science and art.
  • Combining gesture + voice control can dramatically improve accessibility.

🚀 What’s Next for Gestura

  • 🛠️ Build a gesture calibration tool to personalize the experience.
  • 🧠 Add machine learning to adapt gesture detection to each user over time.
  • 🌐 Integrate with web browsers for scroll, tab switch, and navigation.
  • 🎤 Use the FreeWILi’s microphone for full-device integration (once streaming support is added).

Built With

Share this project:

Updates