Head and Voice Controlled Mouse

Inspiration

How can technology enhance social well-being? We focus on bridging the gap between interactive devices and people with disabilities. For individuals who have lost the use of their hands, we propose a wearable system based on inertial sensing and voice detection, enabling cursor control through head movements and mouse functions—such as clicking, scrolling, and calibration—All via voice commands, creating a more natural and efficient human-computer interaction.

What it does

The sensors wirelessly transmit horizontal and vertical inertial displacement data to a computer application, where it is processed and mapped to cursor movement speed. Meanwhile, the application continuously monitors ambient audio, recognizes and executes valid voice commands, and filters out non-command speech as noise.

How we built it

The system uses the MPU6050 as its core sensor to capture raw data from inertial displacement–induced electrical signal changes. An ESP8266 microcontroller receives this data and transmits it wirelessly to a computer. A Python-based application processes the data to control cursor movement. In parallel, an audio thread captures sound via a microphone and uses the speech_recognition library to convert speech into text, executing predefined commands accordingly. Finally, a graphical user interface (GUI) was developed to monitor device status, data acquisition, and operation modes in real time.

Challenges we ran into

We encountered challenges in both hardware and software design. Initially, we considered using a magnetic sensor to map rotational angle or speed to cursor movement. However, since this approach relied on numerical integration, it led to significant accumulated errors, causing cursor drift. In addition, the sensor was susceptible to interference and occasionally failed to produce stable readings. Ultimately, we chose a more stable and straightforward inertial sensing solution.

The subsequent challenge was tuning the dead zone and sensitivity parameters. Through repeated testing, we optimized these coefficients to achieve a balance where cursor movement is neither too fast to control nor too sluggish, resulting in smooth and stable performance.

The next challenge was implementing mouse click functionality. We proposed two approaches. The first was to use rotation direction and speed along an additional axis to trigger left and right clicks. However, head rotation inevitably introduced multi-directional disturbances, causing cursor drift and making precise selection difficult.

Ultimately, we decoupled the clicking module from inertial control and adopted a second approach—voice-based control—thereby improving system stability and usability.

Accomplishments that we're proud of

We successfully implemented both voice detection and inertial control, enabling stable hands-free cursor movement and command execution. After building the prototype, we further optimized the hardware structure, significantly reducing the size of the components and improving wearability and overall system integration.

In addition, we iteratively refined signal processing parameters to enhance system stability. We also upgraded the communication between the sensors and the computer application from initial wired testing to Wi-Fi-based wireless transmission, resulting in smoother and more reliable performance. Ultimately, the system achieved significant improvements in stability, portability, and usability, forming a more complete assistive interaction solution.

What we learned

We developed a solid understanding of key skills across hardware design and software implementation, including sensor selection and integration, raw data acquisition and processing, and the modeling and mapping of inertial signals. On the software side, we performed data processing and parameter tuning (e.g., sensitivity and deadzone range), improving system stability and reliability.

We also gained hands-on experience in inter-device communication, including network configuration, Wi-Fi-based data transmission, and real-time data reception and parsing between the sensors and the computer application. This strengthened our understanding of embedded systems and cross hardware–software integration.

Finally, we reflected on human–AI collaboration. As a team of freshmans and sophomores with limited experience, we relied heavily on AI-generated support during development. We realized that engineering is less about mastering every low-level detail and more about integrating existing technologies creatively into a coherent system. In this process, idea generation and resource coordination became crucial, while AI provided cross-domain technical support and humans focused on integration, interpretation, and system design.

What's next for Head and Voice Controlled Mouse

We will further improve the head- and voice-controlled mouse system. First, we plan to study standard mouse drivers and event-handling mechanisms to optimize inter-thread communication, reducing latency and jitter for smoother, near “zero-delay” cursor movement.

Second, we will enhance voice control by improving speech recognition accuracy and real-time performance, ensuring stable command execution in different environments.

Finally, we aim to explore a new mapping method that directly converts inertial tilt angles into screen coordinates instead of velocity-based control. This requires studying the relationship between screen coordinates and head orientation. If successful, the cursor will stay fixed at the point corresponding to head direction, rather than drifting like a gyroscope-based system, improving stability and precision.

Built With

• c++
• esp8266
• google-web-speech-api
• mpu6050
• python