What it does
GhostMouse is a flexible-use Bluetooth “touchpad” that transforms any flat surface into a wireless peripheral for a user’s computer or Android device. Once the device is paired with GhostMouse, users will obtain the ability to move their cursor on-screen through finger swipes and can trigger a mouse click by tapping anywhere on the surface. Additionally, the acceleration of the on-screen cursor corresponds to the velocity of the user gestures, providing a range of mouse behavior from slow, precise responses to faster, broad motions.
How we built it
GhostMouse achieves its functionality by projecting a plane of infrared light onto the surface and uses a infrared camera sensor to detect user fingertip positions. As the user interacts with the surface, we collect data on the displacement in fingertip positions and process it into real-time cursor responses seen on the paired PC/Android device. A multitude of different filtering approaches are used to remove signal jitter, as well as signal artifacts caused by errant user behavior. The Bluetooth HID mouse protocols are handled by an Bluefruit EZ-Key HID controller.
Challenges we ran into
Our biggest challenges while building Ghost Mouse can be separated into 2 main categories: data collection and data processing. Physically collecting data on the position of user fingertips was one of the largest obstacles we faced during the developmental process. Our product utilizes an infrared camera from Nintendo's Wiimote to track infrared hotspots; our initial attempts to use the camera to track fingertips illuminated with a line laser failed due to a lack of availability of appropriate 940nm line lasers (the sensor we used is primarily sensitive to 940nm wavelengths). Our testing with 980nm and 808nm lasers showed that our sensor was not sensitive enough to those wavelengths to reliably detect fingertips. We had to pivot towards the use of infrared LEDs, which, while significantly less collimated, were easy to acquire in 940nm format. With the use of LEDs to illuminate our fingertips, we found that the camera could accurately isolate user finger positions.
Once we had this data, we faced another significant obstacle in the process of processing the raw sensor data into a desirable user response. A significant amount of filtering and processing was implemented to reduce the drift, jitter, and jumpiness of the raw camera data. The most challenging signal artifact we had to address were the "spikes" caused by Z-Axis gestures. Z-Axis gestures are cases where the user lifts/places their finger on the interactive surface; due to the angled orientation of our camera sensor, Z-Axis gestures cause significant and unintentional "spikes" in the position data read by our camera. Our signal processing algorithm attempts to recognize intentional user Z-Axis gestures and actively removes unintentional mouse gestures that they would otherwise cause.
What's next for Ghost Mouse
The use of an array of infrared LEDs as the illuminator for our device is highly space and power inefficient; we would like to source a high powered 940nm laser module with the appropriate optics. This would eliminate the need for the large "lightbar" module that we have, and, assuming a <1W laser (which would be pushing the boundaries of optical safety), cut our power demands tenfold at least.
On the software side, we would like to implement multi finger gesture recognition, to allow for functionality such as 2 finger scroll.