Mice are useful, but there is a sense of satisfaction that comes from moving objects with your hands in a natural way. There are lots of awesome software programs for visualizing biochemical molecules, engineering parts, and other models. We wanted to develop a way to control these models with natural hand movements.
A TED Talk by Johnny Chung Lee gave us the inspiration for how to make this dream a reality. He created a cheap Smartboard by hacking the Wii Remote to track an IR light pen. We thought we could put the IR lights on fingertips and transform the Wii Remote hack software to track and interpret gestures.
How it works
- User wears gloves with IR LED lights attached to the fingers and makes gestures.
- Wii remote tracks LEDs and transmits information to the computer via Bluetooth.
- Data on how the LEDs move is translated into mouse commands that control 3D models, Google Earth or other applications.
Challenges I ran into
Tracking Multiple IR light sources. The Wii is capable of tracking up to four IR light sources and assigns a number to each. It was a challenge to make sure the program assigns the right importance to each one (index finger vs other fingers).
Implementing the grab gesture. The grab is a very important gesture that triggers the mouse click. It needs to be implemented precisely so that what the user is grabbing is actually what is clicked on. Otherwise, folders and icons can be dragged/dropped/DELETED in unwanted ways (Yea, Sam). We overcame this by creating two "modes" of control. The one handed mode is used to quickly move and grab which is great for Google maps. The two handed mode lets the user control the mouse position very accurately with one hand and use the other hand to trigger the grab. This is great for moving through drop-down menus and selecting files.
-Background sources of IR like the sun, people and reflective surfaces. All of these light sources interfered with detecting the location of fingers. Developing strong LED lights and calibration settings that could filter them out was a challenge.
Accomplishments that I'm proud of
- We have a working final product. This was our first hack, first successful hack, and most awesome hack, respectively.
- The hack well across a range of different software programs.
- The gestures are natural and can be customized to user preferences.
- The source code for the Wii hack was in C# which is a language none of us had ever worked in before.
- We overcame lots of unexpected difficulties with materials, software and everything (This happens with every hack, but is still something to be proud of.)
What I learned
- Simple, reliable hardware is incredibly important.
- Variable naming can be wonderfully helpful or the bane of your existence. You decide!
- Lots of programming knowledge with events, forms and interfacing with the operating system.
- Adding a right click , zoom, and scroll.
- Integration with touch operating systems.
- Using the technology to read handwriting for text searches.