Due to the recent emergence of the Internet of Things being incorporated into our households and businesses, we've noticed a distinct lack of different options for people to interact with their appliances. There seems to be this influx of sound recognition technology prevalent today with new technologies such as SIRI and Amazon Echo; however, we thought, what about using visual technology as a means to interact with our appliances? We feel as though the current market doesn't properly represent those who cannot use speech as a means to interact with technology - this could be due to issues such as stuttering, being mute, or having a unique unrecognizable accent. The technology we made could cover the holes left behind on speech recognition.

What it does

Our app tracks hand gestures unique to each individual and displays the amount of fingers one is holding up. With this technology, this information is relayed to an Arduino Uno to replicate an interaction between household objects (LEDs representing lights for example) and user. Depending on the amount of fingers one holds, a different interaction occurs on the Arduino Uno.


  1. User covers hand over 9 distinct points on camera.
  2. Program is run immediately determining number of fingers one is holding up.
  3. Data is relayed to Arduino Uno and LEDs light up according to the amount of fingers held up.

How we built it

We initially explored the capabilities of face tracking technology before and thought that we could focus on another area of human recognition.

Using the OpenCV open-source library, we created an overlay to show how the hand is recognized through the camera.

Next, we needed to find the distinct points in which the identified overlay has "arches" and with this we count the number of fingers held up.

Then we wired together an Arduino Uno to simply glow corresponding to the amount of fingers held.

This was accomplished with Serial ports to the Arduino with data being transferred from computer application to output on the Arduino Uno

The list of technicalities:

Support of Mac, Windows and Linux. Utilization of C++ language and embedded C. Utilization of public libraries such as OpenCV and boost. Utilization of Arduino to demonstrate the output of household appliances. Added support for the Intel Edison.

Share this project: