pianoEd

Our setup
Our ESP8266 and circuit

Inspiration

As a team of musicians, we wanted to share our love of music with others by creating a way to teach people who are traditionally unable to learn music due to special circumstances. This was our inspiration to create pianoEd, a program to teach visually impaired people how to play piano.

What it does

Given some input music, this system uses a camera to detect whether any of a player’s fingers are over the right note, and it sends a vibration to the corresponding finger that is above the correct note.

How we built it

We used OpenCV on a Logitech camera to locate positions of fingertips through contour detection. If any of the fingertips’ locations matched the location of the required note, a request was sent to the Arduino to cause the respective motor on that finger to vibrate.

Challenges we ran into

Since our project has both a hardware and software component, we ran into some challenges in creating and integrating the 2 components together.

For hardware: *We had to build a circuit differently from our origin plan due to a lack of critical circuit components *We had to use alternative diodes due to the lack of proper diodes *Power consumption was an issue so we had to scale down to 1 motor for haptic feedback instead of 5 (for each finger) *We did not have access to a 3D printer, so we had to improvise a camera stand

For software: *We had difficulty setting up a server/client to send data from our OpenCV program to the ESP8266, so we adapted by using serial communication *We had originally used a pretrained deep convolutional neural network to predict the locations of the fingertips. However, the computations for the predictions were too expensive for each frame, so we switched to using OpenCV’s contour detection.

Accomplishments that we're proud of

Despite all the challenges we faced in building this project, we never gave up on our idea, persisted as a team, and gave our all to try to make it work. We had to improvise many alternative solutions to the designs and plans we originally had, and we each slept a total of less than 6 hours this weekend working on this, but I am incredibly proud of my team for our joint effort and dedication to this project.

What we learned

*How to work with advanced circuit components *Some of us had the opportunity to work with hardware for the first time (and learned to solder) *How computer vision works *How to use OpenCV *How to set up an ESP8266 *How to work as a team

What's next for pianoEd

*Wireless communication between ESP8266 and the OpenCV program *Expansion to 5 fingers for haptic feedback instead of just 1 *Potential use in teaching people how to type