OpenCV of notes
Our motivation was a grand piano that has sat in our project lab at SFU for the past 2 years. The piano was Richard Kwok's grandfathers friend and was being converted into a piano scroll playing piano. We had an excessive amount of piano scrolls that were acting as door stops and we wanted to hear these songs from the early 20th century. We decided to pursue a method to digitally convert the piano scrolls into a digital copy of the song.
The system scrolls through the entire piano scroll and uses openCV to convert the scroll markings to individual notes. The array of notes are converted in near real time to an MIDI file that can be played once complete.
The scrolling through the piano scroll utilized a DC motor control by arduino via an H-Bridge that was wrapped around a Microsoft water bottle. While the notes were recorded using openCV via a Raspberry Pi 3, which was programmed in python. The result was a matrix representing each frame of notes from the Raspberry Pi camera. This array was exported to an MIDI file that could then be played.
Challenges we ran into
The openCV required a calibration method to assure accurate image recognition. The external environment lighting conditions added extra complexity in the image recognition process. The lack of musical background in the members and the necessity to decrypt the piano scroll for the appropriate note keys was an additional challenge. The image recognition of the notes had to be dynamic for different orientations due to variable camera positions.
Accomplishments that we're proud of
The device works and plays back the digitized music. The design process was very fluid with minimal set backs. The back-end processes were very well-designed with minimal fluids. Richard won best use of a sponsor technology in a technical pickup line.
What we learned
We learned how piano scrolls where designed and how they were written based off desired tempo of the musician. Beginner musical knowledge relating to notes, keys and pitches. We learned about using OpenCV for image processing, and honed our Python skills while scripting the controller for our hack.
As we chose to do a hardware hack, we also learned about the applied use of circuit design, h-bridges (L293D chip), power management, autoCAD tools and rapid prototyping, friction reduction through bearings, and the importance of sheave alignment in belt-drive-like systems. We also were exposed to a variety of sensors for encoding, including laser emitters, infrared pickups, and light sensors, as well as PWM and GPIO control via an embedded system.
The environment allowed us to network with and get lots of feedback from sponsors - many were interested to hear about our piano project and wanted to weigh in with advice.
What's next for Piano Men
Live playback of the system