Inspired by our senior design project to build a tool for filling out paper forms for the visually impaired, we decided to tackle a key MVP using computer vision.
What it does
We focused on recognizing a pen tip, and coordinates of an A4 paper in the image plane. Then, we translated the image coordinates using homography to cartesian coordinates. Basically projecting the paper and everything else in the image to the plane of the paper. Finally, we mapped these coordinates to sounds to provide audio feedback. Each quadrant of the page elicits different music feedback!
How I built it
We collaborated remotely using Google Colab, various Python libraries including opencv, and camera image processing.
Challenges I ran into
Pen tip recognition was very difficult!
Accomplishments that I'm proud of
Working together remotely and breaking down our project into small, accomplishable parts. Also, making progress on our senior design project!
What I learned
What's next for Sheet Music
We're working on generalizing to different sizes of paper forms, providing speech feedback, and testing the tool with visually impaired users for feedback.