Sheet Music

Sheet Music logo

Inspiration

Inspired by our senior design project to build a tool for filling out paper forms for the visually impaired, we decided to tackle a key MVP using computer vision.

What it does

We focused on recognizing a pen tip, and coordinates of an A4 paper in the image plane. Then, we translated the image coordinates using homography to cartesian coordinates. Basically projecting the paper and everything else in the image to the plane of the paper. Finally, we mapped these coordinates to sounds to provide audio feedback. Each quadrant of the page elicits different music feedback!

How I built it

We collaborated remotely using Google Colab, various Python libraries including opencv, and camera image processing.

Challenges I ran into

Pen tip recognition was very difficult!

Accomplishments that I'm proud of

Working together remotely and breaking down our project into small, accomplishable parts. Also, making progress on our senior design project!

What I learned

Teamwork!

What's next for Sheet Music

We're working on generalizing to different sizes of paper forms, providing speech feedback, and testing the tool with visually impaired users for feedback.