As we have seen through our university careers, there are students who suffer from disabilities who can benefit greatly from accessing high-quality lecture notes. Many professors struggle to find note-takers for their courses which leaves these students with a great disadvantage. Our mission is to ensure that their notes increase in quality, thereby improving their learning experiences - STONKS!

What it does

This service automatically creates and updates a Google Doc with text-based notes derived from the professor's live handwritten lecture content.

How we built it

We used Google Cloud Vision, OpenCV, a camera, a Raspberry-Pi, and Google Docs APIs to build a product using Python, which is able to convert handwritten notes to text-based online notes.

At first, we used a webcam to capture an image of the handwritten notes. This image was then parsed by Google Cloud Vision API to detect various characters which were then transcripted into text-based words in a new text file. This text file was then read to collect the data and then sent to a new Google Doc which is dynamically updated as the professor continues to write their notes.

Challenges we ran into

One of the major challenges that we faced was strategically dividing tasks amongst the team members in accordance with each individuals' expertise. With time, we were able to assess each others' skills and divide work accordingly to achieve our goal.

Another challenge that we faced was that the supplies we originally requested were out of stock (Raspberry-Pi camera) however, we were able to improvise by getting a camera from a different kit.

One of the major technical challenges we had to overcome was receiving permissions for the utilization of Google Docs APIs to create and get access to a new document. This was overcome by researching, testing and debugging our code to finally get authorization for the API to create a new document using an individual's email.

Accomplishments that we are proud of

The main goal of STONKS was accomplished as we were able to create a product that will help disabled students to optimize their learning through the provision of quality notes.

What we learned

We learned how to utilize Google Cloud Vision and OpenCV which are both extremely useful and powerful computer vision systems that use machine learning.

What's next for STONKS?

The next step for STONKS is distinguishing between handwritten texts and visual representations such as drawings, charts, and schematics. Moreover, we are hoping to implement a math-based character recognition set to be able to recognize handwritten mathematical equations.

Built With

Share this project: