Using the mask to highlight marker on whiteboard
An unmasked image
Haar Cascade to detect upper body
SmartNotes Flow Chart
What it does
This project is designed to be an easier, cheaper, and faster alternative to the modern smart board. With SmartNote, there is no need for pre-installation other than a video capture device and a whiteboard-projector setup that most classrooms have. As the professor projects the notes document (example question or equation, etc...) onto the white board and writes notes on it, the camera records the video and sends it to our program along with the notes document. The program will then place each set of whiteboard notes within the document at the proper location (As displayed on the whiteboard).
How we built it
The entire project is built in three parts: the user interface, the graphical analysis, and the text document reading. The user interface is built using python3 with tkinter. For the graphical analysis, the python library OpenCV is used to highlight the whiteboard marker and document text, while ignoring all other elements. After that, images are returned when OpenCV detects that a solution (or set of writing) has been completed on the board. From this, the text document portion is read using Microsoft's Azure API for computer vision, and then the original document is searched with this text to find where each image is to be placed. All of this document work is done using _ python-docx _, which concludes with actually writing the images and saving the final edited document.
Challenges we ran into
One of the first challenges we run into is that the whiteboard is made out of glass and the projector with fainted image, both of which affected the image quality. The refresh rate of the projector matching the refresh rate of the phone is also a technical difficulty.
Accomplishments that we're proud of
Color-masking was accurate even with a poor projector and glass whiteboard by editing the contrast and saturation of the images to get the red whiteboard marker.
What we learned
Half of the team members were new to Python and began to learn the syntax of the language while integrating it with the GUI. In OpenCV, we learned about Haar Cascades and video processing, and how to apply these concepts to the project. All team members were new to learning Microsoft's Azure API tools. One main focus was to learn not only about the functions provided by Azure through its API, but also to work with sending requests, formatting headers, and parsing through the ultimate response to gather the data important to our project.