It has been increasingly evident that paper is becoming more and more obsolete in today's society. PaperVision is an innovation that will launch paper into the 21st century.
What it does
PaperVision will inspire the people to draw their ideas on paper and see them on the computer. It creates a freely interactive experience with ideas that are put down on the paper. Two examples that we tackled are mazes and graphing tools. Anyone can draw a maze on a piece of paper so long as it has a start and a finish and end up playing through it on a computer with PaperVision. In addition, PaperVision can take a hand drawn graph and output to the user the appropriate equation. Another work in progress of PaperVision, is the gaming aspect, such as a version of Space Invaders. Enemies are hand drawn and fought in real time.
How we built it
Using OpenCV embedded in Python, we were able to extract information from the hand drawn images. Specifically, we used the techniques of erosion and dilation to remove noise such as faint shadows and glares that could disrupt the clarity and the usability of an image. This allowed us to distinguish between ink and paper: black and white. OpenCV has a function that deals with contours that enabled us to recognize shapes, namely lines, such as the edges of a maze. We also other integrated libraries such as turtle graphics to complete the experience. In the graphing tool, we sampled several points from the hand drawn image and used linear algebra techniques including matrix row reduction and regression to determine an equation that matches the graph.
Challenges we ran into
Some of the challenges we faced included black ink detection and distinguishing between colors. Also, Computer Vision did not cooperate at first, but ended up working with fine tuning and patience on our part. When dealing with the maze proportion of the project, it was necessary for us to identify black ink as boundaries. However, due to the fact that the black ink was taken from something drawn on paper, the pixels did not correspond. To solve this, we needed create a linear mapping between the set of pixels from the drawn image and the game image. In order to distinguish between colors, namely black and white, we discovered that erosion and dilation did not remove all the noise created from various factors including light intensity. We accounted for this by having a threshold to categorize different pixels.
Accomplishments that we're proud of
Although we faced many challenges along the way, we also had accomplishments that we were proud of. Most importantly, we were able to a finish working project in the time alloted with our first idea. To add, we applied the math such as linear algebra that we learned in class to an education graphing tool that could be used.
What we learned
We learned that Computer Vision is a powerful tool that can connect both the digital and physical worlds in a cyber-physical future. Another lesson that we learned from Computer Vision is that not everything, especially in the real world is as precise and perfect as we expect it to be. Quite literally, not everything is black and white. It is important to adjust and fine tune the data we are given to work better with us.
What's next for PaperVision
PaperVision has the potential to revolutionize the usage of Computer Vision. It can be used for entertainment, education tools, and games such as space invaders, if not more.