We really enjoyed how our professor for embedded systems class uploaded pictures of his whiteboard notes and drawings to his website, so we thought why not make it even easier to do that? The images aren't really searchable, and it's definitely a bit of a manual process to upload all the files from your phone onto some older website and then embed the links in your page. That's where the idea for Noteify was born.
What it does
Using a raspberry pi, the user can press a button and the connected camera will take a picture of handwritten notes on a whiteboard or paper. The picture is then uploaded to google cloud, where it is then fetched periodically by the backend to process and enhance the image. Next, the handwritten text is extracted from the image using Google Vision. Finally, the notes are tagged and searchable from the frontend. Noteify transforms whiteboard scribbles into indexed and searchable online notes in seconds!
How we built it
We used React and Semantic-UI on the frontend, while on the backend we used flask and mongodb to hold the tags of the images. This was running on an amazon e2c instance, and we also used a google cloud bucket to hold the images sent from raspberry pi. The website was deployed with Google App Engine. The image processing was performed with the Pillow library in Python. Noteify utilizes Natural Language Toolkit and Google Vision to extract text and keywords from the image.
Challenges we ran into
The most difficult part was getting the backend database working. Surprisingly, the raspberry pi portion, the image processing and the text extraction all came pretty quickly. A few hours were spent debugging the backend, and after that the last hurdle was connecting it with the website's frontend. The team persevered and brought Noteify to a fully operational product!
Accomplishments that we're proud of
Everything works from the button click on the raspberry pi all the way to the website's search and display functionality! From the start, the whole team was energized by the project vision. Noteify serves as a strong proof of concept and has a lot of room for expansion and growth. We're extremely proud to have developed such a polished and effective prototype - the whole team is wishing Noteify was deployed in their courses!
What we learned
Every member of the team gained valuable technical knowledge. We now have experience working both the image and natural language processing pipelines. Additionally, our team acquired a deeper understanding of MongoDB and ReactJS. More importantly, the team learned how to prototype and accelerate the software development process while building and deploying Noteify!