Since the beginning of the hackathon, we’ve wanted to learn more about Python, machine learning, how to use Microsoft Azure, and how to implement various algorithms. We found that, while there were many ways for students to take and look at notes, it was sometimes tedious to find resources for the topics covered in lectures. Thus, BigRedNotes was created in order for students to upload their notes onto a web app and receive a text from Twilio, which would contain supplemental link(s) that could help with understanding the material more.
What it does
Students are to upload their notes onto the web app by taking a picture with their phone. Along with the notes, the student user must input his/her phone number. Using a PHP script, we document the HTML input (the phone picture, and a text file composed of the phone number). Then, the PHP script calls a Python script which takes the phone picture, parses it through Azure’s Computer Vision API to generate a JSON file, turns the JSON file into a long string, and then strips the string in order to get just the text. This Python script then uses “NLTK,” an implementation of a natural language processing (NLP) algorithm to generate the most descriptive keyword for the notes in question. We then use the Keras neural network to map these keywords to valid YouTube links (preventing “circles,” for instance, from returning YouTube music videos of songs named “Circles”), used Flask to capture the relevant links through the YouTube API and then used the Twilio API to text these links to the phone number (previously stored in the text file).
Tl;dr: you can scan handwritten notes, and you will get a mobile text containing links to useful YouTube videos to aid you in your learning process.
How we built it
For the project, we primarily used Python, PHP, HTML, and CSS as programming languages, the Azure CV API in order to parse handwritten notes into plain text, NLTK (a natural language processing algorithm) to parse the stripped plain text into keywords, Keras to convert the keywords into valid YouTube links, and Flask and the YouTube and Twilio APIs to send the corresponding text to the given number by the student user. Though it seems relatively simple (put in a photo and your number, get a text with a video), it had many moving parts which were all individually extremely complex.
Challenges we ran into
At the beginning of the hackathon, we had the intent to create a financial app that would help students manage their finances. However, after three hours of planning, we realized how overly complicated the entire project was. We then tried to attempt making a math integral calculator, but discovered that our project already existed with companies such as Wolfram Alpha and MathPix. In order to be more creative, we decided to adopt a new project: a note-upload service that could allow students to contribute notes that they take from textbooks and lectures. After twelve hours, we ran into problems with making web apps and coding in PHP, and changed our project to something more understandable by us. Thankfully, this turned out well.
One of our biggest challenges was not being able to integrate our code. Due to the nature of all of the four projects we worked on, we planned to split up and tackle separate complex bits. However, while we did take some time to build the individual pieces of our final application, it took us a lot longer to properly integrate and chain each of the pieces into a fully functioning, automated process. Not updating as frequently to GitHub at the later stages didn’t help much either: we still had issues making sure that we were coordinated in terms of code and in terms of the packages we had access to from the start.
Accomplishments that we're proud of
We’re primarily proud of the fact that we were able to persevere in completing our project to the best of our abilities, even after changing project ideas three times. We were able to finally figure out how to determine Youtube links based on word choices in handwritten notes. Machine learning is fairly new to most of us, so being able to use Microsoft Azure and Keras was pretty interesting.
At the same time, this is a first-time hack for all of us. While one of our members has organized hackathons before (knowing the general strategy when it came to what we could and couldn’t do at a hackathon), we didn’t have exposure to a lot of the skills which hackathon veterans would know like the back of their hands, such as setting up a full-on web server or picking the best backend architecture for the job. It’s amazing to us that we were able to integrate Microsoft Azure and the YouTube and Twilio APIs, along with neural networks, natural language processing, and computer vision, all into an interesting web application. We’re proud of the fact that we were able to face these issues head-on and put our noses to the grindstone in order to fix the many problems we had in making this app.
What we learned
The first project idea that a team comes up with will not necessarily become the final project. Switching between projects is relatively common, and it is satisfying to finally figure out what the team’s final project is.
Programming in PHP is difficult. Also, web development is significantly more complex than academia makes it out to be. We primarily learned a lot more about how to manage systems on a local-host, and to build with APIs and tools which can work sequentially with each other to achieve success.
We also made sure that, for the most part, we were relatively well-coordinated with where we put our code, but we realized that integrating our separate pieces as quickly as possible could make us more adaptable and agile for future hackathons. We were able to (relatively quickly) implement individual features and utilize the libraries and APIs we had at our service. However, a big part of what we can improve upon is building placeholders for other functions and encapsulating code well enough that we can integrate it without an issue (committing changes faster would have also helped: we ended up having to migrate to an entirely different repository just because our original repository was too out-of-date due to our negligence).
What's next for BigRedNotes
Overall, we are extremely proud of our first hackathon project. In the future, we hope to build this into an actual website application; we are currently running this through xampp, a local server configuration for testing purposes. We also hope to partner with TakeNote, a local business that uploads more statically-typed notes, in order to expand their business, and to build applications and cases which could benefit the whole K-12 student population. In doing so, we can make BigRedNotes more efficiently populated and give it more far-reaching implications in terms of what converting handwritten or typed notes to relevant media files could provide for the empowerment of digital and collaborative learning. Maybe, for instance, it could be extended to an augmented-reality sphere to make QR codes nearly obsolete: making a vision-readable BigRedNotes-based poster that can automatically link to an organization’s website through the algorithms parsing through the text.