NOTO - Notes Together

Inspiration

College is a place where people from various backgrounds come and learn from a wide range of disciplines. However, it can be challenging for students studying in a secondary language. A solution to this is to provide relevant educational resources from a student language. Thanks to modern technology and the various OpenCourseWare initiatives’ commitment to democratizing the access to education, students all over the world can access educational resources such as video lectures, textbooks, and extra practice in their native language for free.

What it does

Our product, NOTO – Notes Together converts lecture audio files into text using the SpeechRecognition API, extracts keywords using the MonkeyLearn API, translates the keywords into the student’s desired language using GoogleTrans API, and feeds it into our custom Google Programmable Search Engines which search on several OpenCourseWare websites from different languages to fetch relevant resources.

How we built it

We built the project by starting out with the back end, establishing both programs to make custom search queries and filtering out key words using MonkeyLearn API. We then worked on the front end by using wix, and using flask to connect the two. Then we built new features such as speech to text and translation using GoogleTrans API and SpeechRecognition API.

How it works

Velo provides a programmable user interface that allows us to leverage simple graphical elements as well as more advanced backend scripts. By using the Flask library in python, we were able to communicate between python apps and front-end development. Currently, the website only communicates with a local server; however, in the future, this could be made into a completely backend service. I also had the chance to work on speech-to-text software. This software uses Google SpeechRecognition API to automatically transcribe speech. I had to figure out how to transfer and upload a sound file from the front-end to the backend through a lot of data conversions. The transcribed file is then put through a filtering algorithm that returns keywords, and those are translated, placed into a custom search engine, and results are returned as links to the front-end.

Challenges we ran into

Since this is the team’s first ever Hackathon, most of the challenges that were faced were technical. One of the biggest challenges was to find free and reliable APIs and services that work for the tasks in hand, including searching for keywords from text, custom programmable search engines, speech to text, and translation. The other challenge was learning to use Wix Velo to handle uploaded media files. There were limited tutorials for Velo, which made this extra challenging. Lastly, since the project heavily relied on data from OpenCourseWare initiatives, it was difficult to find well-maintained OpenCourseWare projects such as that of MIT’s in other non-English languages. - Ryan

Accomplishments that we're proud of

The team is really proud of setting up the website and getting the speech to text to work. It was a lot of trial-and-error to input audio onto our website; the team has tried to use microphone inputs but yielded no useable results. Ultimately, the team found out a way to upload .wav files onto Wix and from there fetch the audio input and later process it from the back-end. The team is also proud of the success in integrating the front-end Wix user-interface and the back-end Python programs and APIs. There were many technical details to pay attention to (such as edge cases), but the team was able to resolve them at the end.

What we learned

The team learned a lot about working with APIs and connecting both the backend portions and the frontend portions, which involved learning how to utilize Flask. Since we were all first time hackathon goers, we also learned how to work together effectively by using git, while also dealing with learning how to subdivide work yet still have each member understand every part of the application.

What's next for NOTO - Notes Together

The team sees our product being integrated into classrooms where professors and students can work together to find extra material that may help everyone learn better. The team also sees the product being expanded to work with more languages, bridging the gap caused by the exclusivity of education.