Video Presentation
Timelapse Of Project Development
Inspiration
We noticed that as we progressed in our education, we would occasionally miss important pieces of information in classes that would be on tests. That's why when we read the theme, the idea to take better notes instantly came to mind. We thought that if there was some way to be able to save your entire class and be able to review a summarized version, it would allow full engagement in class while keeping the best notes for revising. That is why we set out to make No Notes the app that turns your audio recordings into summarized notes.
What we learned
Throughout this project, we were constantly having to adapt to challenges that would appear. For example, we learned project, and file management, file sharing systems, and the ways to share our ideas. We had to work together to solve an array of problems, for handling user input, dealing with conflicting files, and sorting through enormous amounts of data. In the end, however, we were able to complete everything as a team and learned to be extremely proud of what we accomplished.
How we built it
No Notes was developed with a mixture of Python for the back-end, and Javascript for the front-end. First, on the front end, users will either upload a file or record from their microphone. The audio is then converted to a usable format with Javascript. This is then uploaded to the back-end, where it will undergo processing before being returned. The Python back-end will first split the audio recordings into consumable pieces for Google’s pre-trained speech-recognition AI. The AI will then transform these speech clips into text, combines them together, and outputs a transcript for the audio recording. This transcript will then be processed by an AI called Punctuator2 which fills the transcript with punctuation and helps separate it into ideas. This is then fed into a summarizer which essentially breaks the transcript into ideas made of words, and sentences, and ranks them based on how often they are used or reiterated. It then outputs sentences, acting as a summary of the text. This summary is then returned to the front end and given to the user. The user then has the option to save the transcription, and notes in a database for later use.
Challenges we ran into
We faced many challenges during development. During the initial phases, we wanted to include something like GPT-3 for summarization. However, we realized we wanted to create an app that anyone can run. Eventually, we came across the bert-extractive-summarizer. Not only was it exactly what we were looking for, it was free and easy to run! During the final stages of connecting the summarizer and the transcription, we realized that the summarizer requires proper punctuation in order to rank information and properly summarize! In order to solve this, we looked for ways where we could solve this without having to use a different transcription model. Eventually, we were able to find a speedy model online that can predict punctuation. Exactly what we need! Finally, one of the biggest issues we faced was the microphone audio-to-text conversion issue. We were having issues converting the WEBM audio recorded into a WAV file - needed for transcription. We were able to find references with the configuration settings.
What's next for No Notes?
- As for future development ideas, we hope to be able to upload more types of files
- Host it on a live server with authentication so multiple people can save notes
- Add some sort of NPL model for smarter notes
- Add a functionality where you can convert notes into flashcards
Log in or sign up for Devpost to join the conversation.