Notelify

Inspiration

Ever feel like you're in a note-taking sprint during lectures? Especially in those rapid-fire history classes? We get it. We're tired of students frantically note-taking during lectures instead of focusing on the main ideas. But what if you could get good notes from a simple audio recording of the lecture? That’s why we made Notelify.

What it does

Notelify takes an audio file or text transcription and converts it into an easy-to-read, concise, and meaningful notes PDF from the lecture. We utilize AI to find the most important parts of a lecture based on keywords and meticulous prompt engineering. We also use vector-search to allow students to search for information from a lecture with a click of a button.

How we built it

We created the web application using React as our front-end and Flask as our back-end. We uploaded user-files to Firebase’s Google cloud storage, then used a Google cloud tool to transcribe our audio files into usable text. On our backend, we split our transcribed text (which was tens thousands of tokens) into smaller chunks (1000-2000 tokens). Using a map-reduce langchain with GPT 3.5, we generated LaTeX summaries of each chunk and then reduced these summaries into one large summary. Every time that a lecture is parsed, vector embeddings from that lecture are also stored in a Pinecone Vector Database. By storing vector embeddings, users can also ask Notelify questions about the lecture. By finding the most similar vectors in the Pinecone DB, we're able to answer users' questions, even if they are specific to something a professor mentioned during lecture.

Challenges we ran into

We had a lot of trouble at first setting up Google cloud app engine event handlers. For some reason our react application wasn’t properly receiving data about the completion of our speech-to-text transcription, thus we weren’t able to call our back-end exactly when it finished.

One of the biggest challenges in generating our LaTeX summaries was prompt engineering. We spent hours trying to figure out the best way to generate high quality and accurate lecture notes from simple text. Many times, the LaTeX generated lacked mathematical examples or would not compile. By refining our prompts and giving the chat model context of what it was trying to generate, we were able to generate accurate LaTeX notes that compiled and included examples.

Accomplishments that we're proud of

We are proud to have used prompt engineering with GPT and Langchain to get desirable results with LaTeX. This was one of our first times working in a 4-person team and managing all of our version control and software. We're also proud of how we divided up the tasks.

What we learned

We learned how to improve React Flask projects and prompt engineer LLMs using industry standard technology. We also learned how to use LLMs to answer questions using data that we feed it rather than relying simply on its own knowledge. We also gained experience with Google cloud deployment and DNS management. It's better not to pull an all-nighter on the second night. It's better on the first!

What's next for Notelify

With Notelify we hope to simplify the lives of millions of college students by allowing them to focus more on their lecture content than the semantics of notetaking. We hope to expand this to also take in live audio and generate notes real-time while the professor is speaking.