Lecture Helper

Initial page with files uploaded. Query and key-word typed in.
Sample results page.

Inspiration

We've always had questions when we finished attending lectures and reviewing the material. We decided to create a website where we can ask any question and get answers for ourselves!

What it does

A user will upload a lecture video (mp4) and the transcript of the same video to the website. The user then submits a query about the uploaded material and a keyword related to the query. The website returns up to two answers: one by looking at the uploaded transcript and another by processing a Wikipedia page. Additionally, the website will return document summarizations of the Wikipedia article and transcript used. Finally, the website will embed the uploaded lecture recording to the bottom of the page so that users may re-watch the segment where the transcript answer came from. This will help students with questions about any lecture material or any Wikipedia article. This will help with remote learning and make studying more interactive when they study using this website.

How we built it

We used deep learning models specializing in NLP tasks from the HuggingFace model library that was suited for the tasks of question and answering and document summarization in order to retrieve answers for a user query. We built the website using Flask, HTML, CSS, BootStrap, and Python.

Challenges we ran into

One big challenge was finding a good idea. We spent hours brainstorming ideas for the USAA challenge and finally settled on this project after careful deliberation. After finalizing a topic idea, we ran straight into problems. Originally, we wanted to use the Zoom API to retrieve live transcriptions of a lecture and then answer questions in the chat, but we didn't find sufficient information online to pursue this idea further, causing us to pivot our problem statement. When looking at different NLP models, we found them difficult to configure due to poor documentation, and these models often didn't answer questions sufficiently.

Accomplishments that we're proud of

We are proud of configuring our NLP models to work with our back-end application to create a working front-end UI, and being able to get accurate results from the NLP models that we found. We're also proud of getting 8th place (out of 24 teams) in the CTF!

What we learned

We learned about the different tasks that NLP models are capable of (e.g. text generation, document summarization, image captioning, etc) and learned which tasks suited our needs. We learned more about the capabilities of Flask, HTML, and other technologies that we used. Oh, we also learned how to google answers for a CTF.

What's next for Lecture Helper

Part of our original problem statement was to have this bot in the class itself, answering students' questions if any came up during the lecture. Unfortunately, we found this too difficult to do within the time constraints. So, in the future, we hope to be able to build this capability and get the most out of online lectures. Additionally, we had chosen this project idea because we would have actually used this in our day-to-day lives; so additional improvements and further feature implementations as we use this project in our lives and realize its potential are entirely possible.