VideoNotesAI

Inspiration

Due to COVID-19, millions of students across the world have been forced to quickly adapt to video lectures and online education. To ease this transition and make studying more efficient, I wanted to help students by summarizing any video lecture transcripts while also capturing the full lecture transcript so students can pay full attention to the video lecture instead of being distracted with taking notes.

Also, teachers/professors/tutors (academicians) spend a lot of time generating test papers and quizzes manually. Similarly, students spend a lot of time on self-analysis (self-calibration). Moreover, students are dependent on their mentors for the self-analysis. Hence, I have also integrated a practice text generating NLP-based algorithm using which students can generate practice quizzes for the same video lecture.

What it does

VideoNotesAI is a web application where students can give the URL of any youtube video lecture and can generate the full lecture transcript, summarized transcript containing all the important points and they can even generate practice quiz tests containing a set of questions with answers from which they can do a self-analysis (self-calibration). A similar approach can be used by mentors for creating test papers and quizzes.

How we built it

I have built this web application using Flask. The web app accepts youtube video URLs (as of now, I want to make this work for any video lecture URL), extracts the youtube video and converts the video file to audio file using youtube-dl package, sends the audio file to Azure's Speech-to-Text API to get a full transcript, sends the transcript to an NLP-based microservice to get a summary of the transcript, sends the summary to the frontend to display to the user.

Further, in case the user wants to generate a practice test from this summarized text, then it is achieved by extracting all the important keywords from the text using named entity recognition, then for each of the keyword we will extract corresponding sentences that has the word from the summarized text and now in order to generate multiple choice questions we get some distractors (wrong answer choices) using conceptnet and it's done, the sentences are printed with the keywords as blanks and the distractors along with the answer are provided as choices.

Technologies Used includes:

Python : Python is the programming language in which this entire project has been built.
Flask : Flask is used to build the backend of the web application.
NLTK : This is the python library which is used for summarizing and quiz generation.
Javascript, HTML and CSS : These are the technologies which have been used to create the frontend of the web application.
Microsoft Azure API : This API service has been used for speech-to-text conversion during the generation of the video transcript.
youtube-dl : This is a python package which has been used for converting the youtube video URL into an audio file.

Challenges we ran into

This project was very challenging to be honest. This was my first time working on Azure AI, so it took quite some time to become familiarized. Further, managing packages and dependencies while working with machine learning is always a difficult task.

Accomplishments that we're proud of

This is ne of the most technically sophisticated hackathon project of mine so far, as the project has many parts like youtube-dl (for URL to audio file conversion), Azure Cognitive Services (for the Speech-to-Text API), and NLP (for text summarization and quiz generation).

What we learned

Learnt using Microsoft Azure Services. Created an application which will be really helpful for my personal use too.

What's next for VideoNotesAI

This is just the beginning for VideoNotesAI. I want to make a chrome extension for the same which will directly take the URL from the browser and provide results to the user without much hassle. Further, I want to provide options to download the summarized text and quiz for later reference. Also, I want to add a feature for providing links for further in-depth knowledge based on the video content using NLP.