'Smart Video' is a browser extension that generates MCQ based quiz and notes using NLP and AI for videos in youtube and other ed-tech websites

Purpose & Motivation

In the midst of COVID-19, the whole world has realised the power of online learning and the impact it can have on millions of students. Be it K-12 or higher education, virtual learning is talk of the town. Although there are plenty of video lectures available online and usage is extravagant, often these lectures can get too long, monotonous and quite frankly, boring! People zone out frequently and lose track of what's going on. They end up rewinding the video and watch again which really is a waste of time.

In this process, there is a lack of feedback; there is no one to test you if you've really understood what you've been listening to. What if there was a mechanism to evaluate the understanding? Also, not every student like to take notes. But all of us need something to refer to before an exam. What if there is someone who can automatically generate notes for you while you concentrate on listening to the lecture in the video?

How this application works

I have built a web browser extension that automatically generates a quiz and smart notes for videos in YouTube and other EdTech websites using NLP (Natural Language Processing) and Machine Learning. A user can self-evaluate their understanding by taking the quiz while watching the video. Each question presents the user with an option to move back to the part of the video from where the question was generated. This will help the user in revising the concepts which they weren’t able to understand when they watched the video the first time.

The browser plugin also generate smart notes and summary for the video. The smart notes also provide meanings and examples for certain key words in the notes so that a user can easily understand the overall content of the notes. User can also click on any sentence in the notes to navigate to the point in the video from where the concept summarized in the sentence is explained.

How this application was developed

I used Angular to build the frontend of the web application and Javascript to build the browser plugin. Backend is built using Python.AWS services AWS Transcriber is used to convert speech to text. We are using python packages such as nltk, gensim and our own Algorithms to automatically generate MCQ based quiz and smart notes from the speech in the video.

Difficulties & Challenges you faced

The process to generate quiz, notes or summary from a video is complex and time consuming machine learning problem. It was a challenging task to reduce the total time taken by this process from several minutes to a few seconds. We were able to accomplish this over the weekend.

Accomplishments that I'm proud of

I consider every aspect of the project as an accomplishment as it was a new learning every step of the way.

What I learned

I was able to explore different NLP libraries, their capabilities and restrictions and also explored different open source and cloud solutions and how to use such already existing solution in our project to improve its efficiency.

Steps to test the project:

Steps to serve flask app(Backend):

NOTE: Make sure that you have installed all of the python packages/dependencies listed below: flask, flask_cors, nltk.corpus, urllib, nltk.tokenize, gensim, youtube_dl, punctuator, requests, nltk.tag

  1. Download/Clone the folder SmartVideoBackend from the Github Repository.
  2. cd to SmartVideoBackend folder on your machine and run This will serve Flask app in development mode.

Steps to download and install the plugin:

  1. Download/Clone the folder SmartVideoPlugin from the Github Repository.
  2. Open Chrome web browser.
  3. In browser's search bar, type chrome://extensions/
  4. In the extenstions page, click on 'developer mode' at the top right corner of the page.
  5. Click on the 'Load Unpacked' button.
  6. Choose the folder 'SmartVideoPlugin/' to install Smart Video Plugin on your browser.

Finally, follow the steps mentioned below to test the project:

  1. Open youtube on your chrome web browser.
  2. Click on the plugin icon next to the browser's search bar. Click on Enable Smart Video button.
  3. Browse an educational video. The plugin will automatically generate smart quiz and smart notes.

What's next for Smart Video

I want to add several new features like the following:

  • Capture screenshots of diagrams and formulae from the video using Computer Vision and add it to Smart Notes.
  • Create flash cards
  • Add analytics to help teachers understand the topics that weren't answered correctly by students.
  • Add analytics for parents to better understand how their kids are performing.
  • In smart notes section, we want show links to relevant websites to read more about the central topic of the video.
  • Generate pre-requisites - add links to learn about most important sub-topics on the video as pre-requisites
  • Build an app for mobile devices.

Go-to-Market Evaluation

In the next 2 months, I am planning on building a team to create new features and host the backend of the project on a Cloud Platform. I have decided to keep basic features such as auto-generated quiz free until the COVID-19 pandemic ends. Later, I am planning to implement a freemium model with auto-generated quiz, notes, analytics free for upto 5 videos per day. The revenue generated from subscriptions will be used to host the backend services.

Built With

Share this project: