Redis Audio Transcription Pipeline - We have built an audio transcription pipeline using Redis Gears and Redis Streams.
intelliSchool Browser plugin
intelliSchool Browser plugin - Smart Quiz - Generated automatically for the YouTube video using NLP and Machine Learning
intelliSchool Browser plugin - Smart Notes - Generated automatically for the YouTube video using NLP and Machine Learning
intelliSchool Browser plugin - Smart Quiz Report
intelliSchool Browser plugin - Download notes
intelliSchool Web Application - Dashboard
intelliSchool Web App - Quiz - Generated automatically from the video uploaded using NLP and Machine Learning
intelliSchool Web App - Notes - Generated automatically from the video uploaded using NLP and Machine Learning
intelliSchool Web App - Leaderboard
What is intelliSchool?
Web Application and a browser plugin that generates quiz and notes automatically for any educational video using Redis Event driven architectures, Redis Modules, Natural Language Processing and ML
In the midst of COVID-19, the whole world has realized the power of online learning and the impact it can have on millions of students. Be it K-12 or higher education, virtual learning is talk of the town. Although there are plenty of video lectures available online and usage is extravagant, often these lectures can get too long, monotonous and quite frankly, boring! People zone out frequently and lose track of what's going on. They end up rewinding the video and watch again which really is a waste of time.
In this process, there is a lack of feedback; there is no one to test you if you've really understood what you've been listening to. What if there is a mechanism that automatically generates a quiz for any video to evaluate your understanding? Also, not every student likes to take notes. But all of us need something to refer to before an exam. What if there is someone who can automatically generate notes for you while you concentrate on listening to the lecture in the video?
Also, teachers spend a lot of time preparing content such as quizzes and notes to help their students. What if this process can be automated so that the teachers can spend time on more useful tasks? What if there is a solution to make it easier for parents who teach their children?
In the process of building an application that solves the above problems, can we also explore the possibility of improving the efficiency of the transcription and quiz generation process using Redis Event-driven architectures?
Solution - What it does?
We have built 2 apps that solve the problems mentioned above:
1. intelliSchool Web Application
- Teachers/Students(or Anyone who wants to learn) can upload meeting recordings to our application or paste the link to a youtube video. Application will create a study room and generate a unique link to the study room. Teachers can then send invitations to their students to join the study room. Students can then use the link to enter the study room.
- The application will use the video uploaded by the teacher to automatically generate quiz and notes using Redis Event driven architectures, Redis Modules, Natural Language Processing and Machine Learning. Quiz and notes will be available along with the video lecture next in the study room. Students can also discuss with each other using the 'Ask questions' section. This has been implemented using Redis Streams
- At the end of the quiz, they will be able to view a report that contains their score along with the questions that they answered incorrectly.
- Students can click on ‘Where was this question from?’ button. Our App will take them to the point in the video from where the question was generated so that they don’t end up re-watching/rewinding the whole video again.
- While students are learning a concept, if they have any doubts, they can use the discussion section below the video to ask questions. Teacher, other students or our bot can answer their queries. This is implemented using Redis Streams.
- Students can download the notes. Students can also edit and customise the notes.
2. intelliSchool browser extension
If the user wants to learn from free videos on YouTube, then they can use the intelliSchool browser extension. It implements similar features as the web application and the content is displayed right below the video.
How we built intelliSchool? - Redis Usage
We have implemented the application using Redis Gears, Redis Streams, RedisJSON, Redis Pub/Sub and Redis Data Structures.
Our application converts speech-to-text in order to facilitate the automatic generation of quizzes and notes. Audio transcription is a costly and time consuming process. While building the app, we experimented with speech-to-text services provided by major cloud providers. We observed that it takes several minutes to transcribe even a short video. For best user experience, the application has to generate content in a few seconds. To achieve this performance improvement, we redesigned our application using Redis Event-driven architecture. We built a transcription pipeline with the help of Redis Streams and Redis Gears and were able to reduce the total transcription time from minutes to seconds.
The following diagram depicts the flows between the system's parts:
Flow between the system parts is explained in detail below:
1.When a user uploads a video to intelliSchool, MediaProcessor component extracts audio from the video using ffmpeg. It then breaks the audio into fragments on silence and appends each fragment to TranscriptionInput Redis Stream with a call to XADD. This message consists of the video id, fragment number and the fragments raw encoded using pickle.
2.The addition of a message to the TranscriptionInput Redis Stream triggers the execution of the Transcriber Redis Gear. Transcriber Gear implements a pipeline of operations and is made up of these two main parts:
- It sends each fragment to IBM Watson to transcribe the fragment into text and adds the text to TranscriptionOutput Redis Stream.
- After all the fragments are processed, it accumulates and sorts the text based on fragment id and then adds accumulated text to a Redis hash
3.The addition of a message to the Redis hash triggers the execution of the Generator Redis Gear. This Gear implements a pipeline of operations and is made up of the following parts:
- Gets the transcribed text and adds punctuations
- Chooses Important sentences from the text for further processing
- Generates Quiz and Notes using custom algorithm
- Stores the generated Quiz and Notes as RedisJSON.
- Publishes a ‘Complete’ message to the Status Redis PubSub channel and status is also stored in Status Redis Hash.
4.Frontend will receive a notification about the completion of the process. It will then make API calls to the web server to fetch Quiz and Notes. Web server will return the quiz and notes that are stored as RedisJSON.
The application is implemented in Python 3, and consists of the following parts:
init.py: this initializes Redis and the Redis Gears.
transcriber_gear.py: the Redis gear that implements the transcription pipeline.
generator_gear.py: the Redis gear that generates the quiz.
media_processor.py: MediaProcessor component extracts audio from video, breaks the audio into fragments on silence and appends each fragment to TranscriptionInput Stream.
text_processor.py: the Text processor adds punctuations to the transcribed text, selects important sentences.
api.py: Flask API.
Challenges we ran into
Audio transcription is a costly and time consuming process. While building the app, we experimented with speech-to-text services provided by major cloud providers. We observed that it takes several minutes to transcribe even a short video. Since our entire application was dependent on it, and nobody likes to wait for that long, we had to improve the performance. That is when we switched to Redis event-driven architecture.
Accomplishments that we're proud of
With the help of Redis modules and Redis event-driven architecture, we were able improve the performance of our application significantly by processing audio fragments simultaneously and reducing the transcription time.
What we learned
This was our first venture on an event-driven application. It was quite an inspiration to learn how Redis Streams, pub/sub, RedisGears and other Redis Modules can be used to achieve the performance we were expecting from our application. We also learnt that Redis has various use cases that go ‘beyond cache’.
What's next for intelliSchool
- As we were able to make the audio transcription process faster using Redis Gears and Redis Streams, we are planning to extend the platform to generate content such as quiz, notes and flash-cards for live meetings. The application can be the perfect educational platform that can solve the online education needs of millions of students during the current pandemic
- Capture snapshots of important diagrams and formulae from the video using RedisAI, OpenCV and add it to Smart Notes
- Create flash cards to help students learn easily
- Add analytics to help teachers understand the topics that weren't answered correctly by students. This can be very useful for teachers to come up with new ways of teaching such complex topics
- Add analytics for parents to better understand how their kids are performing
- In smart notes section, we want show links to relevant websites to read more about the central topic of the video
- Generate pre-requisites - add links to learn about most important sub-topics related to the video as pre-requisites
- Watch "Youtube Video" for a video demonstration on how we used Redis to build the app.
- To learn more about our project idea, "Watch" for a video demonstration.
- ibm-watson(You need to replace API keys in the project with your IBM cloud keys)
Running the Demo
To run the demo:
$ git clone https://github.com/msvdpriya/intelliSchool.git $ cd intelliSchool $ pip install -r requirements.txt
$ python3 api.py
Run Web Application
$ cd WebApplication # Install all required npm packages by running following command from the command line in the project root folder (where the package.json is located): $ npm install # Start the app by running following command from the command line in the project root folder, this will compile the Angular app and automatically launch it in the browser on the URL http://localhost:4200: $ npm start
install and activate the browser plugin:
1. Download/Clone the folder BrowserPlugin from the Github Repository. 2. Open Chrome web browser. 3. In browser's search bar, type chrome://extensions/ 4. In the extenstions page, click on 'developer mode' at the top right corner of the page. 5. Click on the 'Load Unpacked' button. 6. Choose the folder 'BrowserPlugin/' to install intelliSchool Plugin on your browser. Now, open youtube, activate the plugin and enjoy learning from millions of educational videos on youtube with the help of intelliSchool plugin.