Class Notes

Class Notes transcribes online classes with subtitles, summaries, topics and sentiments, enabling students, facing poor internet or language barrier, learn better at their own pace

Comment 1

Inspiration

We believe education opens doors. Online classes make high-quality education available to everyone. However, poor learning outcomes are a perennial problem in online classes.

Since the advent of COVID, around 2 years ago, we have been contributing to online classes:

  1. We helped thousands of students access online classes on ~90 BigBlueButton servers, the most popular open-source online classroom software, each server is capable of hosting ~250 concurrent students
  2. We extended features of BigBlueButton by publishing multiple open-source plugins such as Streaming classes on YouTube, Converting class recordings into MP4 format and Optimizing audio and video

However, many of our clients have been complaining that often students are unable to understand much in online classes for many reasons, such as

  1. they have frequent disruptions because of poor internet connections,
  2. they are not as fluent in English to comprehend the lectures, or
  3. they are unable to keep up the pace with lectures.

Now, with AI and Machine Learning, we envisioned building interactive class notes that would aid students in learning better in online classes in a personalized way.

What it does

Class Notes provides you interactive notes of online classes that you can easily refer to, at your own pace, to improve your comprehension. Specifically:

  • MP4 Video with Sub-tiles. View the class recording in MP4 format, along with subtitles, which, unlike BigBlueButton recording, plays smoothly in any browser on a laptop or a mobile device.

  • Transcription. Below the video, you can read the full transcription of the class, annotated with corresponding time.

  • Summary. Interested in just skimming through the lecture? Just go through the crisp summary of the class in a few bullet points.

  • Topics. Want to learn more about specific topic? View the list of key topics discussed during the class and click on a topic to jump to the corresponding section of the lecture.

  • Sentiments. Lastly, get a sense of the tone of the lecturer - positive, neutral or negative, with which specific part of the lecture was delivered.

How we built it

After an online class ends, the BigBlueButton server processes the recording and makes it available for students to refer later.

The Class Notes plugin is triggered as soon as BigBlueButton completes its processing of the recording and converts the recording into MP4 format. It also strips off audio of the MP4 recording using ffmpeg and sends it to AssemblyAI to transcript and provide information such as summary, topics and sentiment analysis.

Class Notes displays that information along with MP4 video the class, enhanced with subtitles, in a nice, easy-to-use UI.

Class Notes also lists out topics of discussion and provide easy navigation to jump off to the section of the class where that topic was discussed.

Challenges we ran into

We first implemented Class Notes with transcription from Google Cloud speech-to-text API. However, we faced two challenges:

  1. Accent of the presenter: We would have to specify the source language - English USA, English UK, English Canada, English Nigeria, English India, and other dialects of English. As we want to automate the transcription process, it would have been inconvenient to ask users to specify the dialect. Also, it won't work as accurately when multiple speakers are involved, involved in a different dialect of English.

  2. Accuracy of transcription: Google Cloud speech-to-text API accuracy is not as good. We were looking for alternates and found AssemblyAI's transcription accuracy significantly better.

Accomplishments that we're proud of

Wider acceptance of Online Classes is one silver lining of lock-downs due to Covid.

We played our part by helping thousands of students access online classes on ~90 BigBlueButton servers, each server capable of hosting ~250 concurrent students, that we ran during this period. We built novel features through our open-sourced plugins for streaming, mp4 video conversion and optimizations for audio and video.

Now we have a similar big opportunity of using Artificial Intelligence in Online Classes to improve learning outcomes and students' successes. Class Notes product is our 1st step in that direction.

What we learned

Artificial Intelligence is changing the world.

Class Notes is a great demonstration of using Artificial Intelligence to transform a subpar online learning experience.

We can further extend this idea to build more tools, such as real-time translation with audio, topic index, clustering based on engagement, virtual assistant etc, which would probably make online learning better than in-class learning.

What's next for Class Notes

Here are some of the features that we've planned for Class Notes:

  1. Currently, Class Notes lists out topics of discussion, during a particular class, and provides easy navigation to jump off to the section of the class where that topic was discussed. We would want to extend this functionality to index topics across all lectures of a given subject. So when a user, for example, would want to know more about a topic, say Cost Function, he can browse through the topic discussions across all the lectures of the subject, say Machine Learning.

  2. Using engagement metrics such as class attendance, answers to polls, chat messages, hands raised, time spent in the class, recordings watched, time spent watching recordings, etc, we may be able to classify students into various clusters of learning outcomes.

  3. We would like to build real-time transcription and translation into multiple languages that we can display during the class.

Share this project:

Updates