COVID-19 has changed the way we learn due to many classes have moved to systems like Zoom and Google Meet. As these changes are necessary for keeping students and teachers safe, these mediums do not provide the same quality and engagement as in-person learning. We want to make a product which takes part of the work away from teachers so they can focus on making their lectures more entertaining and interested.

What it does

Our system allows teachers to create questions for sections they deem important during a lecture. The system automatically summarizes and creates relevant questions based on key words which can be used for quizzes, discussions, or study notes. This system evolves on the study technique of “Active Recall”, where you answer questions while learning to remember and understand topics better.

How we built it

Machine Learning -- This system was accomplished using the Google Cloud Speech-to-Text for speech recognition and the T5-bas training model for summarization. Pre-trained models seemed the most efficient as we did not possess the amount of data or time to create and train an accurate model ourselves. A similar model based on T5, named End-to-End Sequence-to-Sequence Question Generation was used to produce questions based on created summarizations.

Intuitive User Interface -- To develop this interface, we initially started with Figma. This allowed us to visually depict how our website should look and work along with easily acquiring the CSS. The teacher interface was designed to not take too much attention away from the lecture. Teachers would be able to make a room and give students a code to join. Questions made by from the recordings are automatically uploaded to Google Cloud. These questions would be able to be viewed by students using the student section by joining a room. The questions themselves would be released at the end of a lecture. These rooms would display all the questions generated throughout the lecture.

Backend Design -- The backend of Lecture T-Ai had always been decided to be using Flask and JavaScript to both host a database for login info along with recording and storing audio clips for our ML backend. To create an easy method for creating audio clips which are later broken into chunks for transcription, we used JavaScript. We used JavaScript’s inbuilt media encoder to generate .webm files which would be later converted to the .wav files we would need to process the audio.

Challenges we ran into

This system was initially accomplished using the CMU Speech for speech recognition and the T5-small training model for summarization. Since these were both processed on-device, the time to produce a summary was quite long. Due to the time needed to process, alternative methods were researched. This resulted in a change from CMU Speech to Google Cloud Speech-to-Text. This allowed us to utilize Google Cloud’s computing power to significantly reduce the time we needed to produce accurate transcriptions. This also allowed us to use a larger training model (T5-base) for summarization.

Accomplishments that we're proud of

The software was able to successfully summarize and generate questions based on recorded audio. The questions generated are content sensitive and should successfully increase engagement in the classroom.

What we learned

Code that runs efficiently in the background is important. Using multpile layers of translation for files wastes time and processing power. Streamlining this or exporting it to the cloud allow for faster computation and a more seamless experience for the end user.

What's next for Lecture T-Ai

Future updates for Lecture T-Ai include possible mobile app versions along with the ability for students to answer questions and possible get graded by their teacher. This feature was omitted in this version due to some generated questions being erratic/inconsistent. Hopefully with more users on this application we can implement a feature where professors would be able to rate the questions generated, and in turn we would be able to further train the algorithm to generate more applicable questions.

Share this project: