The Problem

With COVID-19, students everywhere are taking online classes at an unprecedented rate. Not only are universities struggling to keep up and find professors and IA's willing to teach classes, but these educators are also struggling to adapt to the online format.

Our Solution

With Autonotes, educators can quickly generate not only a word-for-word transcript of their lecture, but a complete, structured, and in-depth summary of their entire lecture with timestamps for slides if needed.

We accomplish this with a novel word-vector based clustering machine learning algorithm that groups the text by an assigned number of sections based on word-association. These sections are then laid out into a summary document given to the user.

How does it work?

On the front end, we create a web app built on node.js that allows users to upload and submit their chosen audio or video file. This audio is then stored on database and then sent to Google Cloud's Speech-To-Text API which returns a transcript of all the words spoken.

Optionally, the video is sent to our own machine-learning API that uses pixel-subtraction to identify unique slides, and returns a set of slides and timestamps.

Afterwards, we send this transcript to our clustering API, which runs the appropriate machine learning algorithms to return a structured summary, divided by topic.

This summary is then finally rendered on our website, able to be downloaded, for the user to supply to their students.

Next Steps

Since we mainly decided to chill this hackathon to learn new things, much of the above isn't due to be implemented until tomorrow. In particular the transition from a node.js based server to a Ruby on Rails based server, the implementation of a MongoDB database, and the completion of our natural language processing APIs. After the hackathon we're completing the project as a full project we can showcase on our devpost and resumes.