TL;DR

Inspiration

There is a possibility that we miss taking important notes in class. We might have to go through the lecture all again to work on it. As a team, we wanted to ease things for students and bring the transcript of the lecture in a couple of clicks.

What it does

TL;DR takes an audio file and converts it to raw text and corresponding summarised notes for the lecture. Additionally, for the convenience of International students, this web application has the ability to translate the text into popular languages including Spanish, French.

How we built it

Our application is hosted on Node.js Server that asks audio file from the user. The file is encoded and securely uploaded to Google Cloud Storage. The file path is extracted and a request with few configurational setting is sent to Google Cloud Speech-to-text API for processing.

In addition to summarising the content, we add translating feature to help international students. The text is then sent to Google Translate API for translation to popular languages like Spanish, French. This is achieved using the APIs provided by the Google Cloud Platform. In this way, a single application takes care of 3 important feature that reduces a student's reading time.

Challenges we ran into

Our biggest challenge was using Google APIs and stitching all these features to work as one application. Using very limited Google Speech-to-text (STT) documentation, linking Google storage to use STT took us a lot of time. In many cases, the functions/files uploaded needs to be securely sent through keys and the keys had to be passed along different services. We tried to summarise the text using a Neural Net structure with TensorFlow backend. We were able to find very few news contents from BBC News and tried to train our model after preprocessing. In the process, we tried to tokenize the content and train the model. However, due to resource constraint and lack of data, we were not able to complete the task and had to drop back to use available algorithms. We then had to choose DeepAI Algorithm and fed with data from Google STT APIs after some modifications.

Accomplishments that we're proud of

We are proud of the things we achieved in less than 24 hours. Being new to Google Cloud functionalities and TensorFlow, we spent a lot of time in understanding the concepts. On average, our application takes less than 10 seconds to complete the entire task and this is commendable. Moreover, it is gonna be helpful to students and help them save a lot from the hustle.

What we learned

We learned various ways to use the Google Cloud Platform and its API along with stitching various services in a single application. We also learned from our failure while trying to train a model using Neural Net structure and TensorFlow. Understanding Neural net and implementing on a small data set gave us enough information to implement the algo on even bigger data sets like DUC data.

What's next for TL;DR

There are majorly 2 easy applications where TL;DR can be improved:

TL;DR can be linked with NCSU Engineering online to automatically transcript every video uploaded by the staff. We can open a couple of APIs through which requests can be made from EOL website and the output can be attached with the uploaded file.
We aim to build an algorithm that summarises the text locally rather than banking on third-party algorithms. We are planning to use multiple GPUs on Google Cloud.