Zoom Plus Plus

Online classes suck, we are here to fix them with Twilio and AI.

Introduction

Our goal with this project is to improve the zoom lecture experience. With this project, when you want to transcribe a zoom call, you just go to our website, put in the zoom meeting idea, and get back to class! A twilio bot will call into the zoom meeting and begin transcribing the zoom call using the Google Cloud Services Speech to Text service.

The huge benefit of this service is that the teacher doesn't need to set up anything and you don't have to install anything on your computer. Just put in the zoom ID and you're good to go.

Tech Stack

Here is the general flow of data through our project zoomplusplus homepage --> our backend -> Twilio -> Websocket connection from twilio -> Google Cloud Services Speech to Text -> our backend -> frontend to display transcription

We also have a few other data paths for the chat and user registration.

Hard points

We originally tried to use AssemblyAi for this project, but after many many hours of audio struggles, we gave up and used Google Cloud Speech to Text. AssemblyAI has real time audio transcription, but it requires the audio to be encoded in a very specific format (audio/webm, with 16 bit PCM), this was compounded by the fact that Twilio also sends us data in a very unique phone-call audio format (audio/x-mulaw, 8 bit). Converting between these two formats live was especially hard because AssemblyAI didn't give useful error messages.

After 4-5 hours of trial and error, we decided to switch to the Google Cloud Services Speech to Text as that accepts almost any audio format and integration with twilio was super simple.

We had no other huge snags other than a bit of fighting with SSL certificates.

Things we didn't have time to finish

The chat area neighbor feature isn't fully implemented yet. Right now there is only dummy data to simulate what it would be like to have the chat going.

Reflection

We genuinely think that this product is something that students would enjoy to have and we plan to keep trying to improve the project in our free time.