Inspiration

As executives of our school’s computer club, we have weekly meetings to discuss club operations. Our meetings were plagued with productivity issues: when people were absent from meetings, they needed to be filled in afterward. We lacked a consistent way to record meeting content, and resorted to multiple people taking notes, which hindered us from reaching our full capabilities.

These issues in our meetings led us to develop summeet: an AI-powered meeting companion that diligently and intelligently summarizes meeting content and provides useful insights, transcripts, and analysis, allowing all members to focus on the issues that matter.

What it does

Logging in with the session Id and username, users can join a session for a meeting, where a button enables users to get summeet to start recording audio from meetings. summeet then transcribes and summarizes the meeting in real-time. At the end of each meeting, users can download a PDF of the summarized meeting.

How we built it

Leveraging the power of Go and the Gin framework to build summeet, we optimized the web application from the start for scalability and stable performance for many concurrent users. Sending voice data from the frontend to Google Cloud Platform (GCP)’s Speech-to-Text API in real time allowed us to proxy and handle the multitude of simultaneous audio streams. Our backend incorporates a custom algorithm to parse paragraphs and summarize them. Our front-end website incorporates WebSockets to livestream audio to the backend for processing while receiving chat data from other clients concurrently. Apart from the Materialize CSS framework for clean and modern layouts and effects, we created most of the web components from scratch. We additionally used pdfJS for the feature of downloading meeting summaries into an easily readable document for storage. For our infrastructure, we used GCP’s compute engine to run a web-server in the cloud, providing faster and better performance access to the Speech-to-Text API. We implemented our webserver behind an NGINX reverse proxy to provide stable and easily maintainable SSL support, for the best safety and encryption for modern browsers. Our domain, summeet.online, was registered from domain.com, and we used CloudFlare to do our DNS routing.

Challenges we ran into

The first challenge we ran into was streaming live audio from a web frontend to our Go-powered backend. The documentation for microphone audio streaming through javascript is quite lacking compared to other features, so it took skilled searching to figure out how to obtain an audio stream in the encoding that we wanted (opus). We ended up using WebSockets to increase performance in streaming to the backend, since the same stream could be sent to the GCP API without extra processing. WebSockets required encryption to work reliably in modern cross-platform browsers like Chrome and Firefox, so we had to create a reverse proxy to pipe data through. The encoding combined with the buffering of the audio packet streaming still poses an optimization challenge for us, where decreasing buffer size puts a heavier strain on the client processor in encoding rate, while increasing buffer size meant slower text-to-speech processing through the GCP API.

Our second challenge was developing the summarization service. We originally wanted to use an external API to process and summarize the data, but we found that they were poorly designed and inadequate for analyzing extended conversations. Ultimately, we created our own summarization algorithm, which used frequency distribution and weighted averages of words in a conversation to rank sentences by importance.

We also faced challenges properly pointing our domain.com domain to our GCP VM. We resolved this by bypassing the domain.com DNS service, pointing our nameservers to Cloudflare instead to manage our DNS. Other general frontend problems also hindered our progress, such as the custom CSS styling that we implemented requiring working with some very finicky code.

Accomplishments that we're proud of

Successful streaming audio from front-end to back-end Developing and implementing our summarization algorithm Clean and smooth UI and reliable UX

What's next for summeet

There are a plethora of planned improvements on the horizon for summeet. For one, our very hacky code requires a major overhaul of refactoring to be more streamlined. We hope to complete our implementation of planned platform features, including support for a database that holds all of a user’s previous meetings notes, transcripts, and summaries. Moreover, we hope to expand our service to virtual meetings through Slack and Skype integration. The voice isolation ability of our platform could also use improvement, especially in situations where users are in very close proximity of each other. This will help with creating accurate transcripts for meetings in a close setting. Finally, our implementation of the summarization algorithm could be further optimized.

Built With

Share this project:
×

Updates