Steno

Inspiration

Covid-19 has influenced the transformation of the workplace into a digital one. An abundance of meetings now are either hybrid/remote. However, paying attention to social cues and body language virtually, while also attempting to effectively record information, can be exhausting. We developed Steno, a virtual notetaker, to help you concentrate and focus on communicating with your peers. Steno is also capable of providing an accessible environment for people who are hard of hearing, as well as those who lack the motor function to type/write consistently.

What it does

Steno has two different functionalities, either through Steno's website or real-time transcription through an API that can connect to virtual meeting software products. The website uploads a .wav audio file to be sent through our API, where Steno uses AI to convert the audio files to text and create a summary though the context provided. When Steno is alongside you in a voice call, it will listen as the meetings ensues, summarizing the topics discussed throughout the meeting. Steno also gives the user the option to output a full transcription of the meeting instead of a summary, if desired.

How we built it

Our software consists of three parts: an API, a website, and a Discord bot. The API is responsible for receiving audio clips and using the AI platforms, Whisper and GPT3 to create an AI-generated synopsis. It also has the capability to do consistent sessions, a feature we use within the Discord Bot Demo. The discord bot is capable of*digesting a conversation in an audio channel and outputting live summarization as the members converse*. The website provides a simple access point to the API, and allows you to upload an audio file, receiving back a synopsis of the conversation.

Challenges we ran into

Our team's greatest difficulty was developing with Angular JS. Our team had little experience with front-end dev, and debugging our website was a regular occurrence. We also encountered a problem with our demo integration with Discord. Our original python discord bot was using an older, outdated library that used methods of voice recording that were experimental at best. Once we found a library that was properly updated and maintained, our struggles with the discord bot were greatly diminished.

Accomplishments that we're proud of

The most impressive feature of our app, is our API, which can integrate multiple sessions seamlessly and create an overarching synopsis of these sessions through our AI libraries. Even though our website may be lacking visually, we are happy we were able to create a drag & drop file upload in the timeframe we had, especially with our little knowledge in front-end.

What we learned

Throughout the event, our team worked diligently to build a great program while improving our skills. Within all aspects of Steno, our team learned and improved skills including but not limited to: Python, Angular JS, CSS, HTML, Flask, GPT3, and Whisper.

What's next for Steno

Steno has the ability to become commonplace within the virtual workplace. With an ever-increasing amount of businesses moving to either a hybrid or virtual office environment, Steno can be fully utilized to assist with secretarial work, from general meeting notes to full meeting transcriptions. In the future, integration with the most popular applications, Zoom and Microsoft Teams, is desired, if not required, as they dominate the virtual call industry.

Built With

• angular.js
• caddy
• discord
• flask
• gpt3
• python
• whisper