Inspiration

Whether we're attending general staff meetings or getting together with classmates to work on a group project, no one ever wants to be the note taker. You can't pay as much attention to the meeting since you have to concentrate on taking notes, and when meetings get long, it's often difficult to maintain the attention required to take diligent notes.

We wanted a way to stay more focused on the meeting rather than the memo. What if we could utilize the growing smart speaker market in conjunction with NLP algorithms to create comprehensive notes with code?

What it does

Scribblr takes notes for you! Not only that, but it automatically sends an e-blast to meeting attendees at the end and adds discussed deadlines and upcoming meetings to your calendar.

Just tell Alexa to start the meeting. Carry out the meeting as normal and, when you're done, tell Alexa the meeting is over. She will immediately begin to create a summary of your meeting, making note of the most important discussion points:

  • Approaching deadlines and due dates
  • Newly scheduled meetings
  • The most important decisions that were made
  • A short paragraph summary of the overall meeting

Additionally, anything discussed during the meeting associated with a date will be summarized into a calendar event:

  • Date and time of the event
  • Title of the event/task
  • Important topics associated with the event/task

When the meeting is over, those who participated in the meeting will receive an email with the automated Alexa notes and all calendar events will be added to the official company calendar. Why take meeting notes when Alexa can do it for you?

How it works

The hack begins with an Alexa skill. We created a custom Alexa skill that allows the user to start and stop the meeting without skipping a beat. No more asking who is willing to take notes or hoping that the note-taker can keep up with the fast-pace -- just tell Alexa to start the meeting and carry on as normal.

The meeting is then assigned a unique access code that is transmitted to our server via an AWS Lambda Function which initiates the audio recording. Upon completion of a meeting, Alexa makes a request to the server to transcribe the text using the IBM Watson Speech to Text API.

But at the core of Scribblr are its Natural Language Processing (NLP) algorithms:

  • The final transcript is first preprocessed, involving tokenization, stemming, and automated punctuation. Automated punctuation is accomplished using supervised machine learning, entailing a recurrent neural network model trained on over 40 million words.
  • The Transcript Analyzer then integrates with the IBM Watson Natural Language Understanding API to detect keywords, topics, and concepts in order to determine the overarching theme of a meeting. We analyze the connections between these three categories to determine the most important topics discussed during the meeting which is later added to the email summary.
  • We also isolate dates and times to be added to the calendar. When a date or time is isolated, the NLP algorithms search surrounding text to determine an appropriate title as well as key points. Even keywords such as "today", "tomorrow", and "noon" will be identified and appropriately extracted.
  • Action items are isolated by searching for keywords in the transcript and these action items are processed by performing POS tagging, facilitated by a trained machine-learning module, ultimately being appended to the final meeting summary of the most important points discussed.

Challenges we had

There were a lot of moving parts to this hack. Many of the programs we used had dependencies that were incompatible or didn't have the functionality we needed. We often struggled to attempt to work around these conflicting dependencies and had to completely change our approach to the problem. The hardest part was bringing everything together as a singular product -- making them "talk" to each other so to speak.

Since most of our code had to run on a server and we interfaced with a number of APIs, we had to manage multiple sets of credentials and deal with security measures through Google, Amazon, and IBM Watson, but what complicated this even more was that we were all developing on different machines, so what worked on one computer would fail on another and we had to work together to identify the tree of dependencies for each piece of the project.

Accomplishments we're proud of

It works! We worked down until the wire, troubleshooting compatibility issues all night and actually got all four very distinct components to work together seamlessly. This is something we would actually use in our daily lives, at club meetings, study groups, class project meetings, and staff meetings.

Things we learned

We learned a ton about NLP algorithms (as well as their limitations) and how to connect different pieces of a software system to a central server (which was arguably one of the hardest things we had to do, meaning we learned the most here). We also delved deeper into AWS Alexa: integrating Lambda functions, connecting to third-party applications, and publishing a skill.

What's next for Scribblr

We would like to add more functionality for commands during the meeting, such as updating emails, publishing to more than just a central calendar, pausing the meeting, and controlling remote devices directly using Alexa instead of having to go through a server to do so.

Share this project:
×

Updates