WhatTeachersS(AI)

Inspiration

Having taken violin and piano lessons in the past, we both recognize the the frustration of going home to practice and then realizing that we don't remember half the things our tutor said in the lesson. At the same time, it’s awkward and time consuming to switch back and forth between playing your instrument and taking notes during the lesson. And even if you take notes, you may miss the small details like impromptu things that your tutor says while you're playing. Music students need a way to record what happens in a lesson without disrupting the lesson and store it in a way that they can easily and quickly read through.

What it does

WhatTeachersS(AI) takes an audio recording of your lesson and transcribes it into text in real time. It also lets you manually type notes that you find particularly important so that 1) you don't need to switch between apps and 2) all of your notes, both the transcribed and manually written ones, are stored in the same place. Finally, it sends all of this information to you via email. Without databases or user accounts, this lightweight app offers a snappy and seamless user experience that can be used by anyone straight out of the box.

How we built it

We first built some rough mockups using FigJam. This helped us determine what capabilities had to be implemented and how the different components of the app will be routed. In the next few hours, we focused on getting the following features to work:

• Real-time transcription. Implemented this using AssemblyAI's API, with a node.js backend and using websockets to communicate with AssemblyAI servers in the frontend. The frontend logic is abstracted in a custom React Hook.
• Automatic email sending. Used EmailJS and wrote a custom template.
• Maintaining a persistent state between pages. Decided to use a React states at the root level.
• Routing pages and adding linked buttons based on the user flow. Used React Router.

When all the capabilities are in order, we moved on to implementing the UI using Chakra.UI components. Since EmailJS requires HTML form elements, our email page actually has two forms: a visible one that has an input field for the user's email, and a hidden one which is synced with React states containing the transcript, notes, the date, and other information. To fine tune the styling, we used both vanilla CSS and Chakra.UI component props.

Challenges we ran into

• It was difficult to get started on the real-time transcription feature just by reading the docs alone because we had never used AssemblyAI APIs before and we don't understand how to use sockets too well. We then found comprehensive JS examples provided by AssemblyAI which we could follow.
• The example scripts are written in vanilla javascript, which we also weren't very familiar with as we mostly work with React. We had to look through a number of stack overflow posts to figure out how to translate this script into a custom React hook that can be called from any component.

Accomplishments that we're proud of

This is the first time we finished a hackathon project!

What we learned

There is a strong ecosystem of AI tools that provide amazing capabilities to developers who know little about AI.

What's next for WhatTeachersS(AI)

We want to double down on features that are specific to music lessons. We have two concrete plans: 1) Have a post-processing component that reads the transcript and creates a summary based on keywords like "etude", "tempo", etc. We haven't implemented this for now because it's very easy for the user to search for these keywords in the email. 2) In addition to the transcript, also store the audio recording and time-sync it with words or phrases in the transcript. One way that we might do this is by storing (timestamp, word) pairs as new words stream in from the real time transcription api.

Built With

• assemblyai
• chakra.ui
• css
• emailjs
• express.js
• figma
• github
• heroku
• javascript
• node.js
• react
• websockets