NoteNinja

Inspiration

I often find myself struggling to keep up with note-taking in a lecture, only to then realize that I just missed an important point. I wish I could just sit back and immerse myself in the lecture instead. And that's why I decided to automate the note taking process.

What it does

NoteNinja uses your microphone to automatically transcribe your lectures; so you spend less time writing notes and more time engaging in the classroom. Imagine sitting in class fully focused on the discussion, while note Ninja automatically captures every word spoken. No more note-taking and instead you can actively participate, ask questions, and truly absorb the knowledge being shared.

You can simultaneously edit the transcription and even add photos. You can share your notes with your classmates and collaborate on the perfect lecture notes in real time (including simultaneous live transcription).

How I built it

Using AssemblyAI's realtime audio-to-text recognition, the lecture's content is added to a text editor in real time. The audio recording is done via the Web MediaStream API, which is then streamed via WebSockets to AssemblyAI, which uses transformer models to turn the recordings into text (and stream that back to the client via WebSockets).

The UI is responsive and fast thanks to a Single Page Application built via React and Next.js. The text editor is built via Slate and dnd kit. The collaboration features are enabled via Liveblocks (which uses WebSockets in the background).

The demo is hosted on Vercel.

Challenges I ran into

It took me a while before I found a suitable audio-to-text service that offered low latency and good results. Setting up the live transcription pipeline too much more time and lines of code than I expected. Making a text editor that is edited by a user and an AI at the same time was hard! (loss of cursor focus, flickering, etc.)

Accomplishments that I'm proud of

I am proud to have built and shipped a working MVP of what I had in mind. The MVP is usable enough that I look forward to using it next week for my own and I'm excited to take the project forward

What I learned

This is the first time I used Liveblocks and I really loved how easy it was to build a multiplayer app with it. I also learned a lot about Slite (a text editor for React), which was the first time I dealt with a text editor on such a low level and feel comfortable building apps that feature a custom text editor now! Overall, I feel like I improved my React, Typescript skills and have deepened my understanding of web technologies (especially Websockets for the realtime audio transcription).

This was the first hackathon where I was solo; I have been to numerous hackathons before but always in a team of 4+. It was surprising to me how different this experience was, it felt a lot harder to keep pushing on (and staying awake in the past midnight hours, compared to when in a team). I recorded myself the entire time and have learned a few things about how I work, get distracted, etc. by watching the recording.

What's next for NoteNinja

I received positive feedback after showing NoteNinja to a few friends; so I'm looking forward to productizing the project (adding auth, a database, etc.) and evaluating if it can be turned into a startup.

Built With

• assemblyai
• liveblocks
• next.js
• node.js
• react
• typescript