AutoNote

Example output note
Actual note from video

Inspiration

Originally we wanted to generate handwritten notes from text, but that still requires someone to provide the text content! Instead, AssemblyAI returns the text from a recording which we then generate handwritten notes from using pnshiralkar's text-to-handwriting project.

Now we have a fully automated method to create written notes from lectures, virtual or in-person. Pretend you were actually there in class, or pretend to be a better student than any of us are!

What it does

The script first records audio from the system, once you stop the recording it gets uploaded to AssemblyAI for transcription and identification, and the result is then converted to handwriting by the neural network and rendered onto a fake paper document.

Since the process for handwriting takes a bit of time to generate, we integrated Twilio to let us know when it's done and our notes are ready.

How we built it

Our code patches together multiple different existing tools and services, grabbing audio from the system, transcribing audio with AssemblyAI, generating handwritten text, and finally results in a PDF output file and a notification to our phone with Twilio.

Our local code is entirely built in Python, handwriting generation is handled in TensorFlow, recording audio is performed with the pyaudio portaudio library. The AssemblyAI API is accessed directly with python's requests library. Twilio's client library is used to send the SMS.

Challenges we ran into

Originally our plan was to generate our own handwriting samples and train the network, however the amount of time required to tag our custom data and train a model was too great for the time we had, so we had to use a pre-trained model to generate handwriting from text.

Accomplishments that we're proud of

The result is surprisingly accurate, and useful. We might actually use this for real note taking during class, or to make it look like we were paying attention closely during a meeting.

What we learned

Lots of small components to a solution exist, but there are plenty of unexplored possible combinations of tools and services.

The handwriting generation AI was extremely interesting to explore, and we realize how much effort goes into the initial dataset required to produce a decent result.