Chalkboard

About Chalkboard

Inspiration

I came into this hackathon wanting to build a music teaching app, I'm a classically trained musician from Berklee College of Music, and teaching is something I care about deeply. But my teammate wasn't musical, and we realized a more general approach would be a better fit for the track.

That's when the idea clicked: what if we built a tutor you could talk to, one that doesn't just listen, but also sees what you're working on?

Most AI tutors are chatbots. You type, it responds. But that's not how real teaching works. A good teacher watches you write, catches your mistakes as they happen, and guides you through your own reasoning. We wanted to build that.

The Build

A few hours into the hackathon, my teammate was removed from the building, because he lost his wristband. So what started as a team project became a solo sprint.

The architecture has three layers:

1. Voice agent: handles the conversation, listens to the student think out loud
2. Canvas tools: tool calls that let the AI write on the board, erase, add diagrams
3. Vision context: continuously feeds the canvas state to the voice agent so it can "see" what the student is drawing

Challenges

The hardest part was giving the voice agent eyes.

The voice model handles conversation, but it needed to know what the student was writing in real-time. That meant:

• Serializing the canvas state and passing it as context
• Managing tool calls to tldraw (add, erase, annotate)
• Coordinating handoffs between the voice agent and the vision context without breaking the conversational flow

Dealing with all that state, and making sure the right context reached the right model at the right time, was the biggest technical challenge. When you're juggling a live voice conversation, a dynamic canvas, and multiple model contexts, things get messy fast.

What I Learned

Building this solo taught me a lot about scoping under pressure. I had to cut features I wanted (problem selector, AI cursor showing where Grok is "looking") and focus on the core loop: talk, draw, get guided.

But more than the technical stuff, this project reinforced something I've believed since my teaching days: thinking out loud is how we learn. We just never had a tutor that could keep up with that. Now we do.

Built With

• fastapi
• grok
• grokvoice
• next.js
• tldraw
• zustand