Chalk: Whiteboard Tutor

main ui
tangential whiteboard
debugging

Inspiration

I love learning. AI has greatly accelerated this project for myself and many students worldwide regardless of their background. A whiteboard is the bread and butter of classroom teaching. I wondered why not bring that to AI? After all I think students would learn best from a format they are used to.

What it does

CHALK is a live voice tutor for maths and physics that teaches on an whiteboard.

A student asks for a topic, and CHALK generates a structured lesson, narrates it aloud, and draws the supporting equations, graphs, and diagrams as it explains. The student can interrupt with the space bar or by clicking the voice orb, ask a question about the board, and resume without losing the lesson state.

The board is not just a visual output: it tracks what has actually been drawn so the tutor can ground follow-up explanations in the visible diagram.

GPT 5.6 Luna creates the structured lesson plan while GPT Realtime 2.1 handles the live interaction between the interface and the student. My custom built "drawer" intermediary acts as an orchestrator between both the models and the whiteboard surface, providing a platform for Luna to "draw".

How we built it

I used GPT 5.6 Sol on Codex to bring the project to life. After creating the required markdown files with instructions, goals and my vision for the project, Codex began work and we iteratively honed the project to its current state, 700 million or so tokens later.

We built CHALK with a React/TypeScript frontend and a FastAPI backend. The frontend handles WebRTC voice, interruption, whiteboard animation, local rendering, and the interactive lesson state machine. The backend generates and validates lesson steps, repairs invalid output, and keeps API credentials server-side.

Rather than allowing a model to emit arbitrary drawing code, CHALK uses a constrained lesson DSL for text, equations, curves, arrows, constructions, and diagrams. Every generated step is schema-validated, checked for safe mathematical expressions and references, laid out deterministically, and rendered only after it passes those checks.

Challenges we ran into

The hard part was making a generated lesson feel like a teacher drawing rather than a slideshow appearing on screen.

We had to solve several reliability problems: malformed model output, equations and labels colliding, diagrams that were too sparse or cramped, audio events arriving out of order, interrupted narration, and keeping the animated ink synchronized with speech. We addressed these with a validated drawing language, deterministic layout and collision checks, bounded repair paths, prefix-stable rendering, and a state machine that freezes the board immediately when the student speaks.

We also learned that “the model knows what it intended to draw” is not enough. The tutor needs structured context from the board that was actually rendered. The interplay of models -- GPT Realtime powering live speech,

Accomplishments that we're proud of

This is an incredibly token efficient program! (thanks, @OpenAI)
Works on a large variety of cases it has not seen

A tutor can be interrupted mid-sentence and mid-stroke without losing the lesson.
The whiteboard is generated through a safe, constrained drawing pipeline rather than raw model-produced code.
The renderer isolates failures: one bad visual operation does not crash an entire lesson.
Lessons build progressively, preserving already committed ink.
The tutor can use the board’s visible-element context for grounded follow-up questions.
We built a reliable cached-demo path alongside live generation so the demo remains dependable.
The project has extensive deterministic tests across rendering, validation, synchronization, and interruption behavior.

What we learned

Personally I learnt that almost anything is possible with Codex. In this day and age, the bottleneck is no longer knowing the technical details; the actual limitation is determined by how high you can think.

As for the project: Voice, animation, and generation are three different timelines. A system can be technically “done generating” while audio is still playing, and an interruption can arrive at any point between them. Treating those as separate observable states made the product much more reliable.

We also learned that better AI experiences are often less about giving a model more freedom and more about creating a narrow, expressive interface where its ideas can be checked, rendered, and recovered safely.

What's next for Chalk -- Whiteboard Tutor

Next, we want to improve the visual vocabulary with richer geometric relationships, more simulation-backed interactive figures, and stronger diagram-aware follow-up teaching.
I also want to add in a stronger student-chalk dynamic. My next goal post this hackathon is to make the student able to draw into the board and have the AI react to it.