Chalk

Inspiration

The idea started with a tool I saw online: someone drawing in the air over their webcam during meetings. Clever, but the ink was dumb. The human still did all the drawing, and the drawing still disappeared. I kept thinking about tutoring calls and remote classrooms, where you end up explaining something twice because you cannot talk and draw at the same time. So I flipped the roles. Let the AI draw the structure while the human explains, and let the hands do what hands are good at: pointing.

What it does

Chalk is a live teaching board. A teacher just talks, and GPT-5.6 listens and draws the concept map in real time: nodes, labeled edges, layout, even corrections when you say "actually, scratch that." Through the webcam, the teacher can point at nodes and pinch to select them, then say "connect these two," and the model resolves "these" from what is selected. When the lesson ends, students get a replay that rebuilds the diagram step by step, plus an AI-written handout with a recap, a glossary, and three comprehension questions grounded in what was actually said. Nothing on the board is invented. Speech creates, gestures steer.

How I built it

I wrote a full spec before the first line of code and committed it, so the git history literally starts with SPEC.md. Then Codex and I built the product in six phases, each with acceptance criteria I tested myself before approving the next. Every significant choice went into a running decision log, tagged as either a Codex decision or a human decision. That log became the README section on our collaboration.

The core is a small contract: GPT-5.6 on the Responses API returns strict structured output, a list of diagram operations such as add_node, add_edge, set_layout, highlight and no_op. The client applies each batch atomically with animation, which also gives me undo and replay for free, since the ops log is the history. Speech comes from the Web Speech API behind a swappable interface, hand tracking is MediaPipe running fully in the browser, and the canvas is React Flow with elkjs layout. A second GPT-5.6 call writes the class handout. Even the demo video was assembled with Codex, using Remotion.

Challenges I ran into

Structured outputs rejected my schema at runtime because the API requires every field, so optional fields had to become nullable and get normalized back at the boundary. Gestures worked until I zoomed the canvas, then fell apart: fingertip coordinates needed a full pipeline from mirrored video space through screen space into React Flow space, plus hysteresis so one pinch means exactly one selection. And speech recognition taught me humility. It once turned "scratch that" into "struck scratch that transpot," which is how I learned to test the reasoning layer with typed input, so I always knew whether a failure belonged to the model or to the microphone.

Accomplishments that I'm proud of

The deixis moment works end to end: pinch two nodes, say "connect these two," and the edge lands on exactly those node ids. I am also proud of what I did not build. No fake meeting UI, no pretend participants, just a presenter tile with a real camera feed and a board that genuinely does the thing.

What I learned

An agent is a great engineer and a poor product owner. Codex moved fast when my spec was explicit and the acceptance criteria were testable, and it drifted the moment I got vague. Writing decisions down as they happened turned out to be the cheapest documentation I have ever produced. And optionality at API boundaries should be modeled as null, not absence.

What's next for Chalk

A read-only live link so students can follow the board on their phones during class, an OpenAI transcription source for better accuracy than the browser recognizer, and teacher-editable handout templates. I also want to try lecture-length sessions with rolling summarization, so an hour-long class stays coherent on one board.

Built With

• codex
• elkjs
• gpt-5.6
• mediapipe
• next-js
• openai-responses-api
• react
• react-flow
• remotion
• tailwindcss
• typescript
• vercel
• web-speech-api
• zod
• zustand