VibeProf

A handwriting-first note-taking app for iPad + Apple Pencil with a real-time AI tutor that doesn't just talk — it picks up the pen and writes on your work alongside you.

Inspiration

Plenty of AI tutors will chat with you, and a few will talk with you. But none of them actually draw. If you're working a math problem by hand, today's tools make you stop, photograph your page, and read a wall of text back. The teaching moment is lost.

We couldn't find a "vibecode"-style, real-time tutor that does what a good human tutor does: sit next to you, watch your handwriting as you write it, circle the sign error in red, and write the next step in the margin while explaining it out loud. So we built VibeProf, where the canvas — not a chat box — is the primary teaching surface.

What it does

VibeProf is a handwriting-first workspace for homework that works alongside you to solve math and physics problems. You write on a natural pen-and-paper canvas, and an AI tutor watches your work and helps directly on the page instead of in a separate chat box.

You can ask for help two ways: a one-shot Check that looks at your handwritten work and responds with feedback and annotations, or a live Voice mode that talks you through a problem in real time while it draws.

Rather than handing back a wall of text, the tutor marks up the canvas itself — underlining a mistake, sketching an arrow, or writing out the next line of a derivation in clean, readable math. It places its writing in the open space around your work and builds on the hints it has already given, so the page ends up reading like something a human tutor sat down and worked through with you.

How we built it

Frontend: React 19 + Vite + TypeScript. A custom pointer-driven <canvas> engine handles pressure strokes, highlighter, eraser, pan/zoom, and Apple Pencil input, with lucide-react for the UI.
Backend: a deliberately dependency-light Node HTTP/HTTPS server with a file-based JSON store for notebooks, pages, strokes, snapshots, and stroke timelines.
The tutor brain: the AI never touches pixels directly. It returns a validated list of high-level drawing commands that the client renders into real strokes — so the same command schema powers both the text "Check" path and the realtime voice tool (apply_canvas_commands).
Models: a multi-provider registry (Anthropic → OpenAI → Gemini → local mock) with graceful fallback. Voice runs on OpenAI Realtime over WebRTC, with Gemini Live as a backup transport.
On-device feel: local self-signed HTTPS + a cert-download helper so the iPad mic and WebRTC work over your LAN.

Challenges we ran into

Placement. Getting the tutor to write in empty space and not on top of the student's handwriting (or its own earlier hints) took a real spatial model — a free-space map plus AI-annotation bounds fed back into every prompt.
Speech/ink sync. A voice tutor that talks for ten seconds and then draws feels broken. We pushed the model to emit the smallest useful mark first, then narrate around it, so ink and audio land together.
"Erase only your own marks." Cleanly separating AI strokes from student strokes — and supporting character-level edits to the tutor's own writing — without ever clobbering the student's work.
Math that reads like math. Normalizing messy LaTeX into clean handwritten-looking Unicode, and rendering multi-step derivations as a tidy vertical chain.
iPad reality. Microphone + WebRTC on a LAN IP demand HTTPS, so we had to ship local certificate generation and trust as a first-class setup step.

Accomplishments that we're proud of

A tutor that underlines your mistake in red and tells you the fix in real time — the thing we couldn't find anywhere else.
Genuinely usable and real-time on an iPad with Apple Pencil, voice, and live ink — not just a desktop demo.
Resilient multi-provider fallback so a single down API doesn't kill the session.

What we learned

For an AI that shares a canvas with a human, spatial and temporal context matters as much as the prompt — where and when something was written changes the right response.
Latency is a teaching feature: small, early, incremental marks beat one big correct-but-late answer.
Constraining the model to a typed command vocabulary (instead of free-form output) made its behavior far more reliable and safe to render.