DrawExplain

We gave an AI a marker. Your screen is the whiteboard. DrawExplain teaches any slide, photo or PDF by drawing on it while it talks. Vision finds the parts, and code checks every number.

Comment

Inspiration

It's 2 a.m., a slide makes no sense, and a chatbot answers with a wall of text that never points at anything. A teacher would grab a marker and show you, right on the page.

Brainstorming

We wanted an AI that shows exactly how something is done, not just tells. First we thought of generating images or videos of the solution. But a generated picture isn't the page you're stuck on, and following it didn't feel intuitive. So we landed on drawing right on your own page, like a teacher at the board.

What it does

Upload a slide, photo or PDF and ask a question. DrawExplain teaches it step by step, circling parts and writing the working as the voice mentions them. On algorithm slides it really runs the algorithm, so every number on the board is exact. Circle something with the pen to ask a follow-up, or quiz yourself by tapping the image.

How we built it

We split the job so each part does what it is good at:

  1. See (computer vision, no AI): RapidOCR reads every word and OpenCV finds every shape. Each element becomes a numbered region. On an optional Modal GPU, SAM 2.1 sharpens unsure shapes and Qwen2-VL reads formulas.
  2. Plan (gpt-5.5): the model writes the lesson by naming regions, never pixel coordinates, and adds a rough box as a second opinion.
  3. Check (code): every drawing is cross-checked against the region it names. On algorithm pages the model only reads the graph or table, the pixels confirm every weight, and solvers compute every value.
  4. Teach: an ElevenLabs voice with word timestamps drives the strokes on an Excalidraw board.

Design process: four tries

  1. The AI drew on its own: 34% of drawings landed on target.
  2. Vision found the parts first: 83%.
  3. Prompting still skipped steps (0 of 3 fully correct Dijkstra lessons), so code took over the math: 9 of 9.
  4. Testing on 17 real pages exposed scheduling and paging mistakes, so we added solvers for both.

Challenges we ran into

  • The model's coordinates are almost right, and almost right misses the node.
  • A misread scheduling chart made the solver run the wrong algorithm. We now match the drawn order to the closest scheduler.

Accomplishments that we're proud of

We measured everything, on 407 targets and 20 real study pages. Our fact-check score rose from 0.86 to 0.93.

What we learned

Don't ask a language model to do what vision and code do better. Let it read and explain, and let the pixels and the solvers keep it honest.

What's next

Handwritten notes, solvers for more subjects (physics, chemistry), and a classroom mode for teachers.

Resources

OpenAI gpt-5.5, RapidOCR, OpenCV, SAM 2.1 and Qwen2-VL on Modal, ElevenLabs (voice; video music and effects), Excalidraw, React, FastAPI, Vercel and Render. Developed with Claude Code. Full credits

Built With

Share this project:

Updates

Submission history