GIF
Logo Animation
GIF
graphing
GIF
latex conversion
GIF
ball drop physics visualization

Mimir: a tutor that watches, listens, and draws with you ✏️

Many students with learning disabilities struggle with traditional tutoring. Text walls don't click. In-person help is expensive or just not available. So we built a tutor that actually works like a real person - it watches you write, talks back, and draws right alongside you on the same canvas.

Inspiration 💡

Most AI tutors are chatbots with a math plugin. You paste a problem, get a paragraph back, and hope it connects to whatever you're actually stuck on. But not everyone learns by reading.

A real tutor sitting next to you watches your work, points at the mistake, and changes how they explain it when the first attempt doesn't land. We wanted to know why an AI couldn't do that.

That's Mimir. It sees your canvas, hears your voice, and answers by writing and highlighting right where you're working.

What it does 🎯

Mimir is an infinite canvas with a live AI tutor inside it.

You talk instead of typing, and Mimir answers out loud in real time.
You handwrite equations with a stylus. Mimir reads your strokes, so there's no translating math into typed syntax.
It watches your work and reasons about your steps as you go.
It highlights mistakes and points at specific terms without erasing anything you wrote.
Select a handwritten equation and it becomes a live graph. Every letter that isn't x turns into a slider you can drag.
Physics word problems turn into animations you can play and scrub. Watch the car slow down instead of reading about it.
Voice, captions, text, handwriting, larger text, one step at a time, reduced motion. Pick what works for you.

A student writes 2(x + 3) = 14 as 2x + 3 = 14. Mimir doesn't say "wrong." It highlights the 2(x + 3) and asks, "What does that 2 multiply?"

How we built it 🛠️

Multimodal tutor. Gemini Live over LiveKit. Voice, canvas, handwriting, and problem context all feed one tutoring loop. Mimir gets a fresh canvas snapshot when you pause writing, so it sees what changed.

Handwriting-first canvas. Pen, eraser, colors, undo/redo, pasted screenshots. Student strokes are sacred: Mimir annotates around your work, never over it.

Math verification. The model doesn't guess whether your algebra is right. Deterministic validators check that, which frees the AI to work on how to teach instead of whether the answer checks out.

Graphing. Recognized handwriting is normalized into a safe expression and sampled locally. No generated code runs. Coefficients become sliders, and selecting two equations puts both curves on shared axes.

Interactive visualizations. For physics problems, Mimir pulls out velocity, acceleration, and time, validates the units server side, then builds an animation you can play, pause, and rewatch.

The stack. Next.js, React, and TypeScript on the front end, with a custom infinite canvas. FastAPI and Python on the back end. LiveKit for realtime audio and video, Gemini Live as the tutor, Tesseract.js for screenshot OCR, KaTeX and SVG strokes for math rendering, and Docker Compose to run it all together.

Challenges ⚡

Making it feel real time. A tutor that answers five seconds after you've moved on is useless. We wait for a pen pause, grab the latest canvas state, then send it. Responsive without interrupting you mid-thought.

Knowing when a student is actually wrong. A half-finished equation and a wrong equation look identical. We combined model reasoning with deterministic math checks and treated timing as a soft signal instead of a trigger.

Letting the AI draw without taking over. If Mimir fills in every step, it's doing your homework. We split guidance from answering, so it gets highlights, questions, and visual hints, but never rewrites your work.

Connecting speech to space. When Mimir says "look at this term," it has to know what "this" is. We built canvas tools that let the tutor point at a specific piece of the student's handwriting while it talks.

Accessibility from the start. We didn't want a separate accessible version. Voice becomes captions, text becomes speech, an equation becomes an animation, and a wall of explanation becomes one step at a time. Most of those choices ended up helping everyone who tried it.

What we learned 📚

Multimodal AI gets much more useful once it can act on your workspace instead of living in a chat box
Handwriting is messy, ambiguous, and far harder to reason about than we expected
Tutoring is mostly about when to help and how much, not about getting the right answer
Pairing deterministic validation with a generative model works well for math, where being confidently wrong is the worst failure