Meet Pip: it quotes your own line back to you and asks one question, so you find the fix yourself.
Mistakes stay, crossed-out work is kept, and when two reads disagree, confidence drops instead of guessing.
One desk for your notebook, the pages you read and your tutor, plus the full pipeline from camera to Pip.
How it works: snap your page, every line is read as written, Pip asks a question, you fix it on paper and it turns green.
414 tests passing, and one real page read and judged in 8.1 s. Moving few-shot examples to text cut calls from 8-35 s to 2-3 s.

Inspiration

It's 11 PM. There's a page of calculus in front of you, and you know something on it is wrong. Office hours ended this afternoon, so you snap a photo and ask an AI. Ten seconds later you have the right answer, and you still don't know which of your lines was wrong.

AI homework tools are built to produce answers. But learning happens at the exact line where your reasoning broke, and an answer skips right past it. Worse, a model reading your handwriting is free to read it the way it should be. It can quietly turn your

$$\frac{d}{dx}(3x^2+1)^4 = 4(3x^2+1)^3$$

into the correct

$$\frac{d}{dx}(3x^2+1)^4 = 4(3x^2+1)^3 \cdot 6x$$

and the mistake you needed to see is gone. A reader that's allowed to fix your work can never show you where you went wrong.

We wanted the opposite: a study partner that's honest about what you actually wrote, and that makes you find the fix yourself.

What it does

Put your notebook under any camera (your Mac's built-in Desk View, an iPhone, a webcam) and press Analyze.

It reads every line exactly as written. StudBud finds the page, flattens it, boxes each handwritten line, and transcribes it to LaTeX verbatim. Mistakes, crossed-out work and half-finished steps are kept, never corrected.
It's honest about uncertainty. Every line carries a confidence score. When two independent readings of a line disagree, confidence drops and the line is marked unclear instead of guessed.
Pip points to your line, not to an answer. Pip, a small pixel-art tutor, sees only the verbatim page. It quotes your own line back and asks a guiding question. On our chain-rule demo page, it asked: "Since the inside is $3x^2+1$ and not just $x$, what do you need to multiply by after differentiating the outside?"
It knows what you were reading. An optional Chrome extension remembers the pages you choose to share (titles and URLs only). In the same demo, Pip noticed the student had just been reading about the product rule and pointed out that this problem needs the chain rule.
You can just ask out loud. Tap Mic, talk, pause, and it sends. Speech-to-text runs locally on your laptop, and audio is never saved.
You fix it on paper. Write the corrected line, analyze again, and the new line turns green.

How we built it

Vision pipeline (Python, OpenCV, Claude)

Capture. A stability trigger waits until your hand leaves the page. Sources include a webcam, an iPhone, an uploaded photo, or macOS Desk View. Desk View is invisible to OpenCV, so we stream it through a single long-lived ffmpeg AVFoundation process.
Page detection. Quad search over edges and thresholds with sub-pixel corner refinement, an ArUco fallback, homography dewarp and lighting flattening. A quality gate flags blur, glare, hands in frame and partial pages.
Layout. Binarization, printed-rule removal, fraction fusion (so a denominator doesn't become its own line), neighbor-link line grouping and reading order.
Crossed-out detection. Stroke analysis scores how many glyphs a horizontal stroke crosses, tuned so fraction bars and underlines don't trigger it.
Verbatim transcription with Claude. Every line is read twice, independently: once in a whole-page pass and once from its own crop. If the two normalized readings disagree, the line is re-read and its confidence drops. A verbatim guard logs every case where one reading was mathematically "nicer" than what was written, so we can audit that the corrected version never wins.
Page twin. The output is a JSON page twin: stable line IDs, boxes, LaTeX, crossed-out status, confidence, and both raw readings.

Tutor layer (Claude)

Pip receives the verbatim twin plus whatever reading history the student chose to share. Step labels are copied from the twin, and the model never rewrites them. Each step is judged correct, error or unclear, and every reply cites its sources: the notebook snapshot and the page you were reading. Text inside a photo is treated as evidence, never as instructions.

App

A FastAPI bridge serves both the vision pipeline and the dashboard API from one process. The frontend is React 19, Vite, Tailwind and KaTeX. A recognition view overlays each line box on the flattened page, and an inspector shows each line's crop alongside both readings. Voice questions are transcribed locally with faster-whisper, and a Chrome MV3 extension provides reading memory.

How we worked

We split the system into six modules with frozen interface contracts (capture, layout, transcription, app and eval, bridge, UI) and built them in parallel with six Claude Code agents, with one human integrator relaying messages between them. The repo has more than 340 Python tests and 56 JavaScript tests, plus an evaluation harness with a labeled test set.

Challenges we ran into

Latency was hiding in our own prompts. A single model round trip took 8 to 35 seconds. The culprit was eight few-shot example images resent on every call. Moving the examples into text cut each call to about 2 to 3 seconds. On the chain-rule demo page, the full read now takes about 4 seconds, and the whole analyze request about 8.
Fractions and strikes look alike. A fraction bar, an underline and a strike-through are all horizontal lines. We fuse stacked fractions before grouping lines, and score strikes by how many glyphs they cross.
The camera fought back. macOS Desk View only exposes itself to AVFoundation, serves one client at a time, and stays black after a reader crashes. We built a single shared ffmpeg stream with warm-up frame skipping and idle shutdown.
Honesty versus helpfulness. It's tempting to let the model clean up an ambiguous symbol, or to let the tutor just give the answer. We chose to surface uncertainty and make the tutor ask instead. Keeping replies short and question-first is still the hardest part to get right.

Accomplishments that we're proud of

A complete loop from camera to tutor on an ordinary laptop, with no extra hardware.
On our demo pages, every planted mistake survived transcription verbatim, and the crossed-out line was recognized and kept out of grading.
When the two readings of the problem statement disagreed on a single prime mark, StudBud lowered its confidence to 0.78 and said so, instead of pretending to be sure.

What we learned

Accuracy alone isn't the goal. For learning, the most valuable output is a faithful copy of what the student actually did, wrong parts included. We also learned that frozen contracts let parallel agents move fast without breaking each other's work.

What's next

Test on a larger set of real student handwriting and publish the numbers.
Hold Pip to strictly one short question per reply.
Expand to more subjects: chemistry equations and physics free-body diagrams.
Build a classroom view that shows teachers which line most of the class gets stuck on.