Scribe

Inspiration

Most AI tutors live in a chat box, so you're always switching between your notes and a wall of text. On paper, a good tutor leans over and writes in the margin, circles the mistake and sketches the diagram. I wanted that on an iPad.

What it does

Scribe is an iPad notebook with an AI tutor that works on the page. You write with the Pencil or import a PDF, then ask by voice, by typing, or by lassoing a region. The tutor answers in handwriting, right where it's needed:

• Writes and annotates: notes in handwriting, underlines, circles, ticks and cross-outs
• Traces figures: maps and diagrams are split into parts (each country, each region), so it can outline exactly the one you asked about
• Draws: small diagrams, and it grows the page when it runs out of room
• Visualizes: opens small interactive visualizations with controls you can play with
• Talks: streaming speech to text, and spoken answers with the words appearing live as they're read

How we built it

• SwiftUI, PencilKit and PDFKit for the notebook
• A page map: each turn, Vision and PDFKit turn the page into objects with stable IDs: printed text down to the word, blanks, tables, figures and their parts, the student's ink, and free space
• Claude gets a picture of the page plus that map, and calls tools such as write, circle, outline_region and draw. Every call is checked against the map before anything lands, so notes never cover ink or printed text, and answer blanks stay clear.
• ElevenLabs realtime speech to text, and text to speech with character timestamps to sync the words on screen
• Sandboxed WebKit for the visualizations the agent writes

Challenges we ran into

• Placement: an LLM can't place ink by pixel coordinates. The page map and validation layer let it point at things by ID instead.
• Splitting figures into traceable parts from the page's pixels, including cutting thin necks like an isthmus
• Syncing streamed words with the voice, and keeping the UI responsive while audio and network run

Accomplishments that we're proud of

• The tutor feels like it's on the page, not beside it. Answers land in handwriting, next to the work they're about, without covering anything.
• It can point at exactly the right thing, like a single word, a table cell or one country on a map, because every object on the page has an ID.
• Voice that feels live: your words stream in as you speak, and the answer is written on screen in sync with the voice.
• A polished, native iPad app with Liquid Glass controls, folders, undo, and light and dark icons, built over a hackathon weekend.

What we learned

Grounding beats prompting. Giving the model a structured map of the page, and checking its actions against that map, made it far more reliable than any prompt tweak.

What's next for Scribe

Multi-page reasoning, handwriting recognition for full worked solutions, and sharing notes.

Built With

• anthropic
• claude
• elevenlabs
• ipados
• pdfkit
• pencilkit
• swift
• swiftui
• vision
• webkit