Margin AI

Your homework, graded in a snap and coached across the fold. Margin uses on-device Vision + OpenAI agents to find your mistakes and turn them into practice. On Duo, the fold is the margin.

Comment

Inspiration

As a student at Berkeley, I've watched AI change everything about school: how my classes are taught, how professors assign work, and how I study. But most AI tools just hand you the answer, which helps you finish homework but not actually learn. I wanted to build a study tool for this new world: one that checks your work, points to exactly where you went wrong, and coaches you through it instead of solving it for you.

When I saw iPhone Duo, the idea clicked. Teachers have always written feedback in the margin. On a foldable phone, the fold becomes the margin: your work on one side, your tutor on the other.

What it does

Margin turns iPhone Duo into an AI study desk, where every pose of the phone has a job:

  • 📱 Capture (closed): Scan a worksheet. On-device Vision detects the page, flattens it, and reads every line, then an AI model grades each problem with ✓/✗ badges.
  • 📖 Review (open like a book): Your worksheet on one side, your tutor on the other.
  • 💻 Coach (folded like a laptop): Work above the fold, tutor below. The tutor highlights the exact line you got wrong, writes a note in the margin next to the fold, and asks a guiding question instead of giving the answer. Then it quizzes you.
  • ✨ Practice: Margin writes your next worksheet: new problems targeting exactly what you missed.
  • 👥 Office Hours: For TAs and tutors. The inner screen shows talking points about the student's misconception, while the outer screen faces the student with their score and an encouraging nudge. Two people, two screens, one scan.
  • ⛺ Stand (tent pose): A StandBy-style focus mode with a Pomodoro timer and flashcards generated from your own mistakes.

Example: on the demo worksheet, the student wrote $u = x^2 \Rightarrow u' = x$. Margin highlights that exact line, notes that $\frac{d}{dx}x^2 = 2x$, and shows how it changes the final answer to $f'(x) = 2x\sin x + x^2\cos x$.

How I built it

App (SwiftUI, built with Bitrig):

  • ArrangementView in split and overlay styles to adapt across poses
  • reservedRegions(kind: .division) so highlights and margin notes never land in the fold
  • onHingeChange to wake the tutor when the phone folds into coach mode
  • matchedGeometryEffect so the worksheet carries across pose changes without reloading
  • Native toolbars that move to iPhone Duo's vertical bar automatically
  • CameraCaptureAccessory / sceneAccessory for the student-facing outer display in Office Hours, with a simulated preview in the simulator

Two-stage vision pipeline:

  1. On-device (Apple Vision): VNDetectDocumentSegmentationRequest finds the page, CIPerspectiveCorrection flattens it, and VNRecognizeTextRequest extracts every line with its bounding box.
  2. Cloud (OpenAI vision-language model): receives the image plus the OCR boxes and returns structured JSON: each problem, its status, the error, and which exact box to highlight. Grounding the model in real OCR coordinates makes highlights precise instead of guessed.

Agents (Python FastAPI + OpenAI Agents SDK):

  • Tutor agent with function tools (highlight, show_quiz, save_weak_spot, jump_to_page) that directly drive the UI
  • Scanner, Practice Writer, and Office Hours Coach agents with Pydantic structured outputs
  • Endpoints: /scan, /chat, /wake, /practice, /office_hours, exposed to the app through a Cloudflare Tunnel

Challenges I ran into

  • Building for a device that doesn't ship until October: everything was built in the iPhone Duo simulator and Bitrig's 3D fold simulator, with new APIs from the Xcode 27.1 beta that almost nobody has written about yet.
  • No camera in the simulator: the document scanner crashed, so I built a simulator path that runs the real Apple Vision pipeline on a sample image.
  • Imprecise highlights: the vision model alone guessed coordinates poorly. Feeding it on-device OCR bounding boxes fixed this.
  • Layout across poses: the overlay arrangement stacked views on top of each other when the phone was closed, so I had to design each pose deliberately rather than relying on one layout.
  • Backend debugging: a Python dependency conflict (an outdated brotli in Anaconda) broke API responses, plus request-validation errors between the app and the server.

Accomplishments that I'm proud of

  • Built the whole thing solo in one afternoon, from backend to agents to a native iOS app
  • Every iPhone Duo pose has a meaningful purpose, not just a resized layout
  • A full pipeline that goes from a paper worksheet to grading, coaching, a personalized practice set, and flashcards, all connected to the same scan

What I learned

  • How to design for a foldable: poses, reserved regions, and continuity between displays
  • Combining on-device ML with cloud VLMs, and why grounding matters
  • Building agentic systems where tools change the UI, not just the chat
  • Designing AI for learning means knowing when not to give the answer

What's next for Margin

  • Test on a real iPhone Duo when it ships October 23
  • Handwriting-first scanning and support for more subjects (chemistry, languages, CS)
  • A classroom mode where teachers see class-wide weak spots
  • Persistent progress tracking with Supabase and spaced repetition for flashcards

Built With

  • apple-vision
  • bitrig
  • cloudflare-tunnel
  • computer-vision
  • core-image
  • fastapi
  • gpt
  • ios
  • iphone-duo
  • ocr
  • openai
  • openai-agents-sdk
  • pydantic
  • python
  • swift
  • swiftui
  • uvicorn
  • vision-language-model
  • visionkit
  • xcode
Share this project:

Updates

Submission history