Plasticity

Inspiration

Universities often require students to learn independently. Many students use AI to prepare for upcoming exams, but think about the last time you truly understood a complex concept. For us, it was when we sat down with a piece of paper and worked through the entire problem, writing down every intermediate step.

However, most AI tutors exist in a separate chat box. This creates a gap: the student has to describe their handwritten work, while the tutor cannot naturally point to the exact equation, diagram, transition, or mistake.

Plasticity was inspired by the idea that the notebook itself should become the conversation.

What it does

Plasticity is a visual AI tutor for iPad. Students can import a PDF or image, write with Apple Pencil, and ask questions by voice or text. The app studies the visible page, understands the learner’s handwritten work, and responds with explanations plus erasable teaching marks drawn directly on the notebook.

It supports multi-page PDFs, blank pages, pasted images, zoom-aware tutoring, push-to-talk questions, typed chat, AI text-to-speech, check-my-work flows, and structured visual annotations like highlights, arrows, labels, diagrams, and automata.

How we built it

Plasticity was built with substantial help from Codex throughout development. We used Codex to iterate on the PencilKit canvas, debug coordinate-mapping and image-manipulation issues, implement the AI annotation toolbar, review changes, test signed builds on an iPad, and prepare reliable fixes for deployment. It is built in SwiftUI with PencilKit for handwriting, PDFKit for document import, Apple Speech Recognition for push-to-talk, and the OpenAI Responses API for visual tutoring. The app sends the current page or visible crop to GPT-5.6 Sol, along with the learner’s question and recent context.

A key design choice was separating meaning from rendering: the model returns structured annotation data, while the app validates it and renders the final marks locally. For automata and structured diagrams, the model declares states, transitions, or layout intent, and Plasticity computes the geometry deterministically.

Challenges we ran into

One of the hardest parts was making AI feedback feel grounded, editable, and visually reliable. We had to handle coordinate mapping between zoomed-in views and full pages, prevent malformed annotations from breaking the entire response, keep AI-generated ink separate from the learner’s ink, and ensure that arrows, labels, paths, and diagrams appeared in the correct places.

Another challenge was matching the AI’s spoken explanation to the drawing. Because the AI cannot draw directly, it sends instructions describing what to draw and where to place it. We therefore had to develop a renderer that could add ink to the page while ensuring that the appearance and timing of the drawing matched the transcript of the AI’s explanation.

Accomplishments that we're proud of

We’re proud that Plasticity feels like a tutor working on the learner’s actual page instead of a chatbot beside it. The app can visually explain ideas, check corrections, draw reversible teaching marks, and keep the student’s notebook as the center of the learning loop.

We’re also proud of the deterministic renderer, automaton layout system, semantic pointer cues for spoken explanations, pasted-image support, and the test coverage around annotation safety, arrow resolution, SVG clamping, layout, page deletion, and rendering behavior.

What we learned

We learned that visual AI tutoring works best when the model does not directly control pixels. Letting the model describe intent while the app owns layout, validation, and rendering made the experience more reliable and easier to debug.

We also learned that good tutoring is more than giving an answer. The prompt and UI both needed to support diagnosis, explanation, visual grounding, and a next action for the learner.

What's next for Plasticity

Plasticity’s core idea is that the AI is not just answering in text, it is deciding what to draw, where to draw it, and how that drawing should help the learner understand the concept. That makes the system inherently more open-ended than a normal tutoring app. A small misalignment, an unclear arrow, or a label placed in the wrong spot can quickly become a visual mess. Our goal with Plasticity is to develop a framework that gives AI a reliable drawing capability: a way to translate teaching intent into clear, editable, page-aware visual marks. We want the AI to be expressive enough to teach visually, while still giving the app enough structure to prevent messy or confusing output. We also want to improve the connection between speech and drawing. Instead of showing every annotation at once, Plasticity should let the AI decide when each mark appears during the explanation, so the learner sees the drawing unfold in sync with the spoken reasoning, the way a human tutor would sketch while explaining.