About the project

Inspiration

The best tutoring rarely feels like submitting a prompt and waiting for a wall of text. It feels like sitting beside a brilliant friend while they work through a problem in a notebook—and being able to stop them at the exact moment a step stops making sense.

That experience inspired psst.

Most AI tutors know what a student asked, but not where their understanding broke. A pause after a sign change means something different from a pause when a theorem is introduced. A circle around one symbol carries more context than another generic “Can you explain that?” message ever could. We saw an opportunity to turn those tiny, natural interruptions into a completely new learning signal.

Instead of treating interruption as friction, psst treats it as the heart of tutoring. Our goal was ambitious: make an AI explanation feel alive, interruptible, spatially aware, and genuinely responsive—then use those interactions to build a map of confusion at the level of individual reasoning steps.

What it does

psst is an interruptible, spatially aware AI tutor. It turns a math problem into a narrated notebook lesson that unfolds step by step at a human pace.

While the solution is being written, a student can tap the page or press Space to freeze it at that exact point. They can circle or underline the expression that confused them and ask a question. psst maps the handwritten mark back to the underlying mathematical step, gives the model only the work visible so far, and places a concise answer in the margin with a connector pointing back to the selected expression. The tutor explains the moment of confusion without spoiling later steps.

Students can also interrupt by voice. psst opens a low-latency Realtime conversation, transcribes the question, answers aloud, and anchors the response beside the relevant work. “What if?” questions are recognized as alternative paths and receive a focused explanation of what stays the same, what changes, and how the variation unfolds.

Every interruption becomes a structured event: the problem, exact playback position, targeted step, annotation bounds, question, input method, and timestamp. A privacy-safe teacher dashboard aggregates those events without exposing student identities, questions, or drawings. Over time, this can reveal that students do not merely struggle with “quadratics”—they repeatedly lose the thread at one precise transformation inside one explanation.

Students can choose from generated curriculum variants, submit their own problem, replay a lesson, continue to a recommended exercise, and export the finished notebook—including their annotations and the tutor’s margin notes—as portable study material.

How we built it

We built psst as a full-stack Next.js, React, and TypeScript application. The notebook is an SVG-based spatial scene: every displayed token has a semantic ID, step ID, bounding box, and timed reveal. KaTeX renders mathematical notation, while a custom playback state machine controls writing, pausing, resuming, and narration timing.

For lesson creation, we use the OpenAI Responses API with structured outputs validated by Zod. The model returns a schema-backed sequence of mathematical steps, explanatory notes, concept tags, pacing hints, and optional diagrams. That structure lets our renderer treat a derivation as an interactive scene rather than an opaque block of generated prose.

The spatial interaction system is custom-built. Pointer coordinates are transformed into notebook coordinates; freehand strokes become bounding boxes; and overlap, proximity, and semantic geometry are used to resolve the expression a learner intended to select. A collision-aware placement engine then finds room for the answer and draws a visual connector back to its source.

Voice tutoring uses the OpenAI Realtime API over WebRTC for low-latency speech interaction, plus OpenAI speech models for transcription and warm step narration. The browser receives an ephemeral Realtime connection rather than our standard API key. We synchronize the visual lesson with narration so the writing never races ahead of the explanation.

Interrupt events are schema-validated and stored in SQLite. We designed the write path to be idempotent: the initial pause is logged immediately, then the same event is enriched if the student adds a drawing or question. Analytics expose aggregates only. We also built deterministic curriculum generation, local completion history, structured notebook export, and a substantial Vitest suite around playback, spatial targeting, session state, model schemas, persistence, and API behavior.

Challenges we ran into

The hardest problem was not generating an answer; it was preserving the feeling of a live tutor while several systems moved at once. Animation, narration, touch input, microphone state, network calls, and React rendering all had to agree on the exact moment the student interrupted. We built explicit state transitions and aggressively stopped in-flight narration so the tutor could not keep talking over the learner.

Spatial intent was another major challenge. Human marks are messy: a circle may overlap several tokens, an underline may barely touch its target, and rendered LaTeX does not naturally expose meaning. We combined scene metadata with geometric heuristics and extensive edge-case tests to make pointing feel natural rather than brittle.

We also had to constrain the model in ways that improved the product. Generated equations needed to fit a notebook column, voice answers needed to be complete but short, and contextual answers could not reveal work the learner had not seen. Structured outputs, visible-context boundaries, and strict response schemas turned model behavior into something the interface could reliably compose.

Finally, we took privacy seriously. Confusion data is powerful precisely because it is sensitive. We separated the detailed event store from the aggregate teacher view and made sure analytics never display raw questions, drawings, session IDs, or student IDs.

Accomplishments that we're proud of

We are proud that psst is not a chatbot wearing a notebook skin. It introduces a different interaction primitive for AI learning: point at the reasoning, interrupt it in time, and continue from shared context.

We built the entire loop end to end. A learner can watch a fresh problem become a paced audiovisual lesson, stop it mid-explanation, mark a specific expression, receive a grounded answer exactly where they pointed, ask a follow-up by voice, resume without losing their place, and leave with a coherent notebook artifact.

We are especially excited about the dataset this creates. Traditional platforms can record whether an answer was wrong or whether a video was paused. psst can capture where the reasoning failed at sub-step resolution. In rough terms, our signal is not just

$$ \text{struggled} \in {0,1}, $$

but

$$ \text{confusion} = (\text{problem},\ \text{step},\ \text{symbol},\ \text{time},\ \text{question}). $$

That is a meaningfully richer foundation for adaptive education—and, we believe, one of the most novel ideas in this hackathon.

What we learned

We learned that the best AI interfaces are often built by giving the model more structure, not more freedom. Once each mathematical step had semantics, timing, and geometry, we could create interactions that plain chat cannot support.

We also learned that an interruption is not a failure of the lesson. It is high-quality feedback. The exact place a learner says “wait” can teach the system more than a quiz at the end, because it captures uncertainty while it is happening.

Most importantly, we learned how much orchestration sits behind an experience that feels simple. When voice, ink, math layout, model context, analytics, and animation disappear into one natural gesture—pointing at a line and asking “why?”—the technology starts to feel like tutoring instead of software.

What's next for psst

Next, we want to turn individual interrupt events into a true confusion graph across learners and curricula. That means clustering semantically similar questions, visualizing heatmaps directly on derivations, and helping educators identify explanations that consistently fail at the same step.

We also plan to make adaptation happen during the lesson: automatically slow down near historically difficult transitions, insert a bridging example before confusion occurs, and recommend the next problem based on the concepts a specific student interrupted most—not merely whether they reached the correct answer.

Beyond competition math, the same interaction model can transform physics derivations, chemistry mechanisms, code walkthroughs, proofs, and any domain where understanding unfolds spatially and sequentially. Long term, psst can become more than an AI tutor. It can become the missing feedback layer between how experts explain and where learners actually lose the thread.