Recall

Inspiration

Every one of us has been there. You're in a lecture, following along. Then you glance at a message, or your mind wanders for a minute, and when you tune back in the professor is on a new idea, there's an equation on the board you've never seen, and the slide has already changed. You spend the rest of the class trying to rebuild what you missed instead of learning what's in front of you.

We all had our own version of this. One of us would quietly ask a neighbour "wait, what did they just say?", another would make a note to rewatch the recording later (and never did), and another would just give up on that lecture and hope the textbook covered it. When we compared stories, we realised this wasn't a personal focus problem. It's a design problem. A lecture is a stream you can't pause, and once you lose the thread there's no easy way back in.

At a typical speaking rate of about $150$ words per minute, drifting off for just $\Delta t = 2$ minutes means missing roughly

$$ 150 \times 2 = 300 \text{ words} $$

That's often a whole definition, a worked example, or the key step of a derivation. We wanted to build something that gives you those 300 words back in seconds. That's Recall.

What it does

Recall listens along with the lecture and turns it into a live timeline of topics as it happens. Each topic card holds:

a short summary of what was said, plus the original transcript so you can read the source,
equations rendered properly with KaTeX, along with the intuition behind them,
the actual slide the professor was on, matched to the exact page of the lecture PDF,
a contextual chat, so you can ask "what did I miss?" and get an answer that cites the specific moments it came from,
visuals on demand: ask for a chart or a step-by-step animated diagram of a topic, and it's attached to that moment.

Students can mark a topic as Unclear, and instructors and TAs see those doubts grouped by lecture and topic, so they know exactly where the class got lost. After class, the same timelines become a study tool, connected to the course readings and a knowledge graph that links concepts across lectures.

How we built it

Frontend: React 19 + Vite. The timeline is a stack of cards drawn in a requestAnimationFrame loop, so you can scroll back through earlier topics smoothly while new ones keep arriving at the live edge.
Backend: A TypeScript Fastify API that replays lecture transcripts as a live stream and pushes topics to the browser over Server-Sent Events.
The AI pipeline: Incoming transcript is grouped into chunks (by default $w = 160$ words, released every $8$ seconds in simulation). A model-backed segmentation step decides whether each chunk continues the current topic or starts a new one. Then an enrichment step writes the summary, extracts equations, finds the matching slide and keeps a running lecture summary. We used DeepSeek behind a provider-neutral interface, so models can be swapped.
Shared contracts: Every model output is validated with zod schemas shared by the client and server, so a malformed response is caught instead of rendered.
Slides: Cited slide pages are rendered straight from the lecture PDFs in the browser with pdf.js. Each deck is fetched once, and each page is cached as an image.
Knowledge graph: MongoDB stores concepts and their links across lectures, so a topic in lecture 3 can point back to where the idea first appeared.
Data: We built and tested on a real course, a 23-lecture AI Alignment course with slide PDFs, readings and transcripts.
Testing: A deterministic fixture provider lets the whole app run offline without paid API calls. That powers our unit tests and a Playwright end-to-end suite on desktop and mobile.

We split the work into three parallel workstreams: live lecture, readings, and the course-wide knowledge graph. Each had its own guide and shared contracts, so we could build at the same time without stepping on each other.

Challenges we faced

Live without the chaos. Topics arrive while you're reading. If the stream reconnects or the page reloads, you must not get duplicate or reordered topics. And if you've expanded an older topic, new arrivals must not steal your place or your scroll position. We wrote end-to-end tests for exactly this, because it's the whole point of the product: you shouldn't lose the thread again.
Deciding where a topic ends. Lecturers circle back, digress, and answer questions mid-thought. Getting segmentation to produce topics a student would recognise, rather than arbitrary slices of text, took a lot of iteration on prompts and evaluation.
Finding the right slide. Matching what was said to the right page of a 60-page deck is hard, especially when several slides share similar words. We mark low-confidence matches as "possible match" instead of pretending to be sure.
Cards that didn't fit. AI-written titles are much longer than hand-written ones, and our fixed card spacing made topics overlap. We replaced it with measured spacing: each card's offset depends on the real heights of its neighbours, $$ y_i = y_{i-1} + \max!\left(s,\ \frac{h_{i-1} + h_i}{2} + g\right) $$ where $s$ is the minimum spacing, $h$ is a card's scaled height and $g$ is the gap.
Cost and reliability. Real model calls cost money and occasionally fail. Live sessions start paused, so nothing is generated until you press play. Fixture mode keeps development and tests free and repeatable.
Merging three workstreams. Parallel work meant constant merges, new dependencies and shared-contract changes, so we leaned on typechecking and the full test suite before every push.

What we learned

Trust is the product. A summary is only useful if students believe it, which is why every topic links back to the transcript, the slide and the cited moments.
Validate everything a model returns. Schemas between the AI and the UI saved us from many subtle bugs.
Real-time UX is mostly about what doesn't move. Keeping the user's place stable mattered more than any animation.
Offline fixtures make AI apps testable. Without them, we couldn't have shipped with confidence.

What's next

Saving lecture timelines permanently, so a lecture is never lost when the server restarts.
Live audio transcription in the classroom, beyond transcript replay.
A scraper that turns any course website into a ready-to-use course.
Personal catch-up summaries ("you looked away from 10:42 to 10:45, here's what happened") and review tools built on top of the timeline.

We built Recall because we've all lost the thread in a lecture. Now none of us has to stay lost.