Try it: https://throughline.ideantoe.com (press Space to play the demo lecture; bring your own Claude and Gemini keys in ⚙ for the AI parts). Code: https://github.com/kitavkdl/sbuhacks26

Inspiration. I'm an international student. This semester I transcribed every lecture and matched it with the slides afterwards And then, the moments I actually got lost were during class; fast English, and the professor building on something from weeks ago. Notetaking AIs record the lecture you're in. None of them remember the course.

What it does. While a lecture plays (or Live mic listens), Throughline links what is being said right now to the moments in earlier lectures and slides it builds on, as cards with the shared ideas underlined. Click for context, or ask Claude to explain the link or summarize the scene. After class, Quiz me builds a short oral check from the day's links, grades your spoken or typed answer, and sends you back to the earlier moment for anything you missed. Lookup takes a word you only half caught, typed the way it sounded, and tells you what it means in this course and where it came up. Assist suggests what to say when you're asked a question, a question to ask, or (What now?) what to do when the professor turns to the class. Korean mode answers in Korean with technical terms in English.

How I built it. A static web app, no server, in-browser BM25 over past lecture windows and slide pages (light stemming, sentence-bounded bigrams) runs every 2 seconds in under a millisecond. A small logistic model, fitted on judged links, decides which results become cards. Gemini gemini-embedding-2 vectors (shipped with the course, cached in IndexedDB) re-rank by meaning. On each new line Claude Sonnet 5.5 sees what was said just before and right now, ranks about 30 candidates with structured outputs, and returns the useful ones best first; when its first pick isn't a clear match (about one line in four), Claude Opus 5.5 decides. Sonnet's answer shows first and Opus's replaces it; the local ranking takes over if a call fails, and if nothing is useful the best local match stays as a dashed "weak link". Claude Haiku 5.5 (Sonnet 5.5 selectable) streams explanations and writes and grades quiz questions with structured outputs. Live mic uses Gemini live transcription with the course glossary as custom vocabulary, or Chrome speech recognition with glossary hints, best-of-5 alternatives, a fixes table and Gemini re-transcription of each line.

Measuring it. We built judged test sets: candidate links pooled from five retrieval methods, each graded 0–2 by independent LLM judges with hidden sources (weighted κ 0.79–0.96). We tuned on half of 49 moments, then froze the model and judged two more sets: 49 other moments, and a different lecture with its own past. Across all three held-out sets (123 moments, ~3,700 links), bad cards per moment fell by two-thirds or more (2.50 → 1.08, 1.98 → 0.53, 2.16 → 0.46; paired-bootstrap 95% intervals exclude zero), and useful share of shown cards rose from 26–44% to 42–70%. One honest negative: the first-card gain on the tuning lecture (25% → 54%) did not replicate on the new sets. On a lecture that introduces a new topic, the model stays quiet and misses some good links. That is what the meaning layer is for: with real Gemini embeddings it finds more of the clearly-right earlier moments (31–19% → 44–52%), at the cost of about one more weak card per moment.

Finally, Claude picks the cards. A first picker (Sonnet grading the top 12) raised the first card's usefulness from 50 / 47 / 42% to 75 / 78 / 70% on the three OCW sets. We studied its misses and tried six variants on two OCW sets only; the winner (context split into "just before" / "right now", ~30 candidates ranked, Opus when Sonnet is unsure) reached 90% / 90% there. Frozen, it was then run once on six unseen sets — five real lectures from another semester (programming languages, systems, linear algebra, calculus, writing) and an office hour, 164 moments, no slides: first card useful 88% (84–100% per set, up from 77% for the first picker; nDCG 57 → 73%; bad cards 0.88 → 0.58; all intervals exclude zero). Our judges are Claude models too, so Gemini 4 Argon re-graded the first cards blind: about 85%. Query expansion (RM3) and BM25 parameter sweeps did not help, and we kept them out.

Challenges. Spoken lectures are full of filler that looks like signal to keyword search; the fix was to weigh course concepts said in the newest line. Speech recognition mangles jargon ("subclass" → "soft class"), so the glossary is generated from the slides.

What we learned. Measure before tuning: two of our most "obvious" improvements made links worse.

What's next. A week in a real class, counting clicked cards and quiz scores; meetings; more courses in the evaluation.

Built with: JavaScript, HTML/CSS, Claude API (Haiku 5.5, Sonnet 5.5, Opus 5.5), Gemini API (gemini-embedding-2, live transcription), Web Speech API, Web Audio, IndexedDB, Python (data prep), Node (evaluation). AI tools used: Claude Code for development and evaluation tooling; Claude and Gemini APIs in the product; Claude subagents as relevance judges, Gemini as a cross-family check.

Demo data: MIT OpenCourseWare 6.100L (Fall 2022), Dr. Ana Bell, CC BY-NC-SA 4.0. Text only.