AttentionRetention

Inspiration

Lecture fatigue is real. It’s easy to start a lecture focused and then drift off after a few minutes, or play Clash Royale. We wanted to build something that keeps students mentally engaged. By inserting lightweight, well-timed questions during a lecture, we turn passive consumption into something interactive and rewarding.

What it does

We built AttentionRetention as a real-time pipeline that combines speech recognition, retrieval, and LLM-based generation. We use Deepgram to transcribe lecture audio with low latency, then chunk the transcript into small segments and store them as embeddings in Redis text and vector databases. This lets us continuously build a searchable knowledge base of the lecture as it happens. We have an agent to determine the best times to ask questions to maximize retention. When it determines it is time to ask a question, it retrieves the relevant context from multiple databases, compares against its memory to target personalized areas of improvement, and semantically related past segments. That combined context gets fed into a more high-powered agent to generate questions that actually make sense in the flow of the lecture. On the frontend, we built a lightweight interface that receives questions through an API and displays them as clean, interactive popups. Lastly, we utilized Arize's monitoring and evaluation technology to debug as we developed the workflow. We offer full support for Zoom integration and even in-person use through microphone input.

Challenges we ran into

One of the main challenges was combining the different software infrastructures that we were largely unfamiliar with. The key to our success was proactivity in our planning, seamless organization, and emphasis on the easiest ways to connect across platforms (REST APIs). Furthermore, we encountered difficulties with live transcription and processing of the lectures. Since our initial plan was infeasible, we pivoted to an unfamiliar low-level implementation based on loopback driver hacking.

Accomplishments that we're proud of

We’re proud that we were able to build a fully working end-to-end system that goes from live audio all the way to adaptive, contextual questions. The retrieval setup, combining recent transcript data with vector search and agent memory, made a big difference in question quality. On top of that, we built a clean interface that integrates smoothly into the lecture experience, and we added an evaluation loop using Arize to help us improve questions and timing in a more systematic way.

What we learned

A key lesson for us was that real-time systems are less about any single component being fast and more about how well everything is coordinated together. Another insight was that evaluation is just as important as generation. In other words, without some way to measure quality, it’s hard to improve LLM outputs in a meaningful way.

What's next for AttentionRetention

We’re also interested in expanding beyond standard question formats into more creative and visual interactions. Gamification is another big area, with ideas like points, streaks, and rewards to keep users motivated. We’re exploring attention tracking as well, so the system can detect when you’re distracted and respond accordingly. On the content side, we want to generate better study materials like flashcards and summaries automatically, and even extend the system to work with textbooks instead of just audio.