LetsStud

Inspiration

As a student, I often struggled to make proper notes while watching long lecture videos. When I focused on writing notes, I missed important explanations. But when I focused only on listening, I forgot key concepts later. Rewatching entire lectures just to capture missed points was time-consuming and inefficient.

I realized that many students face the same problem — educational videos are everywhere, but converting them into clear, structured notes is still a manual and difficult process. This inspired me to build LetsStud, a system that automatically converts lecture videos into organized study notes using AI.

What it does

LetsStud is an AI-powered multimodal lecture summarizer that converts educational videos into structured notes.

Users simply paste a YouTube lecture link and the system automatically:

Extracts spoken lecture content
Detects important slide or board changes in the video
Extracts text from slides and whiteboards
Aligns information using timestamps
Generates structured study notes

Instead of watching a long lecture multiple times, students can review concise notes generated from the video.

How we built it

LetsStud uses a multimodal AI pipeline that analyzes different types of information from lecture videos.

Video Processing – The system downloads lecture videos using yt-dlp.
Scene Change Detection – Using SSIM-based analysis, the system detects meaningful visual changes such as slide transitions.
Frame Extraction – Key frames are extracted when visual changes occur.
Visual Text Extraction – Gemini Vision extracts readable text from slides or boards in these frames.
Audio Understanding – Gemini models analyze the spoken lecture content to generate transcript insights.
Multimodal Fusion – Audio, visual, and subtitle data are aligned into timestamp windows to create contextual understanding.
Note Generation – Finally, Gemini generates structured lecture notes summarizing the content.

The backend is built using FastAPI, and the frontend interface is developed with Next.js.

Challenges we ran into

One of the biggest challenges was aligning multiple data sources such as speech, visual slides, and subtitles. These signals occur at different times in the video, so synchronizing them accurately required careful timestamp alignment.

Another challenge was reliably detecting meaningful visual changes in lecture videos, since slide transitions or board writing can happen gradually rather than abruptly.

We also worked on optimizing the system architecture to reduce heavy dependencies while keeping the analysis accurate.

Accomplishments that we're proud of

We successfully built a working multimodal AI system that can analyze lecture videos and generate structured notes automatically.

Some key accomplishments include:

Implementing SSIM-based scene detection to identify important visual changes
Extracting text from lecture slides and boards using AI vision models
Combining audio, visual, and subtitle information into a unified context
Creating a tool that helps students review lectures faster and more efficiently

Most importantly, we built a system that addresses a real challenge students face while studying.

What we learned

Through this project, we learned how powerful multimodal AI systems can be when different sources of information are combined.

We also gained experience designing AI pipelines that involve video processing, visual understanding, and language models working together to generate meaningful outputs.