LectureRoyale

Inspiration

Since the Covid-19 pandemic 6(!) years ago, classes of all levels have increasingly been taught online through tools like Zoom. While there are obvious benefits to such a setup for students and instructors alike, there remain major challenges to overcome before remote education can deliver adequate learning outcomes. Studies (and personal experience) show that student attention plummets in online classes compared to in person classes, wreaking havoc on the skill development of our future generations. Perhaps surprisingly, this effect only worsens when students are asked to turn their cameras on, a common mechanism used by instructors to combat rampant disengagement. We offer a new hope, the last hope, to buck this trend and save the kids who, to this point, have been failed.

What is LectureRoyale

Through a comprehensive review of available education literature,we identified two consistently effective mechanisms: real-time gamification and frequent, low-stakes questioning. These features comprise the core philosophy of LectureRoyale, a tool designed to combat pervasive inattentiveness in online classes by introducing good-natured competition in the form of periodic questions generated by our agent. With a one-time five minute setup, instructors can immediately boost students’ enjoyment and learning during class as LectureRoyale listens in on the meeting and asks questions as it deems appropriate. Simultaneously, these periodic quizzes allow instructors to gauge their students’ understanding of different aspects of the content, enabling them to efficiently allocate lecture time to areas of need. LectureRoyale represents a revolution in live online learning, ensuring that 1) students practice using the information they learn and have fun learning in class and 2) instructors have a better sense of what topics they should spend more time going over.

How we built it

When starting out our codebase, we pair programmed using the Zed Editor. Once our codebase got larger, we made multiple repositories and started iterating on different prototypes. We also created example applications for each API that we used. Once we decided on the architecture of our main app, we integrated the other repositories together. This development process allowed us to work on different components of the app independently and allowed us to isolate bugs and API issues.

Challenges we ran into

• Real-time access to audio/video streams on popular streaming platforms
  
  • Goal: we aimed to support Youtube and Twitch streams only.
    
  • Issue: Neither platform offers a practical way to access live audio/video feeds.
    
  • Workaround: Pivot to Zoom/Google Meet, which are more commonly used in a classroom setting anyway.
    
• Real-time access to Zoom audio/video streams
  
  • Goal: support real-time Zoom calls.
    
  • Issue: Zoom’s official APIs are not designed to stream raw audio or video data to external backend services in real time.
    
  • Workaround: Use the Recall API with webhooks receiving continuously updated meeting transcripts every few seconds. This allows near real-time latency without any Zoom app installation or user-side configuration.
    

• Reliable real-time lecture transcription and visual context extraction

• Goal: Convert live lecture content into text by transcribing speech and extracting information from shared visual content.

• Issue: We evaluated WhisperFlow for real-time speech-to-text and Overshoot for video-to-text extraction. In practice, both systems were difficult to integrate even for small-scale test cases, as their APIs were not designed for low-latency, streaming workloads required in live classroom settings.

• Workaround: We abandoned both approaches and instead relied on more reliable transcript-based inputs that better support continuous, real-time processing.

• Managing context for real-time AI queries

  • Goal: Provide the question-generation agent with enough recent lecture context to generate relevant, timely questions while keeping inference fast and cost-effective.
    
  • Issue: Passing large amounts of transcript data into each model query increases latency and inference cost, making naïve full-context approaches impractical for live classroom use.
    
  • Workaround: We implemented a time-based rolling transcript window that continuously updates by adding new content as older content is removed. This preserves the most relevant recent context while keeping model inputs small, using raw transcript slices without additional summarization.
    

• Real-time communication between instructor and student overlays and the backend

  • Goal: Support low-latency, interactive gameplay during a live lecture by synchronizing questions, answers, and leaderboard updates across both instructor and student overlays.
    
  • Issue: This required reliable, bidirectional messaging between multiple overlay clients and a central server so that state changes propagate instantly during class.
    
  • Workaround: We used WebSockets for persistent, bidirectional client, server communication, enabling real-time delivery of new questions and immediate broadcast of answer/score updates. We also used ngrok in our deployed setup to provide a stable, publicly reachable endpoint for these live WebSocket connections.
    

• State synchronization across backend and Electron components

  • Goal: Maintain a consistent, up-to-date application state across the central backend server, the Electron backend process, and the Electron frontend UI.
    
  • Issue: With three independently running components, state changes originating in one layer (e.g., new questions, score updates, or session status) needed to be reliably propagated to the others, making it easy for components to fall out of sync during live use.
    
  • Workaround: We established explicit state boundaries and update flows between components, using Electron’s IPC mechanism to synchronize state between the Electron backend and frontend, and WebSocket events to keep the Electron backend aligned with the central server in real time.
    

• Building a lightweight overlay that doesn’t interfere with the Zoom call

• Goal: Display real-time questions and a leaderboard on top of the live lecture without disrupting Zoom’s audio/video, screen sharing, or user controls.

• Issue: Typical approaches (e.g., browser tabs or separate windows) force students to context-switch away from the lecture, while deeper integrations risk interfering with the meeting experience. We also found Electron slower to iterate on than a standard web app and more complex to structure (separate Electron frontend/backend processes with different styling constraints).

• Workaround: We implemented the overlay as an Electron + React desktop app that runs alongside Zoom and stays visually lightweight. This allowed us to render an always-available UI layer for students and instructors without modifying the Zoom client or interfering with the call itself.

Accomplishments that we're proud of

• Role-specific, usability-first UI
  
  • Separate interfaces for instructors and students, each optimized for its role during a live lecture
    
  • Designed to be immediately intuitive, requiring no training or setup during class
    
  • Lightweight overlay that integrates seamlessly with Zoom without disrupting the lecture experience
    
• 75× faster than manual question authoring
  
  • Measured by comparing our agent’s generation time against a co-founder manually typing an equivalent prompt
    
  • Enables real-time question generation without interrupting the flow of the lecture
    
  • Allows instructors to focus on teaching rather than live content creation
    
• Real-time insight into student understanding
  
  • Periodic questions provide immediate feedback on which concepts students understand or struggle with
    
  • Helps instructors adjust pacing and emphasis during the lecture itself, not after
    
  • Turns passive attendance into active participation without additional instructor overhead

What's next for LectureRoyale

• Expanded feature set
  
  • Teams: Enable collaborative play modes where students answer questions in groups, supporting team-based competition and cooperation.
    
  • Achievements: Introduce achievement badges and progression systems to reward consistency, improvement, and mastery over time.
    
  • Sound effects: Add optional, subtle audio feedback for correct answers and milestones to increase engagement without disrupting lectures.
    
• Upgradable and customizable UI
  
  • Allow instructors to tailor the visual appearance and intensity of the overlay (e.g., minimalist vs. competitive modes)
    
  • Support accessibility-focused themes and adjustable emphasis for different classroom settings

Built With

• electron
• gemini
• javascript
• react
• recall
• typescript
• webhook
• websocket
• zoom