Inspiration
As international students, we often struggle with different accents, fast-paced lectures, and unfamiliar terminology. Sometimes the issue isn’t knowledge, it’s comprehension speed.
Lecturers also come from diverse linguistic backgrounds, which makes real-time understanding even harder. We wanted to build something that removes that barrier instantly.
What it does'
LectureCaptions is a real-time, on-device lecture assistant that: • Converts speech to live captions • Translates captions into Chinese (toggleable) • Generates AI-powered summaries • Automatically creates potential exam questions • Exports structured lecture notes as PDF • Saves timestamped transcripts
How we built it
The project is built entirely in Swift using native Apple frameworks: Real-Time Speech Recognition Multilingual Translation AI-Powered Summarisation (Ollama + Gemma 4B) Glass UI Overlay Structured PDF Export
Challenges we ran into
Accent Recognition Swift 6 Concurrency LLM JSON Reliability Real-Time Performance
What we learned
• How to bridge Swift and Python safely using Process
• Managing MainActor isolation in Swift 6
• Handling async tasks without blocking UI
• Designing around macOS window levels and overlay behavior
• Structuring AI output strictly as JSON for reliability
• The importance of fallback systems (offline summary)
Log in or sign up for Devpost to join the conversation.