Interline

An Offline Bilingual Simultaneous Interpreter

Comment

Inspiration

As an international student, I often rely on translation to understand lectures and conversations. However, existing iOS translation experiences are not always designed for long, continuous sessions. With Apple Intelligence bringing powerful language understanding directly to supported devices, I wanted to explore how on-device AI could make interpretation more contextual, private, and accessible.

What it does

Interline is an on-device bilingual captioning app for iPhone and iPad. It continuously transcribes speech, translates it in real time, and keeps both languages aligned. Apple Intelligence improves completed translations by understanding recent conversational context, resolving ambiguous references, applying glossary terms, and correcting speech-recognition errors when there is sufficient evidence. It also generates titles and summaries for saved conversations. Sessions can be searched, replayed with recorded audio, and exported as TXT or SRT files.

How we built it

Interline is built with SwiftUI and Apple’s Speech, Translation, Foundation Models, AVFoundation, and SwiftData frameworks. Apple’s Translation framework provides a fast first-pass translation. Interline then uses the on-device Foundation Models framework to request structured, context-aware refinements from Apple Intelligence. This two-stage design combines responsiveness with deeper language understanding while keeping speech and transcripts on the device.

Challenges we ran into

One major challenge was speech-recognition quality across different languages. Recognition boundaries and intermediate results vary significantly between writing systems. Chinese, Japanese, and Korean were especially difficult: captions could appear too late, be divided at unnatural positions, or change substantially as more speech arrived. Language-aware buffering and segmentation improved the experience, but recognition quality for these languages is not yet fully solved. Another challenge was balancing translation quality, latency, and sustained performance. Running speech recognition, translation, and Apple Intelligence continuously can generate heat, while slower devices may receive captions faster than they can refine them. Interline adapts the refinement frequency, limits queued AI work, and retains the fast translation whenever refinement cannot keep up.

Accomplishments that we're proud of

I am proud that Interline can support long conversations without making Apple Intelligence a single point of failure. Transcription and fast translation remain responsive even when AI refinement is busy, unavailable, or paused because of device temperature. I am also proud of the contextual refinement system. Instead of simply asking a language model to translate isolated sentences, Interline provides recent context, recognition alternatives, timing signals, and glossary terms through structured prompts. It can improve the current caption—and, when new context reveals a clear earlier mistake, safely revise the previous one. The result is a complete experience that extends beyond live captions, with searchable history, AI-generated summaries, synchronized audio playback, and transcript export.

What we learned

I learned that on-device AI is most effective when it is given a focused role, reliable context, structured outputs, and safe fallbacks. Real-time translation is not just a model call—it requires coordinating multiple systems that operate at different speeds while carefully managing device resources.

What's next for Interline

Next, I plan to continue improving transcription and caption segmentation, particularly for Chinese, Japanese, and Korean. I also want to add speaker separation, support more languages, reduce energy usage during long sessions, and explore additional Apple Intelligence capabilities for reviewing and organizing saved conversations.

Built With

Share this project:

Updates