Nua - Offline On-Device Lecture Translator

• 

Inspiration: From Manipur to MIT

I come from Manipur, a remote northeastern state in India bordering Myanmar. Growing up there, we spoke Manipuri, but all of our primary textbooks and technical subjects were taught and written in English. There was an undeniable, constant linguistic disconnect. While I have endless appreciation for my hometown teachers, the reality was tough: English was simply not our first language.

Years later, I moved to the capital, New Delhi, and gained access to high-speed internet. My world cracked open. I discovered that world-class lectures from institutions like MIT, Harvard, and Stanford were sitting online completely for free. As I watched them, a profound thought hit me: What if the kids growing up in Manipur right now could access this exact same elite knowledge, but delivered in a way that felt entirely native to them? What if a Harvard professor could sit inside a rural classroom and explain complex concepts natively?

That is the soul of Nua. We aren't building a rigid, robotic word-for-word translator. When my teachers back home taught us, they took English textbooks and explained the core concepts using our everyday phrasing. Nua replicates that instructional philosophy entirely on-device, completely offline, and completely free. While our immediate focus targets English-to-Hindi ("Hinglish") to establish a highly scalable, code-mixed foundation, the architecture is built as an open blueprint to strip away the language barrier for students in any remote, disconnected corner of the world.

What it does

Nua is a 100% offline, zero-server-cost Android application designed for localized on-device video lecture translation and audio dubbing. The app targets the natural bilingual teaching style found in global classrooms—preserving essential English scientific nomenclature (like "Mitochondria" or "Stack Frame") while translating the surrounding explanatory language into local speech.

By operating entirely on-device, Nua removes the dependency on expensive cellular data packages and high-speed internet, allowing students in off-grid areas to import a lecture video, compile it once, and stream high-quality dubbed educational content repeatedly without consuming a single kilobyte of data.

How we built it

Nua is written natively for Android using Kotlin, Jetpack Compose, and the type-safe Navigation3 architecture. To bypass the heavy hardware constraints of mobile processors, we engineered the app as an On-Device Asset Compiler.

The pipeline runs asynchronously inside a high-priority Android ForegroundService across five sequential stages:

• Audio Extraction: A zero-memory MediaCodec interface processes audio packets using a recycled 16 KB primitive byte buffer, downmixing channels to a flat 16kHz mono WAV file straight to disk.
• ASR Transcription: An offline Vosk ASR engine scans the local WAV stream to output text segments tagged with frame-accurate millisecond timestamps.
• Context-Aware Translation: A local, 4-bit quantized Gemma 2B INT4 model runs via MediaPipe GenAI/LiteRT, utilizing a sliding window dialogue buffer to retain conversational context across sentences.
• Speech Synthesis: The native Android TTS engine builds localized vocal assets.
• Decoupled Local Workspace Storage: Assets are compiled into an isolated project directory:

context.filesDir/sessions/session_{video_id}/
    ├── manifest.json            # Timestamps and timeline directives
    ├── raw_lecture.mp4          # Unaltered source video track
    └── vocal_chunks/
        ├── vocal_chunk_0.wav    # Localized speech assets
        └── vocal_chunk_n.wav

During playback, the app utilizes a Dual-Player Synchronization Engine. The system mounts two independent instances of ExoPlayer live on the hardware (videoPlayer for visuals and room tone; vocalPlayer for the localized speech tracks). Event triggers are managed with zero main-thread polling overhead by scheduling events directly onto ExoPlayer's internal background decoding clock using the native PlayerMessage API.

Challenges we ran into

1. The RAM Wall & Thermal Throttling

In early development, we attempted to process the AI inference using a live, predictive 60-second streaming buffer. On physical phones, holding a 1.2GB local LLM in active memory alongside video rendering engines caused the Android Low Memory Killer (LMK) to force-close the app. Furthermore, continuous NPU processing triggered extreme thermal throttling, dropping token generation speeds below real-time speech clocks.

The Solution: We shifted to an Offline Batch Pre-Processing architecture. The phone executes the heavy AI compilation pipeline once behind an ingestion screen. Because video layers aren't rendering yet, the system sandbox isolates its RAM entirely to the LLM. Once compiled, assets run as flat files from disk, dropping playback CPU usage to near 0% and keeping the phone completely cold.

2. The Syllable Expansion Problem

Hindi naturally requires 15% to 30% more syllables than English to express identical conceptual blocks. Squeezing long translations into short original windows makes the voice sound unnaturally fast, while letting it run long breaks the timing of upcoming sentences.

The Solution: We deployed an Elastic Timeline Evaluation Matrix derived from our initial Python proof-of-concept. The system measures the synthesized voice file duration ($D_{\text{trans}}$) against the original speaker's physical window ($D_{\text{orig}}$) to compute a Temporal Expansion Ratio ($R_t$):

$$R_t = \frac{D_{\text{trans}}}{D_{\text{orig}}}$$

If $R_t > 1.15$, the segment flags a FREEZE_HOLD directive. The moment the playhead crosses that timestamp, a PlayerMessage callback invokes a visual frame lock (videoPlayer.pause()), freezes the professor's visual slide on screen, applies a parametric sidechain attenuation curve to smoothly duck the ambient background volume to 0.251 (-12dB), and triggers vocalPlayer.play(). The moment the vocal track signals an end callback (STATE_ENDED), the engine reverses the cross-fade envelope and unfreezes the video timeline automatically.

Accomplishments that we're proud of

• Zero-Marginal-Cost Scaling: By shifting 100% of the heavy machine learning workloads to the client edge hardware, we proved that AI-driven educational accessibility can scale to millions of users with an operational server cost of exactly $0.
• Buttery-Smooth Timeline Scrubbing: We engineered a two-tier coordinate tracking mapper that converts a student's seek bar interactions seamlessly between the Physical Media Domain ($T_{\text{physical}}$) and the Virtual Expanded Student Domain ($T_{\text{virtual}}$):

$$T_{\text{virtual}} = T_{\text{physical}} + \sum \Delta D_{\text{freeze}}$$

When a student grabs the slider, a fast binary search lookup recalculates the absolute core timestamps, steps the visual hardware player directly to the frame, cancels pending message schedules, and repopulates future events in less than 5 milliseconds—delivering the responsive performance of the premium YouTube app.

What we learned

We learned that mobile hardware constraints are powerful design parameters, not structural roadblocks. You do not need to burn battery life re-encoding heavy visual pixels on a phone when you can achieve identical results by dynamically orchestrating low-overhead background coordinate layers and mixing independent audio streams on the fly.

What's next for Nua - Offline On-Device Lecture Translator

• Deep Local Language Expansion: Expanding our localized translation schema to support deeper Indian regional mother tongues—starting with my native language, Manipuri (Meiteilon)—to fully bridge the gap that inspired this journey.
• P2P Mesh Distribution: Building an integrated peer-to-peer sharing protocol using Wi-Fi Direct. This will allow students who have compiled a lecture session package in town to wirelessly distribute the complete offline .zip asset bundle to classmates in deep rural dead zones, bypassing the internet entirely.

Built With

• 2b
• android
• asr
• exoplayer
• gemma
• genai
• jetpack
• kotlin
• media3
• mediapipe
• okhttp
• vosk