Cliplex

Capture moments from Reels, YouTube Shorts, or TikTok and turn them into interactive language lessons with on-device transcription, translation, subtitles, and tap-to-learn vocabulary.

Comment

ClipLex — Turn Short Videos Into Private Multilingual Language Lessons

Inspiration

People already spend hours watching videos in languages they want to learn, but most learning opportunities disappear as soon as the video moves on.

You hear a useful sentence, understand part of it, and then it is gone.

ClipLex turns those moments into reusable language lessons.

Instead of uploading captured speech to a cloud service, ClipLex captures a permitted video or audio moment directly on Android and turns it into an on-device learning experience.

What ClipLex does

ClipLex is built for multilingual learning.

Learners can choose languages such as English, Hindi, Tamil, Telugu, Malayalam, Kannada, Bengali and Marathi, or use Any Language for multilingual speech recognition.

With ClipLex, a learner can:

  • Capture up to 3 minutes of permitted playback
  • Transcribe speech locally using Whisper or Android on-device recognition
  • Translate sentences and vocabulary on-device
  • Replay captured video with synchronized captions
  • Tap words for meaning and pronunciation
  • Save vocabulary
  • Generate quizzes
  • Practise pronunciation privately

The core flow is:

Watch → Capture → Understand → Practise

A Hindi video can become a lesson.
A Tamil conversation can become a lesson.
An English clip can become a lesson.

ClipLex is not tied to one fixed language pair. The learner chooses what they are learning and the language they want explanations in.

Captured media, transcripts, vocabulary, and lesson data stay on the device.

Why Arm optimization matters

Running multilingual speech recognition locally is useful only if it feels fast enough on a real mobile phone.

So a major part of ClipLex was optimizing Whisper inference on a real Arm64 Android device instead of only testing on a desktop.

For a controlled benchmark, I used an OPPO CPH2781 with:

  • arm64-v8a
  • Whisper Tiny English Q5_1
  • 6 CPU threads
  • 16 kHz mono audio
  • 2 warm-up runs
  • 5 measured runs
  • Fixed 9.4-second English audio sample

Result

Baseline median: 2393.764 ms

Optimized median: 692.659 ms

3.456× faster

71.06% lower native inference latency

The normalized transcript remained equivalent, and timed segments stayed enabled for synchronized playback.

The English benchmark gives a controlled and reproducible way to measure performance. In normal use, ClipLex can switch to multilingual Whisper models for other supported learning languages.

How it was optimized

ClipLex uses whisper.cpp and ggml through JNI.

The production build uses:

  • ARM64-only packaging
  • Quantized Whisper models
  • -O3
  • Arm NEON
  • Arm FMA
  • OpenMP
  • ggml repacking
  • 6-thread tuning
  • Reused Whisper model/context
  • audio_ctx=512 for the validated short-English fast path

I also tested smaller contexts, but rejected configurations that damaged transcript quality.

The goal was not simply to get the lowest benchmark number.

The goal was to make inference faster without breaking the learning experience.

Architecture

Android playback
      ↓
MediaProjection + AudioPlaybackCapture
      ↓
PCM → 16 kHz
      ↓
whisper.cpp on Arm64
      ↓
Timed transcript
      ↓
On-device translation
      ↓
Video / Audio lesson
      ↓
Vocabulary + Quiz + Pronunciation

ClipLex uses:

  • Kotlin
  • Jetpack Compose
  • Android NDK
  • C++ / JNI
  • whisper.cpp
  • ggml
  • ML Kit
  • Room
  • DataStore

Challenges

Android playback capture

MediaProjection and playback capture depend on Android, the source app, and the content itself.

Protected or DRM content may block capture.

ClipLex respects those restrictions and does not attempt to bypass them.

Speed vs accuracy

Reducing Whisper context improved performance, but aggressive settings could hurt transcription quality.

The final configuration was selected only after comparing both speed and transcript output.

Keeping everything on-device

Capture, transcription, translation, lesson storage, playback, vocabulary, and practice all have to work together without relying on a backend.

That made lifecycle management, memory use, model loading, and asynchronous UI state important parts of the engineering work.

What I learned

The biggest lesson was that mobile AI optimization is not just about choosing a smaller model.

Performance came from optimizing the complete system:

model + quantization + threads + context + native execution + lifecycle + UX

And every optimization needs to be validated against the real product experience.

Current result

ClipLex now runs as a release-optimized ARM64 Android application with a complete multilingual on-device learning workflow.

On the physical test phone, Whisper inference improved from:

2393.764 ms → 692.659 ms

while preserving the expected transcript.

That is the core idea behind ClipLex:

Turn the multilingual content people already watch into private, personalized language practice — directly on an Arm-powered phone.

Built With

  • android
  • android-kotlin
  • arm
  • arm64
  • audioplaybackcapture
  • c++
  • datastore
  • edge-ai
  • ggml
  • jetpack-compose
  • jni
  • language-learning
  • machine-learning
  • mediaprojection
  • ml-kit
  • neon
  • offline-ai
  • on-device-ai
  • on-device-translation
  • openmp
  • quantization
  • room
  • speech-recognition
  • whisper.cpp
Share this project:

Updates