Iksha - On-Device AI Assistant

A private AI assistant running Gemma 4 entirely on your Arm phone: voice, vision, live face recognition, and hands-free skills like directions, alarms, and reminders, with nothing sent to a server.

Comment

Iksha

Track: Mobile AI

Iksha is an Android assistant that runs a Gemma 4 model entirely on the phone's own Arm silicon. Chat, voice conversation, live camera narration, face recognition, and device actions all execute locally. The only feature that ever contacts a server is optional web search, and it is off by default.

Iksha (ईक्षा) is Sanskrit for sight. The name is the thesis: the app exists to give sight to someone who cannot see, without sending their world to a datacentre to do it.

Inspiration

Assistive apps for blind users have a structural problem. To describe what a camera sees, they stream that camera to a server. The person most dependent on the technology is the one whose home, family, documents and daily life get continuously uploaded — and they are the least able to audit what happens next.

That trade-off is not a law of nature. It is a consequence of the model living somewhere else. Arm-based phones now ship enough compute to run a capable multimodal model locally, which means the trade-off can simply be removed rather than mitigated with a privacy policy.

So i built the assistant that never has to ask for that trust: Assistive Mode, which narrates the world in first-person spatial language ("the door is directly in front of you, three steps ahead"), recognises people you have introduced it to, and does it all with the network off.

What it does

  • Local Gemma 4 inference in two builds, E2B and E4B, chosen automatically from the device's actual free memory.
  • Assistive Mode — continuous camera narration for blind and low-vision users, fully hands-free: the microphone reopens by itself the moment the assistant stops speaking, because a "tap to speak" button is useless to someone who cannot see it.
  • Live face recognition with continuity. Introduce someone once by saying "this is Sam". From then on Iksha tracks them across frames, keeps their identity through turns of the head and motion blur, remembers what they were last doing, and re-identifies them when they leave the room and come back.
  • Face Unlock with an enforced PIN fallback, key-stretched on-device with 120,000 rounds and a random salt.
  • Skills — alarms, timers, reminders, calendar events, directions and calling, chosen by the model itself from what you said, plus user-authored custom skills.
  • Offline voice, both directions: on-device speech recognition when there is no signal, and a fully offline synthesised voice always.
  • Documents — photos, multi-page PDFs and text files read by the vision encoder.

How i built it

Native Kotlin and Jetpack Compose. Gemma 4 runs through Google's LiteRT-LM runtime. Face detection is MediaPipe BlazeFace; face embeddings are a raw TFLite MobileFaceNet interpreter. Speech recognition and synthesis are sherpa-onnx. Optional web search proxies through a stateless Vercel function; there is no database anywhere in the product.

Targeting Arm

Iksha ships arm64-v8a exclusively. There is no x86 path and no 32-bit path, because every performance decision in the app assumes a modern Arm SoC.

The most important of those decisions is how work is split across the Arm SoC's processors. The obvious approach — accelerate everything — is wrong here. Iksha runs:

  • the language model on the GPU via Backend.GPU(),
  • the vision and audio encoders on the CPU, deliberately, even though a GPU path exists,
  • and the face detection and embedding models on the CPU, deliberately, with the TFLite GPU delegate left unwired.

The reason is contention, not capability. The face models and the vision encoder are small enough that the CPU handles them comfortably, and putting them on the GPU makes them compete with the LLM for the same OpenCL queue.I measured the encoders being slower when sharing the GPU than when split off to the CPU. On a phone, the scarce resource is not raw throughput, it is the GPU queue the LLM is already saturating.

Memory as the real constraint

The KV cache, not the weights, is what kills a phone. Iksha reads ActivityManager.MemoryInfo at startup and sizes the context window from it — 4096 tokens above 7.5 GB total with 2 GB free, 2048 above 5.5 GB, 1024 otherwise — then falls back down the rungs if initialisation fails, and persists the last known-good window so the next launch starts where the device actually succeeded.

Prompts are then budgeted against whatever window the engine really came up with, trimmed in order of expendability: conversation history first, then the tail of any web context, and only the user's own words as a last resort. This makes "input token ids are too long" structurally impossible rather than merely unlikely.

The same memory reading drives the model recommendation: E2B for 4 GB devices, E4B for 8 GB, shown to the user at setup with the reasoning visible.

Latency work

  • The routing pass is skipped entirely when it cannot produce a useful answer. Every message runs a small classification pass to decide whether it needs a tool or a web search. With the network off and no skills enabled, that pass is structurally incapable of returning anything but "answer directly" — so it is not run at all, removing a full round-trip before the first token on the offline path that matters most.
  • Face embedding is throttled per tracked person, not per frame. Detection runs every frame (a few milliseconds); the expensive embedding runs every 0.9 s while someone is unidentified and every 4 s once they are named. A stable recognised face costs almost nothing to keep following, which is what makes continuous recognition affordable alongside a running LLM.
  • Tracking frames are scaled by longest side to 512 px, preserving aspect ratio. An early version forced fixed landscape dimensions onto a portrait frame, squashing every face and degrading both detection and the embedding.

Challenges i ran into

The NPU i could not use. Iksha is built for Arm silicon, and the natural next step was the NPU. Backend.NPU() in LiteRT-LM requires a vendor dispatch library (libLiteRtDispatch_Qualcomm.so) that is not shipped in the published AAR. I decompiled the AAR to confirm it, on two SDK versions. Worse, calling it without the library does not throw — it aborts in native code with a SIGABRT that no Kotlin try/catch can contain, taking the process with it. I removed the NPU path rather than ship a crash, and the app says plainly that NPU support is on the roadmap rather than claiming it.

A dependency bug that only appeared after a reply finished. litert-lm-android:0.15.0 declares kotlinx-coroutines 1.9.0 in its POM, but its compiled bytecode calls a SendChannel.close$default overload that only exists in a newer version. Because both the app and the SDK asked for the same version, Gradle's "highest wins" resolution never kicked in. The result was a NoSuchMethodError at the exact moment a streamed reply completed. Root-caused from a device logcat and fixed by raising our own declaration.

Recognition that had never actually run. Face recognition worked when you asked a question and never at any other time. The cause was not the model: CameraStreamManager declared a frame callback, MainActivity implemented it — and nothing ever invoked it. The passive recognition loop had never executed once. Finding that changed the whole design, and led to a proper tracker with box matching, vote accumulation across frames, and a memory of people who have left.

Making the model use its own eyes. Even with a camera image attached, Gemma would sometimes answer "I'm an AI, I don't have a personal view". Nothing in the prompt asserted that a real image was present. It now does, explicitly, and the scene description handed to the model is a continuous account — who is here, where, for how long, what they were last doing, who recently left — instead of a snapshot of one frame.

Tools, not a rulebook. Our first skills implementation had the model emit free text which Kotlin then tried to split with rules like "does it start with the word from". Every phrasing outside those rules broke. I rebuilt it as real tool calling with named, typed parameters: the model does the language understanding, and Kotlin only normalises single clean values and fires the Intent. "Best route from IIT Bombay to Gateway of India", "how do I get to Gateway from IIT B" and "I'm at IIT Bombay, need to reach Gateway" now all produce the same two fields, because a language model handles paraphrase natively and a regex never will.

What i learned

Running a model locally is the easy part now. The hard part is everything around it: deciding which processor on the Arm SoC each workload belongs on, sizing memory against a device you have never seen, and keeping a 4B model's behaviour stable enough to trust with a device action.

I also learned to stop guessing. Nearly every real bug in this project was found in a device logcat, and the one time i theorised a cause instead of waiting for a log, i were wrong. Package visibility, permission-gated activities, dependency POM mismatches — none of these were visible from reading our own code.

What's next

NPU acceleration the moment the vendor dispatch library is published. Structured in-chat outputs for the quiz and flashcard skills. And a wider on-device benchmark sweep across Arm SoC generations, so the model recommendation is grounded in measured throughput rather than memory alone.

Built With

  • android
  • arm
  • arm64-v8a
  • camerax
  • computer-vision
  • edge-ai
  • face-recognition
  • firebase
  • gemma
  • gpu-compute
  • gsap
  • jetpack-compose
  • kotlin
  • litert-lm
  • mediapipe
  • onnx
  • opencl
  • sherpa-onnx
  • speech-recognition
  • tensorflow-lite
  • text-to-speech
  • typescript
  • vosk
Share this project:

Updates

Submission history