Bhasha Bridge

app menu
app status
translator app
first translation
hindi to english translation

I kept running into the same wall: every "offline" translation app on Android either wasn't actually offline, or ran a model so small it produced garbage. Meanwhile the situations where offline translation matters most a clinic counter, a border crossing, a village with no signal are exactly where you can't assume connectivity. I had an earlier prototype (v3.4.1) that limped along on a single test device with a translation engine that had never really been profiled. I wanted to know: could a real 200M-parameter transformer actually run well on an ordinary Arm phone, not a flagship, if someone actually optimized it instead of just shipping it?

What I learned

The biggest lesson had nothing to do with model architecture it was that you cannot trust an assumption you haven't measured. The IndicTrans2 ONNX export I started from looked like it should support incremental decoding. It didn't. The exported graph physically had no ports for a KV cache: input_ids, encoder_hidden_states, encoder_attention_mask — nothing to carry state between steps. Every single generated token was re-attending the entire prefix from scratch. That's O(n^2) decoding, and it's the worst possible shape for exactly the long sentences a translator needs to handle well.

I also learned that ORT's own profiler is blind to SIMD kernel dispatch — it can't tell you if Arm's KleidiAI SME kernel actually fired. I only found out by pulling simple perf traces and disassembling the hottest 40-byte loop by hand, and finding smopa za0.s, ... sitting there. That was one of the more satisfying "so it is real" moments of the whole project.

And a quieter lesson: a phone's thermal state can produce a 35% swing in latency with zero code changes. I nearly shipped a "regression" that was actually just the test device warming up from 31°C to 34°C over an afternoon of installs.

How I built it

The core rebuild was re-exporting IndicTrans2 as three separate ONNX graphs instead of one — encoder, decoder_init, and decoder_step — with the attention cache flattened into 72 named tensors (18 layers × 4) that survive the graph boundary. Optimum doesn't have a config for IndicTrans2's custom architecture, so this was a hand export, verified against the fp32 reference with a 7-check numeric gate before it touched the app at all.

From there:

Quantized to INT8 with ORT dynamic quantization — 1869 MB → 472 MB, greedy output byte-identical to fp32.
Made the runtime Arm-aware. Cpu Capabilities reads /proc/cpu info HWCAP flags and core topology from /sys/.../cpu freq; Execution Policy derives thread count and arena settings from what it finds no device list anywhere in the code.
Instrumented everything before optimizing anything. Cold start was ~27 seconds. Turned out 49% of that was a JSON dictionary parser reading one character at a time. Fixed the parser, parallelized session loading, cached the optimized ONNX graph format down to ~5.1 seconds.
Benchmarked on real hardware, not a simulator. Nine Android devices (later twelve including an iOS cross-validation), Armv8.0 through Armv9, same APK, no recompile: 50.3 → 412.8 tokens/sec depending on silicon.
Wrote down every negative result too. "Use all four big cores" seemed obviously right and regressed latency by 8% with 5× the jitter. That's in the docs alongside everything that worked.

Challenges I faced

The KV-cache export itself was the hardest single piece there was no library, no config, no reference implementation for this exact architecture. It was trial, numeric verification, and a lot of tensor-shape debugging.

The second hardest thing was resisting the urge to claim more than I'd measured. Early on I mis-attributed a memory win to mapped Initializers solving OOM kills the retest showed it was noise (1/6 vs 2/6 kills) and actually introduced a 2× latency spike on page re-read. It would have been easy to quietly bury that. Instead it's in the report as a retraction, because a project that only argues for itself isn't trustworthy to a judge or to me.

And a genuinely embarrassing one: a truncation bug where long sentences got silently cut off mid-word, present since v3.4.1, that survived an earlier commit that claimed to fix it. An audit caught it, not a test. That's now fixed and covered by a real regression check but it's a reminder that "I fixed it" and "I measured that it's fixed" are different sentences.