NorthCare Edge

Arm-optimized offline AI for frontline healthcare, cutting on-device voice inference by 53.8% while maintaining our 100/100 benchmark quality gate.

Comment

Inspiration

In Northern Ghana, a home visit does not wait for a strong internet connection.

Frontline healthcare workers may need to document a visit, capture information by voice, identify important details, and hand off an urgent case even when connectivity is unreliable or unavailable. That was the problem behind NorthCare AI: building a healthcare workflow that could continue working offline.

For the Arm AI Optimization Challenge, we took that existing application and asked a more specific engineering question:

What happens when we take a real offline AI workflow and optimize its most expensive component for an Arm-powered smartphone?

The result is NorthCare Edge — an Arm-focused optimization of the NorthCare offline healthcare application. Instead of building another isolated AI demo, we measured the AI already used by the application, identified its actual bottleneck, tested different optimization strategies, rejected the ones that did not work, and shipped the successful optimization into the real Voice-to-Care workflow.

What NorthCare Edge Does

NorthCare Edge is an Android-first, offline-first healthcare application designed for authorised frontline workers.

Its AI workflow allows a worker to record a visit using voice. Speech is transcribed locally, a lightweight language model extracts structured information, and the worker reviews and confirms the result before anything is saved.

The application also includes an on-device AI assistant and an offline referral workflow using signed QR referral passports, allowing information to be verified between devices without requiring a cloud round-trip.

The important part for this challenge is that the AI path runs locally on the Arm device.

M4A
 ↓
Native MediaCodec decode
 ↓
Whisper — on-device speech recognition
 ↓
Transcript
 ↓
Qwen 2.5 0.5B — on-device extraction
 ↓
Worker confirmation
 ↓
SQLite

NorthCare Edge was tested on a Samsung Galaxy S20 Ultra (SM-G988B, arm64-v8a) using the CPU backend.

The Optimization Problem

We began by freezing the existing AI configuration and measuring it before changing anything.

The baseline used:

  • Whisper ggml-base.en
  • Qwen 2.5 0.5B Instruct Q4_K_M
  • 4 Whisper threads
  • 4 Qwen threads
  • CPU execution
  • The same synthetic benchmark fixture for every comparison

The first profiling run immediately showed where the problem was.

Stage Time Share
Whisper load 4.5 s 8.3%
Whisper transcription (decode + inference) 42.4 s 78.5%
Qwen load 2.5 s 4.6%
Qwen inference 1.8 s 3.3%
Other / harness 2.8 s 5.2%
Total 54.0 s 100%

Whisper was responsible for the overwhelming majority of the measured pipeline time.

That gave us a clear optimization target instead of changing models or settings simply because they looked interesting.

Measure → Experiment → Reject → Ship

Our optimization process followed one rule:

Every optimization must show what changed, how much it changed, and how it was measured.

We tested one variable at a time.

Increasing Whisper threads from 4 to 6 was actually worse on the target Arm phone, making the experiment approximately 72% slower. It was rejected.

Other configuration changes, including removing the prompt and enabling speedUp, did not meet our minimum improvement gate and were also rejected.

These failed experiments remain in the project because they are part of the engineering evidence. We did not remove unsuccessful results simply because they made the final result look less impressive.

The successful optimization was replacing:

Whisper base.en
~148 MB

with:

Whisper tiny.en
~77.7 MB

The smaller model was accepted only after it passed our fixed synthetic regression fixture. Once the result was verified, the configuration was promoted into the production Voice-to-Care path.

The Measured Result

All comparisons were performed on the same Arm64 device using the same benchmark workload.

Metric Baseline NorthCare Edge Improvement
Whisper transcription 42.4 s 19.6 s 53.8% faster
End-to-end AI pipeline 54.0 s 26.5 s 50.9% faster
Whisper model size ~148 MB ~77.7 MB 47.5% smaller
Synthetic regression fixture 100/100 100/100 Held

The 100/100 result is a fixed synthetic regression test, not a clinical accuracy study or medical validation. It checks that the expected phrases and required extraction fields remain intact after optimization.

This distinction matters to us. We wanted to demonstrate an engineering improvement without making unsupported clinical claims.

Why This Matters on Arm

On a desktop GPU, a model can often be made to feel fast simply by throwing more computational resources at it.

A mobile device has different constraints.

Compute, memory,thermal behaviour, battery consumption, and responsiveness all matter. A model that is unnecessarily large or computationally expensive can make an otherwise useful offline application difficult to use in the field.

NorthCare Edge demonstrates this trade-off on an actual Arm64 smartphone, rather than presenting a theoretical model comparison.

The accepted optimization reduced the Whisper model footprint by approximately 47.5% and reduced measured Whisper transcription time by 53.8% on the target device, while the synthetic regression fixture remained unchanged.

Just as importantly, the optimization was not performed in isolation. The smaller model was actually promoted into the Voice-to-Care workflow used by the application.

Edge Lab

To make the optimization process visible, we built Edge Lab directly into the application.

Edge Lab allows the development team — and judges reviewing the project — to see the benchmark process rather than relying only on numbers written in the README.

The lab provides:

  • Benchmark execution on the device
  • Stage-level timing
  • Baseline and optimized comparisons
  • Experiment history
  • Accepted and rejected experiments
  • Optimization timeline
  • Reproducible synthetic fixtures

The important idea is that the benchmark is part of the product's engineering tooling.

A judge can see not only that the optimized version is faster, but also what we tried before reaching that result.

Challenges We Faced

The biggest lesson was that, intuitive optimization is not always correct on mobile hardware.

Our first assumption was that adding more CPU threads might improve Whisper performance. On the Galaxy S20 Ultra, the opposite happened. The 4→6 thread experiment became approximately 72% slower.

We also learned that apparently inexpensive configuration changes are not automatically useful. Several settings failed to produce our minimum improvement and were rejected.

Another challenge was measurement accuracy. Whisper's current transcribe() path bundles decoding and inference within the timing boundary, so we deliberately report the measurement as “Whisper transcription (decode + inference)” rather than claiming that we measured an isolated decoder or kernel.

Finally, quality could not simply be judged by whether the output “looked good.” We needed a repeatable regression gate. The smaller model was therefore promoted only after it reproduced the expected synthetic fixture results.

What We Learned

The most important lesson was simple:

Profile before optimizing.

Without profiling, we could easily have spent our limited development time optimizing Qwen, even though Qwen accounted for only about 3.3% of the measured end-to-end pipeline.

We also learned that negative results are valuable engineering evidence. The rejected experiments helped us understand the behaviour of the target Arm device and prevented us from presenting optimization as a collection of guesses.

Most importantly, model optimization should be evaluated in the context of the application. The winning model was not merely faster inside a benchmark; it was integrated into the actual Voice-to-Care workflow.

What We Are Proud Of

NorthCare Edge combines a real offline healthcare workflow with a measurable Arm optimization study.

We are particularly proud of:

  • Running the evidence workload on a real Arm64 smartphone
  • Identifying Whisper as the dominant bottleneck through profiling
  • Testing optimization strategies instead of assuming they would work
  • Keeping failed experiments visible
  • Reducing Whisper transcription time by 53.8%
  • Reducing the Whisper model footprint by 47.5%
  • Maintaining a 100/100 synthetic regression fixture
  • Building Edge Lab to make the optimization process visible inside the application
  • Keeping human confirmation,and safety boundaries in the healthcare workflow

NorthCare Edge is ultimately about a simple idea:

If AI has to work in the field, it should be engineered for the device that carries it.

NorthCare Edge -Global intelligence. Local execution.

Built With

Share this project:

Updates

Submission history