Challenge track
Mobile AI. PocketTune Bench optimizes and validates on-device inference configurations on Arm client hardware, with a release Android APK and a complementary Apple Silicon profile.
Inspiration
On-device AI optimization is full of attractive but incomplete numbers. A quantized model can be much smaller and look faster while silently changing its predictions. An accelerator-friendly policy can make steady-state inference three times faster while adding hundreds of milliseconds to model load.
PocketTune Bench asks a more useful question: which valid configuration should this device use for this process lifetime?
What it does
PocketTune Bench is a reproducible model health check and configuration recommender for Arm client devices. It runs one pinned MobileNetV3 Small image-classification workload through Android ExecuTorch/XNNPACK and Apple M4 Core ML profiles, checks quality before speed, records exact artifact and corpus identities, and converts repeated measurements into a workload-scoped recommendation.
The project outputs:
a self-contained Android release APK containing both exact-hash PTE models and the fixed timing corpus; counterbalanced multi-batch timing with thermal, battery, build, model, corpus, and preprocessing identity; separate fresh-process Android peak-PSS and Apple M4 peak-RSS passes; strict JSON Schema validation, immutable raw JSONL, deterministic summaries, and a hash-checked unified recommendation; an M4 workload curve that exposes the cold-load versus steady-state tradeoff.
Android optimization
Our first global static INT8 export looked excellent: 72.99% smaller and 1.83x faster. But it achieved only 10% top-1 agreement on the timing corpus. PocketTune rejected that result instead of publishing a false win.
We replaced it with dynamic per-input activation quantization and per-channel symmetric INT8 weights restricted to the classifier's Linear modules. The exact ExecuTorch PTE agrees with the FP32 reference on 62 of 64 validation images (96.875%) and is 47.37% smaller.
On a Xiaomi Pad 7 Pro (SM8635), the release APK completed two opposite-order 60-record sessions and six fresh-process memory passes. INT8 median latency was 4.60% lower in the FP32-first session and 2.12% lower in the INT8-first replication; the pooled reduction was 2.52%. Median peak PSS fell from 172,152 KiB to 163,079 KiB, a 5.27% reduction. We use 2.12% as the conservative replicated latency claim.
Apple M4 optimization
Both Core ML policies use the same 5,183,875-byte FP16 package. CPU_ONLY is the load-sensitive baseline. ALL enables automatic Core ML compute-unit selection; PocketTune does not claim which individual GPU or Neural Engine unit executed the graph.
Across two opposite-order 60-record sessions, ALL delivered 3.07x and 2.97x steady-state median speedups. But pooled median model load was 760.6 ms for ALL versus 110.5 ms for CPU_ONLY.
The recommendation is therefore workload-aware:
use CPU_ONLY for short, load-critical processes through 1,047 inferences; disclose 1,048–1,090 as a session-sensitive crossover band; use ALL when already loaded or from 1,091 single-image inferences per process.
How we built it
The Android app is written in Kotlin and runs ExecuTorch 1.3.1 with XNNPACK on Arm64. Python tools export and quality-check MobileNetV3 Small models, generate deterministic corpora, validate raw JSONL against strict schemas, aggregate opposite-order sessions, and produce the final recommendation.
The Apple path uses Core ML Tools and a separate M4 benchmark runner. Android and M4 remain distinct platform profiles; their absolute latency values are never presented as a cross-device ranking.
How to run and validate
For the fastest judge path, download the measured Arm64 APK from the public v0.1.0 release, install it on an API 28+ Android device, open PocketTune Bench, and run the available-model smoke test. Both configurations should return a finite 1 x 1000 output vector.
To reproduce every frozen recommendation from the committed raw evidence with Python 3 and standard shell tools: ./scripts/reproduce_frozen_reports.sh
Expected final line: PASS: opposite-order Android summaries, M4 workload curve, and unified recommendation reproduce byte for byte.
The repository also documents complete environment setup, model and corpus recreation, release assembly, device staging, and formal measurement protocols.
Challenges
The hardest problem was numerical quality in global PT2E MobileNetV3 quantization. Increasing calibration coverage did not rescue it. Narrowing the quantization scope did. That investigation changed PocketTune from a generic speed test into a quality-first decision system.
A second challenge was measurement order. We repeated both Android and M4 sessions with the starting order reversed, kept every valid record, and disclosed stability ratios rather than selecting only the fastest run.
Accomplishments
Caught and rejected our most exciting but invalid early optimization. Promoted a byte-reproducible Android candidate that passes the quality gate. Bound APK, model, corpus, preprocessing, device, runtime, and source identities to every formal record. Reproduced both Android and M4 results with opposite starting orders. Turned a 3x steady-state headline into an actionable process-lifetime decision. Published the exact measured APK, raw evidence, schemas, analyzers, reports, diagrams, and reproducible demo source.
What we learned
Model optimization is a constrained decision problem. Size and speed matter only after output quality and artifact identity are established. Startup cost can reverse a steady-state winner, and reproducible negative results are part of the product rather than something to hide.
What's next
We plan to repeat the accepted Android configuration on a Snapdragon 8 Gen 2 phone as a secondary-device generalization check and extend the same evidence contract to additional vision models without weakening the model-specific quality gate.