LastLook

On-device visual memory for the things you leave behind.

Comment

Inspiration

Small personal objects are easy to leave behind in cafés, classrooms, hotels, and shared workspaces. Before you walk away, your phone should remember what you are about to forget.

LastLook turns that idea into a private, offline Android experience and a reproducible Arm optimization study. The goal was not merely to make an AI demo run on Arm, but to improve the complete on-device pipeline without hiding accuracy losses behind a single speed number.

What it does

LastLook gives an Android phone a short visual memory of a place:

  1. Capture a first scan and tap up to six objects to remember.
  2. Capture the area again before leaving.
  3. Run the local AI check and inspect boxes around registered objects that remain visible.

An offline sample scene gives judges a deterministic test flow. The application declares no Internet permission, so camera images, masks, embeddings, and decisions stay on the phone.

How we built it

The Arm64 Android pipeline combines:

  • EdgeTAM promptable object segmentation;
  • MobileNetV3 Large embeddings quantized to INT8;
  • cosine similarity and Hungarian one-to-one assignment;
  • a locked 0.55 matching threshold;
  • ExecuTorch 1.3.1 with the XNNPACK Android runtime and its official Arm KleidiAI integration;
  • Kotlin, Jetpack Compose, and CameraX.

The retained optimizations include registration-guided prompts, accepted-mask coverage pruning, a registered-object-count-plus-one candidate budget, quality-gated registration refinement, crop fallback, packed one-bit masks, prompt/crop buffer recycling, sequential tensor lifetimes, memory-mapped model loading, and a package-versioned integrity cache.

Optimization results

The baseline was a complete EdgeTAM FP32 + MobileNetV3 FP32 pipeline on an Arm64 Android phone. The final configuration was locked before formal held-out evaluation.

Metric Baseline Final Change
F1 0.7422 0.8266 +11.37% relative
Median latency 4615 ms 2661 ms -42.34%
P95 latency 5003 ms 3259 ms -34.86%
Full benchmark wall time 224.753 s 128.692 s -42.74%
Peak PSS 588.6 MB 547.4 MB -7.01%
Model size 46.0 MB 39.343 MB -14.47%
Decoder calls 2017 1156 -42.69%

Five temperature-controlled final runs produced the same 112 TP / 13 FP / 34 FN, F1 0.82656825, and zero crashes across 48 held-out scenes and 289 item decisions.

Challenges we ran into

The fastest isolated component was often not the best complete system. We measured and rejected reduced EdgeTAM resolutions, FP16 and INT8 segmenter exports, Vulkan, Qualcomm QNN HTP, LiteRT GPU, alternative embedding networks, change-region prompts, feature alignment, multimask selection, batched decoding, and altered matching objectives whenever they lost accuracy or failed to improve end-to-end performance.

One QNN HTP encoder reached 96–99 ms per call, but its full path took 4.776 seconds and reduced F1. A 1024→768 hybrid scan cut latency and energy, but changed object decisions. These failures redirected the work toward preserving reliable masks and matches while eliminating surrounding work.

Accomplishments that we are proud of

  • A real three-step camera experience that runs completely on-device.
  • A 42.34% median latency reduction while F1 improved by 11.37% relative.
  • Five identical formal runs with zero crashes.
  • A compact Arm64 judge APK with a deterministic offline sample.
  • Raw logs, model hashes, fixed splits, validation tools, accepted experiments, and rejected experiments published together.
  • A measured KleidiAI A/B that prevented an unnecessary custom runtime fork because the official ExecuTorch Android build already enabled it.

What we learned

End-to-end evidence matters more than attractive microbenchmarks. Quantizing only the embedder was safer than forcing lower precision into the segmenter. Candidate budgeting needed one spare proposal to retain real items. Sequential tensor lifetimes and packed masks improved memory without changing predictions. Negative results are reusable engineering evidence, not failed deliverables.

Why Mobile AI

This interaction happens when a person is about to leave. Local inference protects potentially sensitive camera views, works without connectivity, avoids a network round trip, and keeps the reminder on the device already in the user's hand.

Developer and community impact

The repository provides an Android EdgeTAM/ExecuTorch deployment path, INT8 embedding export, a fixed-split on-device benchmark runner, resource sampling, mask-packing and prompt-pruning utilities, raw positive and negative backend results, a compact APK, and reproducible chart tooling using uv.

Source, APK, raw evidence, and setup instructions: github.com/barebone-lab/lastlook

Build, run, and validate

Requirements are JDK 17, Android SDK platform 36, and an Arm64 Android 8.0+ phone. Build the compact demo with ./gradlew :app:assembleDebug, install the Arm64 APK with adb, and choose Load Sample Scene for the deterministic judge flow. Complete benchmark build commands, expected outputs, model hashes, and raw-run paths are documented in the public repository.

What's next

Future work will add second-scan physical-instance localization labels, broaden the device and lighting matrix, evaluate newer mobile Arm CPUs and NPUs, and continue improving mask reuse and startup behavior without changing locked decisions.

Disclosures

The fixed 60-pair source dataset was executed before the deterministic development/benchmark split, so it is a locked evaluation set rather than a pristine never-observed academic test set. Current labels measure registered-item presence; they do not support a correct-instance-location claim. LastLook is not a safety or theft-prevention guarantee.

Built With

Share this project:

Updates