Zygote: On-Device AI Agent Harness

On-device agent harness optimized for Arm: KleidiAI-tuned 1.2B on a $150 Galaxy M17, 34x faster first token, plus close-to-hardware shell access root-capable, sandbox-aware device control. Zero cloud.

Comment

Inspiration

A $150 Samsung Galaxy M17 — Exynos 1330, 2×Cortex-A78, no NPU, 8GB RAM with only ~2.2GB free for weights — and the question: what does an agent look like when it has to live entirely on that? Not a cloud client, not a demo that streams from a server. Something that works in airplane mode, owns no external dependency, and is genuinely useful on the cheapest hardware people actually buy.

What it is

Zygote is an on-device agent harness for Arm phones: a native Kotlin agent loop (tools, sessions, telemetry) wrapped around a llama.cpp engine (KleidiAI, Q4_0) with an embedded OpenAI-compatible server (127.0.0.1:8787) and a hyper-minimal PWA served from app assets. A two-tier LFM2.5 stack: a 230M router (194 t/s prefill ≈ 2 ms intent classification) and a 1.2B generator (664 MB Q4_0). The agent has tools — a close-to-hardware shell (sandbox-aware, su fallback, hard timeouts), to-dos, and phone control — with honest per-session, segment-accurate telemetry.

How we built it

  • Native engine: llama.cpp built with KleidiAI=ON; the A78 (Armv8.2 dotprod, no SME2/i8mm) correctly routes to the Q4_0 DotProd GEMV path — worth 2.06× prefill throughput over Q4_K_M.
  • KV-prefix caching on a hybrid arch: LFM2.5 is a Gated DeltaNet (linear-attention) hybrid whose recurrent state can't be partially erased by default (n_rs_seq = 0), so llama_memory_seq_rm silently no-oped → batch position gaps → "failed to initialize batch". We enabled rollback snapshots in a vendored llama.cpp patch, sized them to the ubatch constraint (n_rs_seq = 256), and added a full-reset fallback.
  • Latency stack: tool-schema slimming + relevantTools() keyword filter, 4096 context, Q8_0 KV cache, 8-thread big.LITTLE scheduling.
  • Server + PWA: raw-Socket OpenAI-compatible SSE server, run-in-flight guard, tool-loop orchestration, WebView-served React PWA.

Measured results

  • Time to first token (multi-turn): 59.6 s → 1.7 s (34×)
  • Decode speed: 7.5 t/s → 15.6 t/s
  • KV cache: 1 GB @ 8K context → Q8_0 @ 4096

The decode bound is memory bandwidth: every token re-reads the 664 MB weight matrix — ~9.2 GB/s at 14 t/s, roughly half of LPDDR4X. That number shaped every decision.

$$ \text{tokens/s} \approx \frac{BW_{mem}}{W_{model}} = \frac{9.2\text{ GB/s}}{0.664\text{ GB}} \approx 14 $$

Challenges

  1. The hybrid KV bug (above) — digging into llama.cpp's memory-allocator contiguity checks to find a silent no-op was dropping our trim.
  2. Tool-call reliability on a 1.2B — LFM2.5 emits multiple calls per block; the parser had to extract every name(args), and malformed calls forced a schema-slimming + relevance-filter redesign.
  3. Sandboxed shell — an Android app's sh inherits cwd / with no HOME: we give it a writable home, su fallback, and hard timeouts.
  4. Speculative decoding (honest negative): a 230M draft + 1.2B verify loop was fully built and measured at 20–39% acceptance — the 230M is a different model family, so the draft doesn't predict the target. It ships behind a flag with the numbers documented. A measured negative result beats a claimed one.

What we learned

Small models are a system problem: quantization path, cache layout, scheduling, and tooling each move the needle more than model swaps. On Arm, KleidiAI's microkernels are the single biggest multiplier available. And honest telemetry — no emulator numbers — is what makes an optimization story credible.

All code, benchmarks, and 6 known-issue tickets: github.com/system1970/zygote

Built With

Share this project:

Updates