Needle

Generate less. Route locally. Audit everything.

Comment

Inspiration

Most mobile AI projects optimize how fast a model generates. Needle asks a more fundamental question: how much generation is necessary at all?

On a phone, unnecessary generation consumes memory, increases thermal pressure, delays useful work, and expands the surface where private information might leave the device. We designed Needle around a simple principle: generate less, know locally, audit everything.

What it does

Needle is an offline knowledge micro-router demonstrated on a physical Arm64 iPhone.

A deterministic Swift host first examines each request. Structurally complete and bounded requests can be resolved without model generation. When semantic selection is required, an on-device Gemma model proposes exactly one typed route. The host—not the model—validates authorization and arguments, executes only admitted native tools, and records a local audit receipt.

Needle exposes a frozen first-party Knowledge MCP catalog with three local tasks:

  • knowledge.find — bounded search over approved local documents
  • knowledge.extract — exact field extraction with source provenance
  • knowledge.calculate — deterministic arithmetic over grounded operands

Unsupported, ambiguous, and unsafe requests fail closed through clarification, delegation, or rejection instead of being converted into plausible but unauthorized answers.

How we built it

Needle combines:

  • Swift 6.2 and SwiftUI for the application, deterministic host authority, and receipt inspector
  • An official instruction-tuned Gemma 4 E2B QAT Q4_0 GGUF
  • A pinned llama.cpp runtime performing local CPU inference on Arm64
  • A bounded JSON route contract with first-complete-object stopping
  • A first-party in-process Knowledge MCP implementation
  • SQLite/local application storage for approved knowledge sources
  • SHA-256-linked evidence and request receipts
  • Physical-device benchmark, Airplane Mode acceptance, and independent RVI/PKTAP observation harnesses

The judged workflow contains no cloud fallback, arbitrary external MCP server, downloaded executable code, or remote inference API. Model weights are acquired separately under the applicable Gemma terms and are intentionally excluded from the public repository.

Optimization: generate less

We compared the model-only baseline and Needle's optimized routing policy on the same physical iPhone, model artifact, runtime, frozen 30-case suite, prompt contract, context, output ceiling, and decode policy.

  • Baseline generated tokens: 5,541
  • Needle generated tokens: 2,011
  • Reduction: 63.71%
  • Baseline routed/completed correctly: 21/30
  • Needle routing accuracy: 30/30
  • Policy-critical decisions: 10/10
  • Model invocations: 30 → 10

The optimization does not weaken correctness to save tokens. Deterministic Swift resolves bounded cases while Gemma remains responsible for the ten semantic knowledge-tool routes.

Offline and auditable execution

A separate physical-iPhone acceptance pass completed 10/10 scripted cases in operator-attested Airplane Mode. Independent full-interval RVI/PKTAP observation found zero public-IP events attributable to the Needle application process.

Every request produces a local receipt containing the selected route, reason, model artifact identity, profile, timing spans, host validation result, native execution result, source identifiers, and the application-process network counter.

Our claim is deliberately bounded: we do not claim whole-iOS network silence, and we do not claim promoted latency, energy, or thermal improvement.

Challenges we ran into

Thermal reality on physical hardware

Long repeated inference campaigns caused serious thermal state and invalidated otherwise attractive performance comparisons. Instead of hiding this, we made the benchmark fail closed, decomposed runs into recoverable batches, and rejected latency promotion when the thermal evidence was not admissible.

Proving an air gap without overstating it

Device-wide packet captures include unrelated iOS and developer-connection traffic. We therefore separated operator Airplane Mode attestation, the application's own network counter, and independently observed process-attributable public-IP events. This produced a precise, auditable claim rather than an unrealistic whole-device assertion.

Smaller is not automatically better

We evaluated a smaller FunctionGemma candidate, but it failed the preregistered routing-quality gate. We rejected it. A smaller router is not an optimization when it chooses the wrong route.

Keeping the model under host authority

The model can propose a route, but it must never invent authorization or mutate host-owned arguments. We hardened typed argument preservation, schema validation, permission-filtered task enumeration, and deterministic failure behavior.

Accomplishments that we're proud of

  • 30/30 correct routes on the frozen physical-device suite
  • 10/10 policy-critical decisions
  • 63.71% fewer generated tokens: 5,541 → 2,011
  • 10/10 Airplane Mode acceptance cases
  • Zero independently observed public-IP events attributable to the Needle application process during acceptance
  • Reusable Swift package and local Knowledge MCP catalog
  • One-command judge verifier with fail-closed readiness gates
  • A dedicated public repository containing code, tests, specifications, and hash-linked evidence

What we learned

Mobile AI optimization is a system-design problem, not only a matrix-multiplication problem. The highest-value token may be the token the device never needs to generate.

We also learned that negative results strengthen a submission when they are governed correctly. Rejecting a smaller but unreliable model and refusing to promote thermally invalid latency results made Needle more credible, not less.

What's next

We plan to expand the admitted local knowledge catalog while preserving the same authority model: permission-filtered enumeration, typed inputs, deterministic native execution, explicit provenance, and local receipts.

Potential applications include private enterprise knowledge, regulated field workflows, offline maintenance, local compliance checks, and assistive mobile tools where connectivity is unavailable or inappropriate.

Links

Built With

Share this project:

Updates