TaskPatch for Arm

Ship a task, not another model: byte-exact quantized updates for adaptive Physical AI on Arm. 12.4 MB instead of 352.2 MB, verified on Raspberry Pi 5.

Comment

Inspiration

A field camera's task can change much more often than its underlying perception model. New environments, class sets, safety rules, or output contracts should not require sending another complete model to every device.

For edge fleets, full-model updates waste bandwidth and flash storage. Runtime LoRA adapters are smaller, but they add adapter-specific machinery and recurring computation to every inference. TaskPatch started with a different question:

Can a fine-tuned task be delivered as a small, exact patch while the deployed result remains an ordinary quantized model?

What it does

TaskPatch turns a low-rank task delta into a patch of the final quantized artifact:

  1. Fold the task delta into the frozen base weights offline.
  2. Quantize the base and fused task with identical settings.
  3. Diff their quantized tensor blocks or required byte span.
  4. Apply the patch on the Arm device.
  5. Verify that the reconstructed artifact has the expected SHA-256.
patch(quantized_base, task_patch)
    == quantize(fuse(base, task_delta))
    BYTE FOR BYTE

The deployed result is still a standard GGUF or ONNX file. Compatible runtimes do not need a TaskPatch inference path, adapter router, or on-device dequantize/merge/requantize step.

Physical AI demonstration

The perception path runs a DEIMv2-X animals detector through ONNX Runtime on a Raspberry Pi 5. Recorded camera footage is processed locally and produces bounding boxes, class labels, and confidence scores. Both cats are detected in all seven sampled frames.

A seeded delta targeting 20 encoder convolutions was then folded and quantized. After INT8 quantization, only the corresponding 20 quantized weights and their 20 per-tensor scales changed: 40 of 1,300 initializers.

The changed initializer payload is 4.4 MB, or 7.4% of the quantized weights. Because ONNX protobuf fields are interleaved, the current patch format sends one 12.4 MB contiguous span, equal to 20% of the 62.1 MB model. Applying that span reconstructs the directly quantized task model byte for byte.

The seeded vision delta demonstrates the deployment mechanism; it is not presented as accuracy evidence for a newly learned class set.

Quantized language-model stress test

The same method was tested independently with llama.cpp and Qwen2.5-0.5B on the Raspberry Pi 5:

  • Full Q4_0 task model: 352.2 MB
  • Per-task patch: 12.4 MB
  • Update reduction: approximately 28×
  • Patch share: 3.5% of the model
  • In-place write primitive: 4.2 ms
  • Decode throughput: 31.1 tokens/s
  • Reconstructed artifact: SHA-256 identical

The 4.2 ms figure is the measured page-cache-visible write primitive; full-file hashing and storage flush are reported separately.

Three trained LoRAs—shout, json, and bullet—also demonstrate visible behavioral changes while using one active model file and small per-task patches. Byte-exact reconstruction was tested across Q4_0, Q8_0, and Q4_K_M. The Q8_0 byte-identity experiment uses seeded synthetic tasks, while the trained-task reconstruction records cover Q4_0 and Q4_K_M.

How it was built

TaskPatch includes:

  • A Python/PyTorch package for exact low-rank fusion, packaging, switching, restoration, and verification.
  • gguf_patch.py for creating, validating, applying, and verifying quantized GGUF patches.
  • onnx_patch.py for the equivalent ONNX workflow.
  • Safe sibling-task switching with base-family and tensor-site fingerprints.
  • Automatic rollback when an unknown sibling patch fails full-file verification.
  • Arm capability detection for NEON, dotprod, i8mm, SVE, and SME variants.
  • Reproducible training seeds and byte hashing for scalar, fp32, and bfloat16 tensors.
  • Benchmark harnesses for page cache, storage, thermal behavior, task fidelity, runtime LoRA, and KleidiAI.
  • A project-authored ONNX fixture that runs the complete fuse, INT8, diff, patch, SHA-256, and ONNX Runtime path without downloading a private model.
  • GitHub Actions and a documentation checker that derives README claims from the committed measurement JSON.

Arm-specific optimization

Deployment measurements were collected on a Raspberry Pi 5 with four Cortex-A76 cores, 8 GB RAM, passive cooling, and SD-card storage.

The board exposes NEON, FP16, and dotprod, but not i8mm, SVE, or SME. Rather than assuming that an optimization helps every workload, the same pinned llama.cpp revision was built with KleidiAI enabled and disabled.

For the measured 135M configuration:

  • Q8_0 prompt processing improved from 751 to 1,028 tokens/s, a 37% gain.
  • Q8_0 decode remained at parity.
  • Q4_0 decode was 9% slower with KleidiAI.

The result is deliberately scoped to this model and board: use KleidiAI for Q8_0 prompt-heavy work in this measured configuration, leave it off for Q4_0 decode, and remeasure before generalizing.

Challenges

Proving tensor locality

TaskPatch depends on deterministic, tensor-local quantization. Untouched tensors must produce identical quantized bytes.

Q4_0, Q8_0, Q4_K_M, and ONNX Runtime dynamic INT8 passed this test. Per-task calibration imatrices did not: differences spread to 209 of 272 tensors. Freezing one imatrix computed from the base restored locality to exactly 60 of 272 tensors and restored byte identity.

Stable ONNX reconstruction

Editing an ONNX graph and serializing it again did not reproduce the same file because protobuf serialization is not byte-stable. The solution was to patch the measured byte span directly and verify the complete output hash.

Safe direct task switching

Applying task B directly over task A is unsafe when their tensor-site sets differ; bytes unique to task A could survive. TaskPatch now seals the active state with the base family, exact site-set fingerprint, expected output hash, and file identity. Unknown combinations require full verification and automatically roll back on failure.

Honest benchmarking

Page cache, filesystem type, thermal state, and storage flush can completely change the result. The benchmark records these conditions and rejects tmpfs data for storage claims.

Warm patching and a warm separate model were effectively at parity. The measured 2× latency advantage appeared only when the alternative full model was cold. TaskPatch therefore claims smaller updates and exact reconstruction unconditionally, but only claims latency gains under the conditions where they were measured.

What was learned

Optimization on Arm is workload-, phase-, and quantization-specific. INT8 reduced the detector from 238.8 MB to 62.1 MB, but it was slower than fp32 in the measured Cortex-A76/ONNX Runtime configuration. KleidiAI accelerated Q8_0 prompt processing but did not improve Q4_0 decode.

Runtime LoRA also remains a valid alternative. It uses approximately 4× less per-task storage, while folding recovered 5.4–6.2% decode throughput in this experiment and produced a plain, byte-verifiable model artifact.

The best deployment choice depends on whether storage, update bandwidth, runtime simplicity, or decode throughput matters most.

Accomplishments

  • Approximately 28× smaller Qwen task updates.
  • Byte-identical reconstruction on Raspberry Pi 5.
  • Cross-machine proof: patches built on a Mac reconstructed files on the Pi that were never transferred as complete task models.
  • Independent demonstrations in ONNX Runtime and llama.cpp.
  • Q4_0, Q8_0, Q4_K_M, and frozen-imatrix coverage.
  • A measured 37% KleidiAI gain for Q8_0 prompt processing.
  • Public tests, raw measurement records, CI, licensing information, and complete Arm reproduction instructions.

What's next

The next steps are staged patch activation with verify-and-atomic-rename or two model slots, per-tensor ONNX patch ranges, external-data ONNX and safetensors support, trained perception-task patches, larger language models, and measurements on newer Arm cores with i8mm or SME2.

The goal is a reusable deployment layer where an edge fleet keeps one trusted base artifact and receives small, verifiable task updates without changing its inference runtime.

Built With

  • ai
  • arm
  • cortex-a76
  • edge
  • lora
  • physical
  • transformers
Share this project:

Updates

Submission history