Arm64 Inference Scout

Project overview

Arm64 Inference Scout is a reproducible, code-first tool for deciding whether INT8 quantization is appropriate for a compact AI inference workload on Arm64 hardware. It turns the vague claim that a model is “smaller and faster” into a local, reviewable measurement.

What it does

The CLI creates a deterministic classification-style workload, compares FP32 against symmetric INT8 inference, and writes JSON plus a Markdown decision report. It measures median per-sample latency, parameter bytes, and top-1 output agreement. It deliberately exposes latency regressions or quality loss rather than hiding them.

Arm optimization work

INT8 reduces parameter storage by roughly four times. Scout measures the actual latency trade-off on the local Arm64 host, so developers can make an evidence-based choice before moving to an Arm-optimized production runtime such as LiteRT, ONNX Runtime, ExecuTorch, or llama.cpp.

How we built it

We built a small Python and NumPy benchmark with transparent symmetric quantization, INT32 accumulation, seeded inputs, warmup, and median latency measurement. The resulting report records its host architecture and advises repeating the run on the deployment target.

Challenges and learning

The benchmark showed a major storage reduction while the naive NumPy INT8 path was slower on this machine. That outcome is valuable: quantization is not automatically a speed optimization. The tool gives a team evidence to choose the right runtime and target hardware instead of optimizing by assumption.

Validation

The public repository includes setup instructions, unit tests, linting, methodology, and the full benchmark report format.

Built With

• python