Project overview
Arm Cloud Optimizer is an evidence-first developer workflow for CPU inference services on Arm64 cloud instances. It measures a safe micro-batching change before teams deploy it, retaining a benchmark artifact instead of relying on generic performance claims.
What it does
The CLI runs identical dense FP32 inference in two modes: request-by-request and fixed-size micro-batches. It measures median elapsed time and requests per second, compares numerical outputs using a configurable float32 tolerance, and writes JSON evidence plus rollout advice.
Arm optimization work
On the local Arm64 machine, the reproducible 512×192→96 workload with batch size 32 measured 1.523 ms serial versus 0.099 ms micro-batched: 15.39× throughput with a maximum output drift of 0.0000343, within the 0.0001 tolerance. The tool recommends a bounded queue only after the output check succeeds.
How we built it
We used Python and NumPy with a deterministic seeded workload, warmed benchmark runs, median timing, and explicit equivalence checks. The project produces a machine-readable report suitable for a change review.
Challenges and learning
Batching can improve throughput while harming queue latency or silently changing numerical behavior. The hardest part was making the comparison conservative: the report explicitly excludes networking, tokenization, model loading, queueing, and p95/p99 latency, and it asks developers to rerun it on the exact Arm cloud instance before deployment.
Validation
The public repository includes setup instructions, unit tests, linting, a sample benchmark report, and detailed limitations. It has no network calls, credentials, or cloud-account requirement.
Log in or sign up for Devpost to join the conversation.