Inspiration
On-device LLM demos often optimize a model by changing its quantization level, but this can hide the contribution of the underlying hardware acceleration. We wanted to answer a more specific question: for a given LLM and Arm Android device, which quantization level and build configuration actually provides the best balance of speed, memory usage, and quality?
ArmFit Tuner was inspired by the opportunity to separate two commonly conflated optimization factors: quantization and Arm-specific hardware acceleration. We built and tested it on a budget-tier Samsung Galaxy A16 rather than a flagship device, making the results more representative of constrained Android hardware.
What it does
ArmFit Tuner is an automated benchmarking tool for LLM inference on Arm Android devices. Given a Hugging Face model and an ADB-connected device, it:
- Downloads and converts the model to GGUF.
- Creates F16, Q8_0, and Q4_0 versions.
- Benchmarks every quantization level using both a generic llama.cpp build and a KleidiAI-enabled build.
- Measures relative quality using perplexity.
- Compares model size and performance against the device's available RAM.
- Produces a report and recommends the best-fitting configuration.
The tool also generates a static HTML dashboard with model summaries, benchmark charts, detailed results, recommendations, and data-driven insights.
How we built it
The project is built around Python, llama.cpp, Android NDK, ADB, Arm KleidiAI, and a static HTML/CSS/JavaScript dashboard.
The Python CLI orchestrates the complete workflow. Separate modules handle model conversion, quantization, device communication, benchmarking, perplexity measurement, scoring, reporting, dashboard generation, and binary discovery.
For the hardware comparison, we built two versions of llama.cpp: a generic baseline and one compiled with Arm KleidiAI. KleidiAI provides optimized matrix multiplication kernels targeting Arm CPU instructions such as DotProd. We then ran the same quantized models through both builds on the same Android device so that the performance difference could be attributed to the Arm-specific acceleration rather than simply to quantization. The benchmark was tested on a Samsung Galaxy A16 with approximately 4.2 GB of free RAM. For Qwen2.5-0.5B-Instruct, the Q4_0 KleidiAI configuration reached 27.0 tok/s decode, compared with 11.3 tok/s for Q4_0 using the baseline build, while reducing TTFT from approximately 2.3 seconds to 0.6 seconds.
Challenges we ran into
One major challenge was ensuring that the benchmark measured the intended hardware effect rather than unrelated differences between configurations. Running both builds against the same model, quantization level, and device allowed us to isolate the contribution of KleidiAI.
Cross-compiling llama.cpp for Android also required careful configuration of the Android NDK, Ninja, CPU architecture flags, and OpenMP runtime. The build process additionally had to account for the device's supported CPU instructions because compiling for unsupported instructions can result in runtime crashes. Another challenge was the live chat component. Although the UI is implemented and unit-tested, running a live llama-server on Android is currently blocked by the llama.cpp Android cross-compilation limitation. We therefore focused the working demo on the CLI benchmark pipeline and automatically generated dashboard.
Accomplishments that we're proud of
We are proud that ArmFit Tuner does more than demonstrate that a quantized model can be smaller or faster. It directly compares the same quantized model with and without Arm-specific acceleration.
The benchmark produced a particularly strong result on the Galaxy A16: Q4_0 with KleidiAI achieved 27.0 tok/s decode, about 3.6x faster than the same quantization level with the generic build, while using only 335.8 MB on disk.
We are also proud of the automated, reusable nature of the tool. It can accept different Hugging Face models and connected Arm Android devices, run the complete benchmark pipeline, and produce both machine-readable and human-readable results instead of requiring manual testing for every configuration.
What we learned
We learned that quantization and hardware acceleration should not be treated as interchangeable optimization strategies. In our measurements, quantization alone did not provide the expected speedup on the tested budget device. Q4_0 without KleidiAI achieved only 11.3 tok/s decode, while Q4_0 with KleidiAI reached 27.0 tok/s.
We also learned that hardware-aware optimization needs to be validated on the actual target device. The tested Galaxy A16 supports DotProd but not I8MM, so the build configuration must match the CPU features that are actually available.
Finally, we learned that benchmark tooling should prioritize trustworthy measurements. The dashboard's insights use the actual benchmark JSON data rather than a generative model, reducing the risk of producing plausible but incorrect explanations of measured results.
What's next for Armfit Tuner
The next step is to complete the pending quality measurements and validate the same performance pattern on the 1.5B model. This will help determine whether the observed KleidiAI advantage generalizes across model sizes rather than being specific to the 0.5B model.
We also want to expand testing across more Arm Android devices and different CPU capabilities, including devices with I8MM support. The live chat interface could be enabled once the Android llama-server cross-compilation limitation is resolved.
Longer term, ArmFit Tuner can become a broader hardware-aware tuning tool that automatically identifies the most suitable LLM configuration for a specific device instead of relying on generic optimization assumptions.
Log in or sign up for Devpost to join the conversation.